Knowledge-Based Intelligent Information and Engineering Systems: 7th International Conference, KES 2003, Oxford, UK, September 3-5, 2003, Proceedings, ... Notes in Artificial Intelligence Pt. 1

Distributed Prediction and Hierarchical Knowledge Discovery by ARTMAP Neural Networks Gail A. Carpenter Department of Cognitive and Neural Systems, Boston University 677 Beacon Street, Boston, MA 02215 USA [email protected]

Abstract Adaptive Resonance Theory (ART) neural networks model real-time prediction, search, learning, and recognition. ART networks function both as models of human cognitive information processing [1,2,3] and as neural systems for technology transfer [4]. A neural computation central to both the scientific and the technological analyses is the ART matching rule [5], which models the interaction between topdown expectation and bottom-up input, thereby creating a focus of attention which, in turn, determines the nature of coded memories. Sites of early and ongoing transfer of ART-based technologies include industrial venues such as the Boeing Corporation [6] and government venues such as MIT Lincoln Laboratory [7]. A recent report on industrial uses of neural networks [8] states: “[The] Boeing … Neural Information Retrieval System is probably still the largest-scale manufacturing application of neural networks. It uses [ART] to cluster binary templates of aeroplane parts in a complex hierarchical network that covers over 100,000 items, grouped into thousands of self-organised clusters. Claimed savings in manufacturing costs are in millions of dollars per annum.” At Lincoln Lab, a team led by Waxman developed an image mining system which incorporates several models of vision and recognition developed in the Boston University Department of Cognitive and Neural Systems (BU/CNS). Over the years a dozen CNS graduates (Aguilar, Baloch, Baxter, Bomberger, Cunningham, Fay, Gove, Ivey, Mehanian, Ross, Rubin, Streilein) have contributed to this effort, which is now located at Alphatech, Inc. Customers for BU/CNS neural network technologies have attributed their selection of ART over alternative systems to the model's defining design principles. In listing the advantages of its THOT® technology, for example, American Heuristics Corporation (AHC) cites several characteristic computational capabilities of this family of neural models, including fast on-line (one-pass) learning, “vigilant” detection of novel patterns, retention of rare patterns, improvement with experience, “weights [which] are understandable in real world terms,” and scalability (www.heuristics.com). Design principles derived from scientific analyses and design constraints imposed by targeted applications have jointly guided the development of many variants of the basic networks, including fuzzy ARTMAP [9], ART-EMAP [10], ARTMAP-IC [11], V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 1-4, 2003.  Springer-Verlag Berlin Heidelberg 2003

2

Gail A. Carpenter

Gaussian ARTMAP [12], and distributed ARTMAP [3,13]. Comparative analysis of these systems has led to the identification of a default ARTMAP network, which features simplicity of design and robust performance in many application domains [4,14]. Selection of one particular ARTMAP algorithm is intended to facilitate ongoing technology transfer. The default ARTMAP algorithm outlines a procedure for labeling an arbitrary number of output classes in a supervised learning problem. A critical aspect of this algorithm is the distributed nature of its internal code representation, which produces continuous-valued test set predictions distributed across output classes. The character of their code representations, distributed vs. winner-take-all, is, in fact, a primary factor differentiating various ARTMAP networks. The original models [9,15] employ winner-take-all coding during training and testing, as do many subsequent variations and the majority of ART systems that have been transferred to technology. ARTMAP variants with winner-take-all (WTA) coding and discrete target class predictions have, however, shown consistent deficits in labeling accuracy and post-processing adjustment capabilities. The talk will describe a recent application that relies on distributed code representations to exploit the ARTMAP capacity for one-to-many learning, which has enabled the development of self-organizing expert systems for multi-level object grouping, information fusion, and discovery of hierarchical knowledge structures. A pilot study has demonstrated the network's ability to infer multi-level fused relationships among groups of objects in an image, without any supervised labeling of these relationships, thereby pointing to new methodologies for self-organizing knowledge discovery.

References [1] [2]

[3] [4]

[5]

S. Grossberg, “The link between brain, learning, attention, and consciousness,” Consciousness and Cognition, vol. 8, pp. 1-44, 1999. ftp://cns-ftp.bu.edu/pub/diana/Gro.concog98.ps.gz S. Grossberg, “How does the cerebral cortex work? Development, learning, attention, and 3D vision by laminar circuits of visual cortex,”Behavioral and Cognitive Neuroscience Reviews, in press, 2003, http://www.cns. bu.edu/Profiles/Grossberg/Gro2003BCNR.pdf G.A. Carpenter, “Distributed learning, recognition, and prediction by ART and ARTMAP neural networks,” Neural Networks, vol. 10, pp. 1473-1494, 1997, http://cns.bu.edu/~gail/115_dART_NN_1997_.pdf O. Parsons and G.A. Carpenter, “ARTMAP neural networks for information fusion and data mining: map production and target recognition methodologies,” Neural Networks, vol. 16, 2003, http://cns.bu.edu/~gail/ ARTMAP_map_2003_.pdf G.A. Carpenter and S. Grossberg, “A massively parallel architecture for a self-organizing neural pattern recognition machine,” Computer Vision, Graphics, and Image Processing, vol. 37, pp. 54-115, 1987.

Distributed Prediction and Hierarchical Knowledge Discovery

3

[6]

T.P. Caudell, S.D.G. Smith, R. Escobedo, and M. Anderson, “NIRS: Large scale ART 1 neural architectures for engineering design retrieval,” Neural Networks, vol. 7, pp. 1339-1350, 1994, http://cns.bu.edu/~gail/ NIRS_Caudell_1994_.pdf [7] W. Streilein, A. Waxman, W. Ross, F. Liu, F., M. Braun, D. Fay, P. Harmon, and C.H. Read, “Fused multi-sensor image mining for feature foundation data,” In Proceedings of 3rd International Conference on Information Fusion, Paris, vol. I, 2000. [8] P. Lisboa, “Industrial use of saftey-related artificial neural netoworks,” Contract Research Report 327/2001, Liverpool John Moores University, 2001. http://www.hse.gov.uk/research/crr_pdf/2001/crr01327. pdf [9] G.A. Carpenter, S. Grossberg, N. Markuzon, J.H. Reynolds, and D.B. Rosen, “Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps,” IEEE Transactions on Neural Networks, vol. 3, pp. 698-713, 1992, http://cns.bu.edu/~gail/ 070_Fuzzy_ARTMAP_1992_.pdf [10] G.A. Carpenter and W.D. Ross, “ART-EMAP: A neural network architecture for object recognition by evidence accumulation,” IEEE Transactions on Neural Networks, vol. 6, pp. 805-818, 1995, http://cns.bu. edu/~gail/097_ART-EMAP_1995_.pdf [11] G.A. Carpenter and N. Markuzon, “ARTMAP-IC and medical diagnosis: Instance counting and inconsistent cases,” Neural Networks, vol. 11, pp. 323-336, 1998. http://cns.bu.edu/~gail/ 117_ARTMAP-IC_1998_.pdf [12] J.R. Williamson, “Gaussian ARTMAP: A neural network for fast incremental learning of noisy multidimensional maps,” Neural Networks, vol. 9, pp. 881-897, 1998, http://cns.bu.edu/~gail/ G-ART_Williamson_1998_.pdf [13] G.A. Carpenter, B.L. Milenova, and B.W. Noeske, “Distributed ARTMAP: a neural network for fast distributed supervised learning,” Neural Networks, vol. 11, pp. 793-813, 1998, http://cns.bu.edu/~gail/ 120_dARTMAP_1998_.pdf [14] G.A. Carpenter “Default ARTMAP,” Proceedings of the International Joint Conference on Neural Networks (IJCNN'03), 2003. http://cns.bu.edu/~gail/Default_ARTMAP_ 2003_.pdf [15] G.A. Carpenter, S. Grossberg, and J.H. Reynolds, “ARTMAP: Supervised real-time learning and classification of nonstationary data by a self-organizing neural network,” Neural Networks, vol. 4, pp. 565-588, 1991, http://cns.bu.edu/~gail/054_ ARTMAP_1991_.pdf.

Biography Gail Carpenter (http://cns.bu.edu/~gail/) obtained her graduate training in mathematics at the University of Wisconsin (PhD, 1974) and taught at MIT and Northeastern University before moving to Boston University, where she is a professor

4

Gail A. Carpenter

in the departments of cognitive and neural systems (CNS) and mathematics. She is director of the CNS Technology Lab and CNS director of graduate studies; serves on the editorial boards of Brain Research, IEEE Transactions on Neural Networks, Neural Computation, Neural Networks, and Neural Processing Letters; has been elected to successive three-year terms on the Board of Governors of the International Neural Network Society (INNS) since its founding, in 1987; and was elected member-at-large of the Council of the American Mathematical Society (1996-1999). She has received the INNS Gabor Award and the Slovak Artificial Intelligence Society Award. She regularly serves as an organizer and program committee member for international conferences and workshops, and has delivered many plenary and invited addresses. Together with Stephen Grossberg and their students and colleagues, Professor Carpenter has developed the Adaptive Resonance Theory (ART) family of neural networks for fast learning, pattern recognition, and prediction, including both unsupervised (ART 1, ART 2, ART 2-A, ART 3, fuzzy ART, distributed ART) and supervised (ARTMAP, fuzzy ARTMAP, ARTEMAP, ARTMAP-IC, ARTMAP-FTR, distributed ARTMAP, default ARTMAP) systems. These ART models have been used for a wide range of applications, such as remote sensing, medical diagnosis, automatic target recognition, mobile robots, and database management. Other research topics include the development, computational analysis, and application of neural models of vision, nerve impulse generation, synaptic transmission, and circadian rhythms.

The Brain's Cognitive Dynamics: The Link between Learning, Attention, Recognition, and Consciousness Stephen Grossberg Center for Adaptive Systems and Department of Cognitive and Neural Systems Boston University, 677 Beacon Street, Boston, MA 02215 [email protected] http://www.cns.bu.edu/Profiles/Grossberg

Abstract The processes whereby our brains continue to learn about a changing world in a stable fashion throughout life are proposed to lead to conscious experiences. These processes include the learning of top-down expectations, the matching of these expectations against bottom-up data, the focusing of attention upon the expected clusters of information, and the development of resonant states between bottom-up and top-down processes as they reach a predictive and attentive consensus between what is expected and what is there in the outside world. It is suggested that all conscious states in the brain are resonant states, and that these resonant states trigger learning of sensory and cognitive representations when they amplify and synchronize distributed neural signals that are bound by the resonance. Thus, processes of learning, intention, attention, synchronization, and consciousness are intimately bound up together. The name Adaptive Resonance Theory, or ART, summarizes the predicted link between these processes. Illustrative psychophysical and neurobiological data have been explained and quantitatively simulated using these concepts in the areas of early vision, visual object recognition, auditory streaming, and speech perception, among others. It is noted how these mechanisms seem to be realized by known laminar circuits of the visual cortex. In particular, they seem to be operative at all levels of the visual system. Indeed, the mammalian neocortex, which is the seat of higher biological intelligence in all modalities, exhibits a remarkably uniform laminar architecture, with six characteristic layers and sublamina. These known laminar ART, or LAMINART, models illustrate the emerging paradigm of Laminar Computing which is attempting to answer the fundamental question: How does laminar computing give rise to biological intelligence? These laminar circuits also illustrate the fact that, in a rapidly growing number of examples, an individual model can quantitatively simulate the recorded dynamics of identified neurons in anatomically characterized circuits and the behaviors that they control. In this precise sense, the classical Mind/Body problem is starting to get solved. It is further noted that many parallel processing streams of the brain often compute properties that are complementary to each other, much as a lock fits a key or the pieces of a puzzle fit together. Hierarchical and parallel interactions within and between these processing streams can overcome their complementary deficiencies by generating emergent properties that compute complete information about a prescribed V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 5-12, 2003.  Springer-Verlag Berlin Heidelberg 2003

6

Stephen Grossberg

form of intelligent behavior. This emerging paradigm of Complementary Computing is proposed to be a better paradigm for understanding biological intelligence than various previous proposals, such as the postulate of independent modules that are specialized to carry out prescribed intelligent tasks. Complementary computing is illustrated by the fact that sensory and cognitive processing in the What processing stream of the brain, that passes through cortical areas V1-V2-V4-IT on the way to prefrontal cortex, obey top-down matching and learning laws that are often complementary to those used for spatial and motor processing in the brain's Where/How processing stream, that passes through cortical areas V1-MT-MST-PPC on the way to prefrontal cortex. These complementary properties enable sensory and cognitive representations to maintain their stability as we learn more about the world, while allowing spatial and motor representations to forget learned maps and gains that are no longer appropriate as our bodies develop and grow from infanthood to adulthood. Procedural memories are proposed to be unconscious because the inhibitory matching process that supports their spatial and motor processes cannot lead to resonance. Because ART principles and mechanisms clarify how incremental learning can occur autonomously without a loss of stability under both unsupervised and supervised conditions in response to a rapidly changing world, algorithms based on ART have been used in a wide range of applications in science and technology.

Biography Stephen Grossberg is Wang Professor of Cognitive and Neural Systems and Professor of Mathematics, Psychology, and Biomedical Engineering at Boston University. He is the founder and Director of the Center for Adaptive Systems, founder and Chairman the Department of Cognitive and Neural Systems, founder and first President of the International Neural Network Society (INNS), and founder and co-editor-in-chief of the official journal, Neural Networks, of INNS and the European Neural Network Society (ENNS) and Japanese Neural Network Society (JNNS). Grossberg has served as an editor of many other journals, including Journal of Cognitive Neuroscience, Behavioral and Brain Sciences, Cognitive Brain Research, Cognitive Science, Adaptive Behavior, Neural Computation, Journal of Mathematical Psychology, Nonlinear Analysis, IEEE Expert, and IEEE Transactions on Neural Networks. He was general chairman of the IEEE First International Conference on Neural Networks and played a key role in organizing the first annual INNS conference. Both conferences have since fused into the International Joint Conference on Neural Networks (IJCNN), the largest conference on biological and technological neural network research in the world. His lecture series at MIT Lincoln Laboratory on neural network technology was instrumental in motivating the laboratory to initiate the national DARPA Study on Neural Networks. He has received a number of awards, including the 1991 IEEE Neural Network Pioneer award, the 1992 INNS Leadership Award, the1992 Thinking Technology Award of the Boston Computer Society, the 2000 Information Science Award of the Association for Intelligent Machinery, the 2002 Charles River Laboratories prize of the Society for Behavioral Toxicology, and the 2003 INNS Helmholtz award. He was elected a fellow of the American

The Brain’s Cognitive Dynamics

7

Psychological Association in 1994, a fellow of the Society of Experimental Psychologists in 1996, and a fellow of the American Psychological Society in 2002. He and his colleagues have pioneered and developed a number of the fundamental principles, mechanisms, and architectures that form the foundation for contemporary neural network research, particularly those which enable individuals to adapt autonomously in real-time to unexpected environmental changes. Such models have been used both to analyse and predict interdisciplinary data about mind and brain, and to suggest novel architectures for technological applications. Grossberg received his graduate training at Stanford University and Rockefeller University, and was a Professor at MIT before assuming his present position at Boston University. Core modeling references from the work of Grossberg and his colleagues for neural models of working memory and short-term memory, learning and long-term memory, expectation, attention, resonance, synchronization, recognition, categorization, memory search, hypothesis testing, and consciousness in vision, visual object recognition, audition, speech, cognition, and cognitive-emotional interactions. Some articles since 1997 can be downloaded from http://www.cns.bu.edu/ Profiles/Grossberg

References [1] [2]

[3] [4] [5] [6] [7]

[8]

Baloch, A.A. and Grossberg, S. (1997). A neural model of high-level motion processing: Line motion and formotion dynamics. Vision Research, 37, 30373059. Banuet, J-P. and Grossberg, S. (1987). Probing cognitive processes through the structure of event-related potentials during learning: An experimental and theoretical analysis. Applied Optics, 26, 4931-4946. Reprinted in Carpenter, G.A. and Grossberg, S. (1991). Pattern Recognition by Self-Organizing Neural Networks. Cambridge, MA: MIT Press. Boardman, I., Grossberg, S., Myers, C., and Cohen, M.A. (1999). Neural dynamics of perceptual order and context effects for variable-rate speech syllables. Perception and Psychophysics, 61, 1477-1500. Bradski, G. and Grossberg, S. (1995). Fast-learning VIEWNET architectures for recognizing three-dimensional objects from multiple two-dimensional views. Neural Networks, 8, 1053-1080. Bradski, G., Carpenter, G.A., and Grossberg, S. (1992). Working memory networks for learning temporal order with application to three-dimensional visual object recognition. Neural Computation, 4, 270-286. Bradski, G, Carpenter, G.A., and Grossberg, S. (1994). STORE working memory networks for storage and recall of arbitrary temporal sequences. Biological Cybernetics, 71, 469-480. Carpenter, G.A. (1989). Neural network models for pattern recognition and associative memory. Neural Networks, 1989, 2, 243-257. Reprinted in Carpenter, G.A. and Grossberg, S. (1991). Pattern Recognition by SelfOrganizing Neural Networks. Cambridge, MA: MIT Press. Carpenter, G.A. (1997). Distributed learning, recognition, and prediction by ART and ARTMAP neural networks. Neural Networks, 10,1473-1494.

8

[9]

[10]

[11]

[12] [13] [14] [15]

[16]

[17] [18] [19] [20]

[21]

Stephen Grossberg

Carpenter, G.A. and Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37, 54-115. Reprinted in Grossberg, S. (1988). Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press. Carpenter, G.A. and Grossberg, S. (1987). ART 2: Self-organization of stable category recognition codes for analog input patterns. Applied Optics, 26,49194930. Reprinted in Carpenter, G.A. and Grossberg, S. (1991). Pattern Recognition by Self-Organizing Neural Networks. Cambridge, MA: MIT Press. Carpenter, G.A. and Grossberg, S. (1990). ART 3: Hierarchical search using chemical transmitters in self-organizing pattern recognition architectures. Neural Networks, 3, 129-152. Reprinted in Carpenter, G.A. and Grossberg, S. (1991). Pattern Recognition by Self-Organizing Neural Networks. Cambridge, MA: MIT Press. Carpenter, G.A. and Grossberg, S. (1991). Pattern Recognition by SelfOrganizing Neural Networks. Cambridge, MA: MIT Press. Carpenter, G.A. and Grossberg, S. (1992). A self-organizing neural network for supervised learning, recognition, and prediction. IEEE Communications Magazine, September, 38-49. Carpenter, G.A. and Grossberg, S. (1993). Normal and amnesic learning, recognition, and memory by a neural model of cortico-hippocampal interactions. Trends in Neurosciences, 16, 131-137. Carpenter. G.A., Grossberg, S., Markuzon, N., Reynolds, J.H., and Rosen, D.B. (1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimentional maps. IEEE Transactions on Neural Networks, 3, 698-713. Carpenter, G.A., Grossberg, S., and Reynolds, J.H. (1991). ARTMAP: Supervised real-time learning and classification of nonstationary data by a selforganizing neural network. Neural Networks, 4, 565-588. Reprinted in Carpenter, G.A. and Grossberg, S. (1991). Pattern Recognition by SelfOrganizing Neural Networks. Cambridge, MA: MIT Press. Carpenter, G.A., Grossberg, S., and Reynolds, J.H. (1995). A fuzzy ARTMAP nonparametric probability estimator for nonstationary pattern recognition problems. IEEE Transactions on Neural Networks, 6, 1330-1336. Carpenter, G.A., Grossberg, S., and Rosen, D.B. (1991). ART 2-A: An adaptive resonance algorithm for rapid category learning and recognition, Neural Networks, 4, 493-504. Chey, J., Grossberg, S., and Mingolla, E. (1997). Neural dynamics of motion grouping: From aperture ambiguity to object speed and direction. Journal of the Optical Society of America A, 14, 2570-2594. Cohen, M.A. and Grossberg, S. (1983). Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactrions on Systems, Man, and Cybernetics, 13, 815-826. Reprinted in S. Grossberg (1987). The Adaptive Brain, Vol. I. Amsterdam: Elsevier Science. Cohen, M.A. and Grossberg, S. (1986). Neural dynamics of speech and language coding: Developmental programs, perceptual grouping, and competition for short term memory. Human Neurobiology, 5, 1-22. Reprinted in S. Grossberg (1987). The Adaptive Brain, Vol. II. Amsterdam: Elsevier Science.

The Brain’s Cognitive Dynamics

9

[22] Cohen, M.A. and Grossberg, S. (1987). Masking fields: A massively parallel neural architecture for learning, recognizing, and predicting multiple groupings of patterned data. Applied Optics, 26, 1866-1891. Reprinted in Grossberg, S. (1988). Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press. [23] Cohen, M.A., Grossberg, S. and Stork, D.G. (1988). Speech perception and production by a self-organizing neural network. In Evolution, Learning, Cognition, and Advanced Architectures. (Y.C. Lee, Ed.). Singapore: World Scientific. Reprinted in Carpenter, G.A. and Grossberg, S. (1991). Pattern Recognition by Self-Organizing Neural Networks. Cambridge, MA: MIT Press. [24] Ellias, S.A. and Grossberg, S. (1975). Pattern formation, contrast control, and oscillations in the short-term memory of shunting on-center off-surround networks. Biological Cybernetics, 20, 69-98. [25] Gove, A., Grossberg, S., and Mingolla, E. (1995). Brightness perception, illusory contours, and corticogeniculate feedback. Visual Neuroscience, 12, 1027-1052. [26] Granger, E., Rubin, M., Grossberg, S., and Lavoie, P. (2001). A what-andwhere fusion neural network for recognition and tracking of multiple radar emitters. Neural Networks, 14, 325-344. [27] Grossberg, S. (1969). On learning and energy-entropy dependence in recurrent and nonrecurrent signed networks. Journal of Statistical Physics, 1969, 1, 319350. [28] Grossberg, S. (1971). Pavlovian pattern learning by nonlinear neural networks. Proceedings of the National Academy of Sciences, 68, 828-831. [29] Grossberg, S. (1972). Pattern learning by functional-differential neural networks with arbitrary path weights. In Delay and functional-differential equations and their applications (K. Schmitt, Ed.). New York: Academic Press, 1972. [30] Grossberg, S. (1973) Contour enhancement, short term memory, and constancies in reverberating neural Networks. Studies in Applied Mathematics, LII, 213-257. Reprinted in Grossberg, S. (1982) Studies of Mind and Brain. New York: Kluwer/Reidel. [31] Grossberg, S. (1974). Classical and instrumental learning by neural networks. In Progress in Theoretical Biology, Vol. 3 (R. Rosen and F. Snell, Eds.), pp. 51141. New York: Academic Press, 1974. [32] Grossberg, S. (1976). Adaptive pattern classification and universal recoding, I: Parallel development and coding of neural feature detectors. Biological Cybernetics, 23, 121-134. Reprinted in Grossberg, S. (1982) Studies of Mind and Brain. New York: Kluwer/Reidel. [33] Grossberg, S. (1976). Adaptive pattern classification and universal recoding, II: Feedback, expectation, olfaction, and illusions. Biological Cybernetics, 23, 187202. [34] Grossberg, S. (1977). Pattern formation by the global limits of a nonlinear competitive interaction in n dimensions. Journal of Mathematical Biology, 4, 237-256.

10

Stephen Grossberg

[35] Grossberg, S. (1978). A theory of human memory: Self-organization and performance of sensory-motor codes, maps and plans. In Progress in Theoretical Biology, Vol. 5 (R. Rosen and F. Snell, Eds.), pp. 233-374. New York, Academic Press. Reprinted in Grossberg, S. (1982) Studies of Mind and Brain. New York: Kluwer/Reidel. [36] Grossberg, S. (1978). Behavioral contrast in short term memory: serial binary memory models or parallel continuous memory models. Journal of Mathematical Psychology, 17, 199-219. Reprinted in Grossberg, S. (1982) Studies of Mind and Brain. New York: Kluwer/Reidel. [37] Grossberg, S. (1978). Competition, decision, and consensus. Journal of Mathematical Analysis and Applications, 66, 470-493. Reprinted in Grossberg, S. (1982) Studies of Mind and Brain. New York: Kluwer/Reidel. [38] Grossberg, S. (1978). Decisions, patterns, and oscillations in nonlinear competitive systems with applications to Volterra-Lotka systems. Journal of Theoretical Biology, 73, 101-130. [39] Grossberg, S. (1980). Biological competition: Decision rules, pattern formation, and oscillations. Proceedings of the National Academy of Sciences, 77, 23382342. [40] Grossberg, S. (1980). How does a brain build a cognitive code? Psychological Review, 87, 1-51. Reprinted in Grossberg, S. (1982) Studies of Mind and Brain. New York: Kluwer/Reidel. [41] Grossberg, S. (1982) Studies of Mind and Brain: Neural principles of learning, perception, development, cognition, and motor control. New York: Kluwer/Reidel. [42] Grossberg, S. (1982). Associative and competitive principles of learning and development: The temporal unfolding and stability of STM and LTM patterns. In Competition and Cooperation in Neural Nets (S. Amari and M. Arbib, Eds.). Lecture Notes in Biomathematics, 45, 295-341. New York: Springer-Verlag. Reprinted in S. Grossberg (1987). The Adaptive Brain, Vol. I. Amsterdam: Elsevier Science. [43] Grossberg, S. (1982). Processing of expected and unexpected events during conditioning and attention: A psychophysiological theory. Psychological Review, 89, 529-572. [44] Grossberg, S. (1984). Some normal and abnormal behavioral syndromes due to transmitter gating of opponent processes. Biological Psychiatry, 19, 1075-1118. [45] Grossberg, S. (1984). Some psychophysiological and pgharmacological correlates of a developmental, cognitive, and motivational theory. In Brain and Information: Event Related Potentials, 425, 58-151. (R. Karrer, J. Cohen, and P. Tueting, Ed.s). New York Academy of Sciences. Reprinted in Grossberg, S. (1987. The Adaptive Brain, Vol. 1. Amsterdam: Elsevier Science. [46] Grossberg, S. (1984). Unitization, automaticity, temporal order, and word recognition. Cognition and Brain Theory, 7, 263-283. [47] Grossberg, S. (1987). Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11, 23-63. Reprinted in Grossberg, S. (1988). Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press. [48] Grossberg, S. (1987). The Adaptive Brain, Vols. I and II. Amsterdam: Elsevier Science.

The Brain’s Cognitive Dynamics

11

[49] Grossberg, S. (1988). Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press. [50] Grossberg, S. (1988). Nonlinear neural networks: Principles, mechanisms, and architectures. Neural Networks, 1, 17-61. Reprinted in Carpenter, G.A. and Grossberg, S. (1991). Pattern Recognition by Self-Organizing Neural Networks. Cambridge, MA: MIT Press. [51] Grossberg, S. (1995). The attentive brain. American Scientist, 83, 438-449. [52] Grossberg, S. (1999). How does the cerebral cortex work: Learning, attention, and grouping by the laminar circuits of visual cortex. Spatial Vision, 12, 163186. [53] Grossberg, S. (1999). The link between brain learning, attention, and consciousness. Consciousness and Cognition, 8, 1- 44. [54] Grossberg, S. (1999). Pitch-based streaming in auditory perception. In Musical networks: Parallel Distributed Perception and Performance (N. Griffith and P. Todd, Eds.),. Cambridge, MA: MIT Press, pp.117-140. [55] Grossberg, S. (2000). The complementary brain: Unifying brain dynamics and modularity. Trends in Cognitive Sciences, 233-246. [56] Grossberg, S. (2000). The imbalanced brain: From normal behavior to schizophrenia. Biological Psychiatry, 81-98. [57] Grossberg, S. (2000). How hallucinations may arise from brain mechanisms of learning, attention, and volition. Journal of the International Neuropsychological Society, 6, 583-592. [58] Grossberg, S., Boardman, I., and Cohen, M.A. (1997). Neural dynamics of variable-rate speech categorization. Journal of Experimental Psychology: Human Perception and Performance, 23,481-503. [59] Grossberg, S. and Grunewald, A. (1997). Cortical synchronization and perceptual framing. Journal of Cognitive Neuroscience, 9, 117-132. [60] Grossberg, S. and Howe, P.D.L. (2002). A laminar cortical model of stereopsis and three-dimensional surface perception. Vision Research, in press. [61] Grossberg, S. and Levine, D. (1976). Some developmental and attentional biases in the contrast enhancement and short-term memory of recurrent neural networks. Journal of Theoretical Biology, 53, 341-380. [62] Grossberg, S. and Levine, D. (1987). Neural dynamics of attentionally modulated Pavlovian cnditioning: Blocking, interstimulus interval, and secondary conditioning. Applied Optics, 26, 5015-5030. Reprinted in Grossberg, S. (1988). Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press. [63] Grossberg, S. and Merrill, J.W.L. (1992). A neural network model of adaptively timed reinforcement learning and hippocampal dynamics. Cognitive Brain Research, 1, 3-38. [64] Grossberg, S. and Merrill, J.W.L. (1996). The hippocampus and cerebellum in adaptively timed learning, recognition, and movement. Journal of Cognitive Neuroscience, 8, 257-277. [65] Grossberg, S., Mingolla, E., and Ross, W.D. (1994). A neural theory of attentive visual search: Interactions of boundary, surface, spatial, and object representations. Psychological Review, 101, 470-489.

12

Stephen Grossberg

[66] Grossberg, S. Mingolla, E., and Ross, W.D. (1997). Visual brain and visual perception: How does the cortex do perceptual grouping? Trends in Neurosciences, 20, 106-111. [67] Grossberg, S., Mingolla, E., and Viswanathan, L. (2001). Neural dynamics of motion integration and segmentation within and across apertures. Vision Research, 41, 2521-2553. [68] Grossberg, S. and Myers, C. (2000). The resonant dynamics of speech perception: Interword integration and duration-dependent backward effects. Psychological Review, 107, 735-767. [69] Grossberg, S. and Raizada, R.D.S. (2000). Contrast-sensitive perceptual grouping and object-based attention in the laminar circuits of primary visual cortex. Vision Research, 40, 1413-1432. [70] Grossberg, S. and Schmajuk, N.A. Neural dynamics of attentional modulated Pavlovian conditioning: Conditioned reinforcement, inhibition, and opponent processing. Psychobiology, 15, 195-240. Reprinted in Grossberg, S. (1988). Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press. Reprinted in Grossberg, S. (1988). Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press. [71] Grossberg, S. and Somers, D. (1991). Synchronized oscillations during cooperative feature linking in a cortical model of visual perception. Neural Networks, 4, 453-466. [72] Grossberg, S. and Stone, G. (1986). Neural dynamics of attention switching and temporal order information in short term memory. Memory and Cognition, 14, 451-468. Reprinted in Grossberg, S. (1988). Neural Networks and Natural Intelligence. Cambridge, MA: MIT Press. [73] Grossberg, S. and Stone, G. (1986). Neural dynamics of word recognition and recall: Attentional priming, learning, and resonance. Psychological Review, 93, 46-74. Reprinted in S. Grossberg (1987). The Adaptive Brain, Vol. II. Amsterdam: Elsevier Science. [74] Grossberg, S. and Williamson, J.R. (1999). A self-organizing neural system for learning to recognize textured scenes. Vision Research, 39, 1385-1406. [75] Grossberg, S. and Williamson, J.R. (2001). A neural model of how horizontal and interlaminar connections of visual cortex develop into adult circuits that carry out perceptual grouping and learning. Cerebral Cortex, 11, 37-58 [76] Grunewald, A. and Grossberg, S. (1998). Self-organization of binocular disparity tuning by reciprocal corticogeniculate interactions. Journal of Cognitive Neuroscience, 10, 100-215. [77] Olson, S.J. and Grossberg, S. (1998). A neural network model for the development of simple and complex cell receptive fields within cortical maps of orientation and ocular dominance. Neural Networks, 11, 189-208. [78] Raizada, R.D.S. and Grossberg, S. (2001). Context-sensitive binding by the laminar circuits of V1 and V2: A unified model of perceptual groupiing, attention, and orientation contrast. Visual Cognition, 8, 431-466. [79] Raizada, R. and Grossberg, S. (2003). Towards a Theory of the Laminar Architecture of Cerebral Cortex: Computational Clues from the Visual System. Cerebral Cortex, 13, 100-113.

Adaptive Data Based Modelling and Estimation with Application to Real Time Vehicular Collision Avoidance Chris J. Harris Department of Electronics and Computer Science, University of Southampton Highfield, Southampton SO17 1BJ, UK [email protected]

Abstract The majority of control and estimation algorithms are based upon linear time invariant models of the process, yet many dynamic processes are nonlinear, stochastic and non-stationary. In this presentation an online data based modelling and estimation approach is described which produces parsimonious dynamic models, which are transparent and appropriate for control and estimation applications. These models are linear in the adjustable parameters – hence are provable, real time and transparent but exponential in the input space dimension. Several approaches are introduced – including automatic structure algorithms to reduce the inherent curse of dimensionality of the approach. The resultant algorithms can be interpreted in rule based form and therefore offer considerable transparency to the user as to the underlying dynamics, equally the user can control the resultant rule base during learning. These algorithms will be applied to (a) helicopter flight control (b) auto-car driving and (c) multiple ship guidance and control.

References [1] [2] [3]

Harris C.J., Hong X, Gan Q. Adaptive Modelling Estimation and Fusion from Data. Springer Verlag, Berlin (2002) Chen S., Hong D., Harris C.J. Sparse multioutput rbf network construction using combined locally regularised OLS & D Optimality. IEE Proc. CTA. Vol. 150 (2) (March 2002) pp.139-146 Hong X., Harris C.J., Chen S. Robust neurofuzzy model knowledge extraction. To appear Trans. IEEE SMC (2003)

Biography Professor Harris has degrees from the Universities of Southampton, Leicester and Oxford. He is a Fellow of the Royal Academy of Engineering, Honorary Professor at University of Hong Kong; Holder of the 2001 Faraday Medal and the 1998 IEE Senior Achievement Medal for research into Nonlinear Signal Processing. Author of V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 13-14, 2003.  Springer-Verlag Berlin Heidelberg 2003

14

Chris J. Harris

7 research books in nonlinear adaptive systems and control theory and over 300 learned papers. Currently he is Emeritus Professor of Computational Intelligence at Southampton University and Director of the UK – MOD Defence Technology Centre in Data and Information Fusion – a £10M a year initiative involving three companies and eight leading UK Universities, supporting over 70 researchers.

Creating a Smart Virtual Personality Nadia Magnenat-Thalmann MIRALab - University of Geneva 24, Rue General-Dufour,1211 Geneva, Switzerland [email protected] www.miralab.ch

As people are getting more and more dependent on using the computer for a variety of tasks, providing interfaces that are intelligent and easy to interact with, has become an important issue in computer science. Current research in computer graphics, artificial intelligence and cognitive psychology aims to give computers a more human face, so that interacting with computers becomes more like interacting with humans. With the emergence of 3D graphics, we are now able to create very believable 3D characters that can move and talk. However, the behaviour that is expressed by such characters is far from believable in a lot of systems. We feel that these characters lack individuality. This talk addresses some of the aspects of creating 3D virtual characters that have a form of individuality, driven by state-of-the-art personality and emotion models (see Figure 1). 3D characters will be personalized not only on an expressive level (for example generating emotion expressions on a 3D face for example), but also on an interactive level (response generation that is coherent with the personality and emotional state of the character) and a perceptive level (having an emotional reaction to the user and her/his environment). In order to create a smart virtual personality, we need to concentrate on several different research topics. Firstly, the simulation of personality and emotional state requires an extensive psychological research of how real humans act/react emotionally to events in their surroundings, as well as an investigation into which independent factors cause humans to act/react in a certain way (also called the personality). Once this has been determined, we need to investigate whether or not such models are suitable for computer simulations, and if so, we have to define a concrete form of these notions and how they interact. Secondly, the response generation mechanism used by a virtual character needs to take personality and emotions into account. It is crucial to find generic constructs for linking personality and emotions with a response generation system (which can be anything from rule-based pattern matching systems to fullscale logical reasoning engines). And finally, the expressive virtual character should have speech capabilities and face and body animation. The animation should be controlled by highlevel parameters such as facial expressions and gestures. Furthermore, the bodies and faces of virtual characters should be easily exchangeable and animation sequences should be character-independent (an example of this is face and body representation and animation using the MPEG-4 standard).

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 15–16, 2003. c Springer-Verlag Berlin Heidelberg 2003


16

Nadia Magnenat-Thalmann

Fig. 1. 3D virtual character overview. From the user’s behaviour, the system determines the impact on its emotional state. This information is then used (together with semantic information of the user’s behaviour) to generate an appropriate response, which is expressed through a 3D virtual character linked with a text-to-speech system

Biography Prof. Nadia Magnenat-Thalmann has pioneered research into virtual humans over the last 20 years. She obtained several Bachelor’s and Master’s degrees in various disciplines and a PhD in Quantum Physics from the University of Geneva. From 1977 to 1989, she was a Professor at the University of Montreal in Canada. She moved to the University of Geneva in 1989, where she founded MIRALab. She has received several scientific and artistic awards for her work in Canada and in Europe. In l997, she has been elected to the Swiss Academy of Technical Sciences, and more recently, she was nominated as a Swiss personality who has contributed to the advance of science in the 150 years history CD-ROM produced by the Swiss Confederation Parliament, 1998, Bern, Switzerland. She has been invited to give hundreds of lectures on various topics, all related to virtual humans. Author and coauthor of a very high number of research papers and books, she has directed and produced several films and real-time mixed reality shows, among the latest are CYBERDANCE (l998), FASHION DREAMS (1999) and the UTOPIANS (2001). She is editor-in-chief of the Visual Computer Journal published by Springer Verlag and editor of several other research journals.

Intelligent Navigation on the Mobile Internet Barry Smyth1,2 1

1

Smart Media Institute, University College Dublin, Dublin, Ireland 2 ChangingWorlds, South County Business Park Leopardstown Road, Dublin, Ireland [email protected]

Summary

For many users the Mobile Internet means accessing information services through their mobile handsets - accessing mobile portals via WAP phones, for example. In this context, the Mobile Internet represents both a dramatic step forward and a significant step backward, from an information access standpoint. While it offers users greater access to valuable information and services “on the move”, mobile handsets are hardly the ideal access device in terms of their screen-size and input capabilities. As a result, mobile portal users are often frustrated by how difficult it is to quickly access the right information at the right time. A mobile portal is typically structured as a hierarchical set of navigation (menu) pages leading to distinct content pages (see Fig. 1). An ideal portal should present a user with relevant content without the need for spurious navigation. However, the reality is far from ideal. Studies highlight that while the average user expects to be able to access content within 30 seconds, the reality is closer to 150 seconds [1]. WAP navigation effort can be modelled as click-distance ([5, 4]) - the number of menu selections and scrolls needed to locate a content item. Our studies indicate that, ideally, content should be positioned no more than 10-12 ‘clicks’ from the portal home page. However, we have also found that many portals suffer from average click-distances (home page to content items) in excess of 20 [3] (see Fig. 1). Personalization research seeks to develop techniques for learning and exploiting user preferences to deliver the right content to the right user at the right time (see [2]). We have shown how personalization techniques for adapting the navigation structure of a portal can reduce click-distance and thus radically reduce navigation effort and improve portal usability ([5]). Personalization technology has led to the development of the ClixSmart NavigatorT M product-suite, developed by ChangingWorlds Ltd. (www.changingworlds.com). ClixSmart Navigator has been deployed successfully with some of Europe’s leading mobile operators. The result: users are able to locate information and services more efficiently through their mobile handsets and this in turn has led to significant increases in portal usage. In fact live-user studies, encompassing thousands of users, indicate usage increases in excess of 30% and dramatic improvements in the user’s online experience ([5, 4]).

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 17–18, 2003. c Springer-Verlag Berlin Heidelberg 2003


18

Barry Smyth

[News and Weather] [Sport] [Business] [Entertainment]

[TV/Video] [Horoscopes] [Lottery Results] [Cinema] [Cinema]

[Cinema Times] [Cinema Booking] [Flix]

Options

Options

Options

Back

[Derry] [Donegal] [Down] [Dublin] [Dublin]

Options

Back

Back

[Meeting House Square] [Omniplex ] [Ormonde ]

Back

Options

Back

Fig. 1. To access their local cinema (the Ormonde) this user must engage in an extended sequence of navigation actions; 16 clicks in total (selects and scrolls) are need in this example

2

Biography

Prof. Barry Smyth is the Digital Chair of Computer Science at University College Dublin, Ireland. His research interests cover many aspects of artificial intelligence, case-based reasoning and personalization. Barry’s research has led to the development of ChangingWorlds Ltd, which develops AI-based portal software for mobile operators, including Vodafone and O2 . He has published widely and received a number of international awards for his basic and applied research, including best paper awards at the International Joint Conference on Artificial Intelligence (IJCAI) and the European Conference on Artificial Intelligence Prestigious Applications of Intelligent Systems (ECAI - PAIS).

References [1] M. Ramsey and J. Nielsen. The WAP Usability Report. Neilsen Norman Group, 2000. 17 [2] D. Reiken. Special issue on personalization. Communications of the ACM, 43(8), 2000. 17 [3] B. Smyth. The Plight of the Mobile Navigator. MobileMetrix, 2002. 17 [4] B. Smyth and P. Cotter. Personalized Adaptive Navigation for Mobile Portals. In Proceedings of the 15th European Conference on Artificial Intelligence - Prestigious Applications of Artificial Intelligence. IOS Press, 2002. 17 [5] B. Smyth and P. Cotter. The Plight of the Navigator: Solving the Navigation Problem for Wireless Portals. In Proceedings of the 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH’02), pages 328– 337. Springer-Verlag, 2002. 17

The Evolution of Evolutionary Computation Xin Yao School of Computer Science, The University of Birmingham Edgbaston, Birmingham B15 2TT, UK [email protected] http://www.cercia.ac.uk

Abstract. Evolutionary computation has enjoyed a tremendous growth for at least a decade in both its theoretical foundations and industrial applications. Its scope has gone far beyond binary string optimisation using a simple genetic algorithm. Many research topics in evolutionary computation nowadays are not necessarily “genetic” or “evolutionary” in any biological sense. This talk will describe some recent research efforts in addressing several fundamental as well as more applied issues in evolutionary computation. Links with traditional computer science and artificial intelligence will be explored whenever appropriate.

Evolutionary Algorithms as Generate-and-Test Evolutionary algorithms (EAs) can be regarded as population-based stochastic generate-and-test [1, 2]. The advantage of formulating EAs as a generate-and-test algorithm is that the relationships between EAs and other search algorithms, such as simulated annealing (SA), tabu search (TS), hill-climbing, etc., can be made clearer and thus easier to explore and understand. Under the framework of generate-andtest, different search algorithms investigated in artificial intelligence, operations research, computer science, and evolutionary computation can be unified. Computational Time Complexity of Evolutionary Algorithms Most work in evolutionary computation has been experimental. Although there have been many reported results of EAs algorithms in solving difficult optimisation problems, the theoretical results on EA’s average computation time have been few. It is unclear theoretically what and where the real power of EAs is. It is also unclear theoretically what role a population plays in EAs. Some recent work has started tackling several fundamental issues in evolutionary computation, such as the conditions under which an EA will exhibit polynomial/exponential time behaviours [3, 4], the conditions under which a population can make a difference in terms of complexity classes [5], and the analytical tools and frameworks that facilitate the analysis of EA’s average computation time [6]. Two Heads Are Better than One Although one of the key features of evolutionary computation is a population, most work did not actually exploit this. We can show through a number of examples that exploiting population V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 19–21, 2003. c Springer-Verlag Berlin Heidelberg 2003


20

Xin Yao

information, rather than just the best individual, can lead to many benefits in evolutionary learning, e.g., improved generalisation ability [7, 8, 9] and better fault-tolerance [10].

References [1] X. Yao, “An overview of evolutionary computation,” Chinese Journal of Advanced Software Research (Allerton Press, Inc., New York, NY 10011), vol. 3, no. 1, pp. 12–29, 1996. 19 [2] X. Yao, ed., Evolutionary Computation: Theory and Applications. Singapore: World Scientific Publishing Co., 1999. 19 [3] J. He and X. Yao, “Drift analysis and average time complexity of evolutionary algorithms,” Artificial Intelligence, vol. 127, pp. 57–85, March 2001. 19 [4] J. He and X. Yao, “Erratum to: Drift analysis and average time complexity of evolutionary algorithms: [artificial intelligence 127 (2001) 57-85],” Artificial Intelligence, vol. 140, pp. 245–248, September 2002. 19 [5] J. He and X. Yao, “From an individual to a population: An analysis of the first hitting time of population-based evolutionary algorithms,” IEEE Transactions on Evolutionary Computation, vol. 6, pp. 495–511, October 2002. 19 [6] J. He and X. Yao, “Towards an analytic framework for analysing the computation time of evolutionary algorithms,” Artificial Intelligence, vol. 145, pp. 59–97, April 2003. 19 [7] P. J. Darwen and X. Yao, “Speciation as automatic categorical modularization,” IEEE Transactions on Evolutionary Computation, vol. 1, no. 2, pp. 101–108, 1997. 20 [8] X. Yao and Y. Liu, “Making use of population information in evolutionary artificial neural networks,” IEEE Trans. on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 28, no. 3, pp. 417–425, 1998. 20 [9] Y. Liu, X. Yao, and T. Higuchi, “Evolutionary ensembles with negative correlation learning,” IEEE Transactions on Evolutionary Computation, vol. 4, pp. 380–387, November 2000. 20 [10] T. Schnier and X. Yao, “Using negative correlation to evolve fault-tolerant circuits,” in Proceedings of the 5th International Conference on Evolvable Systems (ICES-2003), LNCS 2606, Springer, Germany, March 2003. 20

Speaker Bio Xin Yao is a professor of computer science and Director of the Centre of Excellence for Research in Computational Intelligence and Applications (CERCIA) at the University of Birmingham, UK. He is also a visiting professor of the University College, the University of New South Wales, the Australian Defence Force Academy, Canberra, the University of Science and Technology of China, Hefei, the Nanjing University of Aeronautics and Astronautics, Nanjing, and the Northeast Normal University, Changchun. He is an IEEE fellow, the editor in chief of IEEE Transactions on Evolutionary Computation, an associate editor or an editorial board member of five other international journals, and the chair of IEEE NNS Technical Committee on Evolutionary Computation. He is the recipient of the 2001 IEEE Donald G. Fink Prize Paper Award and has given more than 20 invited keynote/plenary speeches at various conferences. His major

The Evolution of Evolutionary Computation

21

research interests include evolutionary computation, neural network ensembles, global optimization, computational time complexity and data mining.

A Unified Model Maintains Knowledge Base Integrity John Debenham University of Technology, Sydney, Faculty of Information Technology, PO Box 123, NSW 2007, Australia [email protected]

Abstract. A knowledge base is maintained by modifying its conceptual model and by using those modifications to specify changes to its implementation. The maintenance problem is to determine which parts of that model should be checked for correctness in response a change in the application. The maintenance problem is not computable for first-order knowledge bases. Two things in the conceptual model are joined by a maintenance link if a modification to one of them means that the other must be checked for correctness, and so possibly modified, if consistency of the model is to be preserved. In a unified conceptual model for first-order knowledge bases the data and knowledge are modelled formally in a uniform way. A characterisation is given of four different kinds of maintenance links in a unified conceptual model. Two of these four kinds of maintenance links can be removed by transforming the conceptual model. In this way the maintenance problem is simplified.

1

Introduction

Maintenance links join two things in the conceptual model if a modification to one of them means that the other must be checked for correctness, and so possibly modified, if consistency of that model is to be preserved. If that other thing requires modification then the links from it to yet other things must be followed, and so on until things are reached that do not require modification. If node A is linked to node B which is linked to node C then nodes A and C are indirectly linked. In a coherent knowledge base everything is indirectly linked to everything else. A good conceptual model for maintenance will have a low density of maintenance links [1]. The set of maintenance links should be minimal in than none may be removed. Informally, one conceptual model is “better” than another if it leads to less checking for correctness. The aim of this work is to generate a good conceptual model. A classification into four classes is given here of the maintenance links for conceptual models expressed in the unified [2] knowledge representation. Methods are given for removing two of these classes of link so reducing the density of maintenance links. Approaches to the maintenance of knowledge bases are principally of two types [3]. First, approaches that take the knowledge base as presented and then try to control the maintenance process [4]. Second, approaches that engineer a model of the knowledge base so that it is in a form that is inherently easy to maintain [5] [6]. The approach described here is of the second type.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 21-27, 2003. c Springer-Verlag Berlin Heidelberg 2003

22

John Debenham

The terms data, information and knowledge are used here in the following sense. The data things in an application are the fundamental, indivisible things. Data things can be represented as simple constants or variables. If an association between things cannot be defined as a succinct, computable rule then it is an implicit association. Otherwise it is an explicit association. An information thing in an application is an implicit association between data things. Information things can be represented as tuples or relations. A knowledge thing in an application is an explicit association between information and/or data things. Knowledge can be represented either as programs in an imperative language or as rules in a declarative language.

2

Conceptual Model

Items are a formalism for describing all data, information and knowledge things in an application [7]. Items incorporate two classes of constraints, and a single rule of decomposition is specified for items. The key to this unified representation is the way in which the “meaning” of an item, called its semantics, is specified. The semantics of an item is a function that recognises the members of the “value set” of that item. The value set of an item will change in time τ, but the item’s semantics should remain constant. The value set of an information item at a certain time τ is the set of tuples that are associated with a relational implementation of that item at that time. Knowledge items have value sets too. Consider the rule “the sale price of parts is the cost price marked up by a universal mark-up factor”; this rule is represented by the item named [part/sale-price, part/cost-price, mark- u p ] with a value set of corresponding quintuples. The idea of defining the semantics of items as recognising functions for the members of their value set extends to complex, recursive knowledge items. An item is a named triple A[ SA, VA, CA ] with item name A, SA is called the item semantics of A, VA is called the item value constraints of A and CA is called the item set constraints of A. The item semantics, SA , is a λ-calculus expression that recognises the members of the value set of item A. The expression for an item’s semantics may contain the semantics of other items {A1 ,..., An } called that item’s components: 1 n 1 ) º......º S (yn ,...,y n ) º J(y1 ,..,y 1 ,..,yn ) ]• λ y11 ...y m ...ym •[ SA (y11,...,y m A 1 m 1 m m 1

n

1

1

n

n

1

n

The item value constraints, VA, is a λ-calculus expression: 1 n 1 ) º......º V (yn ,...,y n ) º K(y1 ,..,y 1 ,..,yn ) ]• λ y11 ...y m ...ym •[ V A (y11 ,...,y m An 1 mn 1 m1 mn 1 1 n 1 that should be satisfied by the members of the value set of item A as they change in time; so if a tuple satisfies SA then it should satisfy VA [8]. The expression for an item’s value constraints contains the value constraints of that item’s components. The item set constraints, CA, is an expression of the form: C A º CA º...º CA º (L)A 1

2

n

where L is a logical combination of: • Card lies in some numerical range; • Uni(Ai) for some i, 1 ≤ i ≤ n, and

A Unified Model Maintains Knowledge Base Integrity

23

• Can(A i , X) for some i, 1 ≤ i ≤ n, where X is a non-empty subset of {A1 ,..., An } - {Ai }; subscripted with the name of the item A, “Uni(a)” means that “all members of the value set of item a must be in this association”. “Can(b, A)” means that “the value set of the set of items A functionally determines the value set of item b”. “Card” means “the number of things in the value set”. The subscripts indicate the item’s components to which that set constraint applies. For example, each part may be associated with a cost-price subject to the “value constraint” that parts whose part-number is less that 1,999 should be associated with a cost price of no more than $300. The information item named part/cost-price then is: part/cost-price[ λxy•[ Spart(x) º Scost-price (y) º costs(x, y) ]•, λxy•[ Vpart(x) º V cost-price(y) º ((x < 1999) 8 (y ≤ 300)) ]•, C part º Ccost-price º (Uni(part) º Can( cost-price, {part})part/cost-price ] Rules, or knowledge, can also be defined as items, although it is neater to define knowledge items using “objects”. “Objects” are item building operators. The knowledge item [part/sale-price, part/cost-price, mark-up] which means “the sale price of parts is the cost price marked up by a uniform markup factor” is: [part/sale-price, part/cost-price, mark-up][ λx1x2y1y2z•[(Spart/sale-price(x1, x2) º Spart/cost -price(y1, y2) º Smark-up (z) ) º ((x 1 = y 1) 8 (x2 = z _ y2))]•, λx1x2y1y2z•[Vpart/sale-price(x1, x2) º V part/cost-price(y1, y2) ºVmark-up (z) ) º (( x1 = y1 ) 8 ( x2 > y2 ))]•, C [part/sale-price, part/cost-price, mark-up] ] Two different items can share common knowledge and so can lead to a profusion of maintenance links. This problem can be avoided by using objects. An n-adic object is an operator that maps n given items into another item for some value of n. Further, the definition of each object will presume that the set of items to which that object may be applied are of a specific “type”. The type of an m-adic item is determined both by whether it is a data item, an information item or a knowledge item and by the value of m. The type is denoted respectively by Dm, Im and Km. Items may also have unspecified, or free, type which is denoted by X m . The formal definition of an object is similar to that of an item. An object named A is a typed triple A[E,F,G] where E is a typed expression called the semantics of A, F is a typed expression called the value constraints of A and G is a typed expression called the set constraints of A. For example, the part/cost-price item can be built from the items part and cost-price using the costs operator: part/cost-price = costs(part, cost-price) costs[λP:X1Q:X1•λxy•[ SP(x) º SQ(y) º costs(x,y) ]••, λP:X1Q:X1•λxy•[VP(x) º V Q(y) º ((1000 < x < 1999) 8 (y ≤ 300)) ]••,

λP:X1Q:X1•[ CP º CQ º (Uni(P) º Can(Q, {P})) n(costs,P,Q) ]•] where n(costs, P, Q) is the name of the item costs(P, Q). Objects also represent value constraints and set constraints in a uniform way. A decomposition operation for objects is defined in [2].

24

John Debenham

A conceptual model consists of a set of items and a set of maintenance links. The items are constructed by applying a set of object operators to a set of fundamental items called the basis. The maintenance links join two items if modification to one of them necessarily means that the other item has at least to be checked for correctness if consistency is to be preserved. Item join provides the basis for item decomposition [7]. Given items A and B, the item with name A ⊗E B is called the join of A and B on E, where E is a set of components common to both A and B. Using the rule of composition ⊗, knowledge items, information items and data items may be joined with one another regardless of type. For example, the knowledge item: [cost-price, tax] [λxy•[Scost-price(x) º Stax (y) º x = y

_ 0.05]•,

λxy•[V cost-price(x) º V tax(y) º x < y]•, C[cost -price, tax] ]

can be joined with the information item part/cost-price on the set {cost-price} to give the information item part/cost-price/tax. In other words: [cost-price, tax] ⊗{cost-price} part/cost-price =

part/cost-price/tax[ λxyz•[ Spart(x) º Scost-price (x) º Stax (y) º costs(x,y) º z = y _ 0.05 ]•, λxyz•[ Vpart(x) º V cost-price(x) º V tax(y) º ((1000<x<1999) 8 (0
In this way items may be joined together to form more complex items. The ⊗ operator also forms the basis of a theory of decomposition in which each item is replaced by a set of simpler items. An item I is decomposable into the set of items D = {I1, I2,..., In} if: Ii has non-trivial semantics for all i, I = I1 ⊗ I2 ⊗ ... ⊗ In , where each join is monotonic; that is, each term in this composition contributes at least one component to I. If item I is decomposable then it will not necessarily have a unique decomposition. The ⊗ operator is applied to objects in a similar way [7]. The rule of decomposition is: “Given a conceptual model discard any items and objects which are decomposable”. For example, this rule requires that the item part/cost-price/tax should be discarded in favour of the two items [cost-price, tax] and part/cost-price.

3

Maintenance Links

A maintenance link joins two items in the conceptual model if modification of one item means that the other item must be checked for correctness, and maybe modified, if the consistency of the conceptual model is to be preserved [9]. The number of maintenance links can be very large. So maintenance links can only form the basis of a practical approach to knowledge base maintenance if there is some way of reducing their density on the conceptual model. For example, given two items A and B, where both are n-adic items with semantics SA and SB respectively, if π is permutation such that:

A Unified Model Maintains Knowledge Base Integrity

25

(: x1x2...x n)[ SA(x1,x 2 ,...,x n ) * SB(π(x1,x 2 ,...,x n )) ] then item B is a sub-item of item A. These two items should be joined with a maintenance link. If A and B are both data items then B is a sub-type of A. Suppose that: X = E D; where D = C A B (1) for items X, D, A and B and objects E and C. Item X is a sub-item of item D. Object E has the effect of extracting a sub-set of the value set of item D to form the value set of item X. Item D is formed from items A and B using object C. Introduce two new objects F and J. Suppose that object F when applied to item A extracts the same subset of item A’s value set as E extracted from the “left-side” (ie. the “A-side”) of D. Likewise J extracts the same subset of B’s value set as E extracted from D. Then: X = C G K; where G = F A and K = J B (2) so G is a sub-item of A, and K is a sub-item of B. The form (2) differs from (1) in that the sub-item maintenance links have been moved one layer closer to the data item layer, and object C has moved one layer away from the data item layer. Using this method repeatedly sub-item maintenance links between non-data items are reduced to sub-type links between data items. It is shown now that there are four kinds of maintenance link in a conceptual model built using the unified knowledge representation. Consider two items A and B, and suppose that their semantics SA and SB have the form: 1 ...y p •[S (y1,...,y 1 ) º .. º S (yp,...,y p ) º J(y1 ,..,y 1 ,..,y p )]• S A = λy11...y m Ap 1 mp A1 1 m1 mp 1 m1 mp 1 1 1 q 1 1 q q 1 1 q S B = λy1 ...y n ...yn •[SB (y1 ,...,y n ) º .. º SB (y1 ,...,yn ) º K(y 1 ,..,yn ,..,yn )]• 1

q

1

1

q

q

1

q

S A contains (p + 1) terms and SB contains (q + 1) terms. Let µ be a maximal subexpression of SA ⊗ B such that: both SA 8 µ and SB 8 µ (a) where µ has the form: λy11...y 1d ...ydr •[SC (y 11 ,...,y 1d ) º ...... º SC (y1r ,...,y dr ) º L(y 11 ,..,y1d ,..,ydr )]• 1

r

1

1

r

r

1

r

If µ is empty, ie. ‘false’, then the semantics of A and B are independent. If µ is nonempty then the semantics of A and B have something in common and A and B should be joined with a maintenance link [10]. Now examine µ to see why A and B should be joined. If µ is non-empty and if both A and B are items in the basis then: A and B are a pair of basis items with logically dependent semantics (b) If µ is non-empty and if A is not in the basis then there are three cases. First, if: S A ° SB ° µ (c) then items A and B are equivalent and should be joined with an equivalence link. Second if (c) does not hold and: either SA ° µ or S B ° µ (d) then either A is a sub-item of B, or B is a sub-item of A and these two items should be joined with a sub-item link. Third, if (c) and (d) do not hold then if ∆ is a minimal sub-expression of SA such that ∆ 8 µ. Then: either SA ( y j ,...,y j ) › ∆, for some j 1 mj j

(e)

26

John Debenham

or J( y1 ,..,y 1 ,...,y p ) › ∆ 1

mj

(f)

mp

Both (e) and (f) may hold. If (e) holds then items A and B share one or more component items to which they should each be joined with a component link. If (f) holds then items A and B may be constructed with two object operators whose respective semantics are logically dependent. Suppose that item A was constructed by object operator C then the semantics of C will imply:

F = λQ1 :Xi11 Q2:Xi22 ...Qj:Xijj •λy11 ...y1d ...ydr •[ SP (y11 ,...,y1d ) º ..... º 1

r

1

1

S P (y1r ,...,ydr ) º L(y11 ,..,y 1d ,..,ydr )]• r r 1 r where the Qi ’s take care of any possible duplication in the Pj ’s. Let E be the object E[ F, T, Á] then C is a sub-object of E; that is, there exists a non-tautological object F such that: C ,w E ⊗M F (g) for some set M and where the join is not necessarily monotonic. Items A and B are weakly equivalent, written A ,w B, if there exists a permutation π such that: (: x1 x2 ...x n )[SA(x1,x 2 ,...,x n ) ° SB(π(x1,x 2 ,...,x n ))] where the x i are the ni variables associated with the i’th component of A. If A is a sub-item of B and if B is a sub-item of A then items A and B are weakly equivalent. If (g) holds then the maintenance links are of three different kinds. If the join in (g) is monotonic then (g) states that C may be decomposed into E and F. If the join in (g) is not monotonic then (g) states that either C ,w E or C ,w F. So, if the join in (g) is not monotonic then either E will be weakly equivalent to C, or C will be a sub-object of E. It has been shown above that sub-item links between non-data items may be reduced to sub-type links between data items. So if: • the semantics of the items in the basis are all logically independent; • all equivalent items and objects have been removed by re-naming, and • sub-item links between non-data items have been reduced to sub-type links between data items then the maintenance links will be between nodes marked with: • a data item that is a sub-type of the data item marked on another node, these are called the sub-type links; • an item and the nodes marked with that item’s components, these are called the component links, and • an item constructed by a decomposable object and nodes constructed with that object’s decomposition, these are called the duplicate links. If the objects employed to construct the conceptual model have been decomposed then the only maintenance links remaining will be the sub-type links and the component links. The sub-type links and the component links cannot be removed from the conceptual model. Unfortunately, decomposable objects, and so too duplicate links, are hard to detect [11]. Suppose that objects A and B are decomposable as follows: A ,w E ⊗M F

A Unified Model Maintains Knowledge Base Integrity

27

B ,w E ⊗M G Then objects A and B should both be linked to object E. If the decompositions of A and B have not been identified then object E may not have been identified and the implicit link between objects A and B may not be identified.

4

Conclusions

Potential maintenance hazards caused by one chunk of knowledge being hidden within another that plague rule-based formalisms have been partially avoided by using a unified representation based on items. Items make it difficult to analyse the structure of a whole application. To make the structure clear, ‘objects’ are introduced as item building operators. Objects are an abstraction representing the structure of knowledge. The use of objects to build items enables the hidden links in the knowledge to be identified and made explicit. A single decomposition operation for objects enables some of these links to be removed thus simplifying maintenance.

References 1

Mayol, E. and Teniente, E. (1999). “Addressing Efficiency Issues During the Process of Integrity Maintenance” in proceedings Tenth International Conference DEXA99, Florence, September 1999, pp270-281. 2 Debenham, J.K. “Validity of First-Order Knowledge Bases”, in proceedings 14th International FLAIRS Conference FLAIRS-2001, Key West, Florida, 21-23 May 2001, pp217-221. 3 Katsuno, H. and Mendelzon, A.O. “On the Difference between Updating a Knowledge Base and Revising It”, in proceedings Second International Conference on Principles of Knowledge Representation and Reasoning, KR’91, Morgan Kaufmann, 1991. 4 Barr, V. “Applying Reliability Engineering to Expert Systems” in proceedings 12th International FLAIRS Conference, Florida, May 1999, pp494-498. 5 Jantke, K.P. and Herrmann, J. “Lattices of Knowledge in Intelligent Systems Validation” in proceedings 12th International FLAIRS Conference, Florida, May 1999, pp499-505. 6 Darwiche, A. (1999). “Compiling Knowledge into Decomposable Negation Normal Form” in proceedings International Joint Conference on Artificial Intelligence, IJCAI’99, Stockholm, Sweden, August 1999, pp 284-289. 7 Debenham, J.K. “Knowledge Engineering”, Springer-Verlag, 1998. 8 Johnson, G. and Santos, E. (2000). “Generalizing Knowledge Representation Rules for Acquiring and Validating Uncertain Knowledge” in proceedings 13th International FLAIRS Conference, Florida, May 2000, pp186-2191. 9 Ramirez, J. and de Antonio, A. (2000) “Semantic Verification of Rule-Based Systems with Arithmetic Constraints” in proceedings 11th International Conference DEXA2000, London, September 2000, pp437-446. 10 Baral, C. and Zhang, Y. “On the Semantics of Knowledge Update” in proceedings Seventeeth International Joint Conference on Artificial Intelligence [IJCAI-01], Seattle, Washington, August 2001, pp97-102. 11 Roger, M., Simonet, A and Simonet, M. (2002). “Bringing together Description Logics and Database in an Object Oriented Model” in proceedings thriteenth Internatinal Conference DEXA 2002, Aix-en-Provence, September 2002, pp504-513.

Formal Argumentation Frameworks for the Extended Generalized Annotated Logic Programs Takehisa Takahashi, Yuichi Umeda, and Hajime Sawamura Department of Information Engineering and Graduate School of Science and Technology Niigata University 8050, Ninocho, Ikarashi, Niigata, 950-2181 Japan {takehisa,umeda,sawamura}@cs.ie.niigata-u.ac.jp http://www.cs.ie.niigata-u.ac.jp/

Abstract. Argumentation is an important form and way of interaction which is considered one of the most essential issues for agent systems as well as for humans. So far, a number of argumentation models have been proposed, in particular for the extended logic programs as knowledge representation. In this paper, we further pursuit, along the same lines, the basic argumentation framework for much more expressive logic programs called the extended generalized annotated logic programs (EGAP). We provide the semantics and dialectical proof theory for it, and prove the soundness and completeness and the equivalence to the well-founded semantics. Then, we develop the basic argumentation framework to that for multi-agents. Argument examples are described in illustration of these results.

1

Introduction

Argumentation is a ubiquitous form and way of dialogue in the human society. Recently in the fields of artificial intelligence and computer science, there has been a growing interest in argumentation as an effective means of interaction for intelligent systems [1], as a general framework for relating nonmonotonic logics of different styles [3] and so on. So far, a number of argumentation frameworks have been proposed [1]. The underlying frameworks are mainly built using the normal logic programs or the extended logic programs (ELP) as knowledge representation languages (e. g., [6]). In this paper, we are concerned with how arguments should be made under incomplete, uncertain, inconsistent or vague knowledge bases. For this, we further pursuit more versatile argumentation framework using much more expressive logic programs called the extended generalized annotated logic programs (EGAP) that has two kinds of negations (default and explicit ones [3]) and are founded on many-valuedness. In Section 2, we define the well-founded semantics for EGAP under the general semantics for the generalized annotated logic programs [4]. In Section 3, V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 28–38, 2003. c Springer-Verlag Berlin Heidelberg 2003


Formal Argumentation Frameworks for EGAP

29

we prepare an abstract argumentation framework of which soundness and completeness are proved. This section is a preparation for the following Section 4 and 5. Section 4 describes the basic argumentation framework which deals with argumentation within a single knowledge base. We give the definitions of arguments and attack relation which are proper to EGAP, and prove the equivalence of the argumentation semantics of the basic argumentation framework to the well-founded semantics of Section 2. In Section 5, we further develop the basic argumentation framework to that for multi-agents, in which the new attack relation is introduced so as to reflect a skeptical view of agents to other knowledge bases.

2

Language and Semantics

We define Generalized Annotated Logic Programs with default negation (EGAP) as follows. Definition 1. (Extended Generalized Annotated Logic Programs [4, 8]). An extended generalized annotated logic program (EGAP ) is a set of ground instances of rules of the form A0 : µ0 ← B1 : µ1 & . . . & Bn : µn & not(Bn+1 : µn+1 ) & . . . & not(Bm : µm ) where not is the default negation symbol. A0 : µ0 and Bi : µi (1 ≤ i ≤ m) are annotated atoms, and not(Bi : µi ), (n+1 ≤ i ≤ m) are annotated default atoms1 . An EGAP with no annotated default atom coincides with a generalized annotated logic program (GAP [4]). In [4], the general and restricted semantics for GAP were given. The wellfounded semantics (WFS) for EGAP based on the restricted semantics of GAP was studied in [8] and [2]. In this section, we provide WFS for EGAP based on the general semantics of GAP along the line of the result of [8]2 . We assume a set of all ideals of a complete lattice (T , ≤) of truth values (denoted by I(T )), and define the interpretation of an EGAP P to be a mapping from the Herbrand base (Hp ) of P to I(T ) [4]. Let I be an interpretation, then the satisfaction is defined such that I |= A : µ iff µ ∈ I(A); I |= not(A : µ) iff I  |= A : µ (refer to [4] for the remaining part). The monotonic and continuous operator TP mapping interpretations into interpretations was defined for the fixpoint semantics in [4]. Definition 2. (ΓP Operator). Let P be an EGAP, and I be an interpretation. The transformation of P to a GAP by I is defined as follows: PI = {A0 : 1 2

The explicit negation “¬” is implicitly handled in this paper so that it is defined by ¬ : T → T and ¬A : µ = A : ¬µ In [2], the coherence principle (¬A : µ → not(A : µ)) was introduced, but we do not take it into account in this paper since we think that the coherence principle is not necessarily adequate to our purpose.

30

Takehisa Takahashi et al.

µ0 ← B1 : µ1 & . . . & Bn : µn | A0 : µ0 ← B1 : µ1 & . . . & Bn : µn & not(Bn+1 : µn+1 ) & . . . & not(Bm : µm ) ∈ P, and I  |= Bn+1 : µn+1 and . . . and I  |= Bm : µm }. Then, the function ΓP mapping interpretations into interpretations is defined such that ΓP (I) = lfp T P . I

ΓP is anti-monotonic, but ΓP ΓP which applies ΓP twice becomes monotonic (this can be proved in a similar way to [8]). Hence it has a least fixpoint. We write simply Γ for ΓP when P is obvious. Definition 3. (Well-Founded Semantics). Let P be a EGAP, and M be a least fixpoint of Γ Γ . Then the well-founded model of P is defined by: − WF S(P ) |= A : µ iff µ ∈ M (A); − WF S(P ) |= not(A : µ) iff µ ∈  (Γ (M ))(A). The least fixpoint of Γ Γ can be found by iteration as follow: I0 = ∆ Iα = Γ
Γ (Iα−1 ) for successor ordinal α Iλ = α<λ Iα for limit ordinal λ where ∆ is the least interpretation, i.e. for all atom A, ∆(A) = ∅ (the empty ideal). Then there exists a least ordinal λ0 such that Iλ0 is the least fixpoint of ΓΓ. Example 1. Suppose the complete lattice T of truth values is FOUR = ({⊥, t, f, }, {⊥ < t, ⊥ < f, t < , f < }), and consider the following EGAP P . arise(unbalance) : t ← large(1N ) : t & is(OA, D level) : t. arise(unbalance) : f ← not(direction(1N, horizontal) : t). large(1N ) : t ← . is(OA, D level) : t ← . direction(1N, vertical) : t ← . This program is a simple example of the rules of failure diagnosis for the rotating machinery. If M is the least fixpoint of Γ Γ , then M (arise(unbalance)) = {⊥, t, f, }, and hence ∀µ ∈ T , WF S(P ) |= arise(unbalance) : µ. WF S(P ) |= arise(unbalance) :  says that the result of the diagnosis about unbalance shows inconsistency. In the systems which treat uncertain knowledge of humans, for example, in expert systems, it is very convenient for systems to be able to directly represent inconsistent information.

3

Abstract Argumentation

Following [3, 6, 7], we introduce an abstract argumentation framework in which both arguments and attack relation on them are not specified in a concrete manner. This section is the preparation of the argumentation frameworks described in the succeeding sections.

Formal Argumentation Frameworks for EGAP

3.1

31

Acceptable and Justified Arguments

We define the argumentation semantics for EGAP as the least fixpoint of the function which collects all acceptable arguments. Definition 4. (Attack Relation). Let Args be an abstract argument set. An attack relation x on Args is a binary relation on Args, i. e., x ⊆ Args2 . Definition 5. (x/y-Acceptable and Justified Argument). Let x and y be attack relations on Args. Suppose Arg1 ∈ Args and S ⊆ Args. Then Arg1 is x/y-acceptable wrt. S if for every Arg2 ∈ Args such that (Arg2 , Arg1 ) ∈ x there exists Arg3 ∈ S such that (Arg3 , Arg2 ) ∈ y. The function FArgs,x/y mapping from P(Args) to P(Args) is defined by FArgs,x/y (S) = {Arg ∈ Args | Arg is x/y-acceptable wrt. S}. Since FArgs,x/y is monotonic, it has a least fixpoint, denoted by JArgs,x/y (we write simply Jx/y when Args is obvious ). An argument Arg is x/y-justified if Arg ∈ Jx/y ; an argument is x/y-overruled if it is attacked by an x/y-justified argument; and an argument is x/y-defensible if it is neither x/y-justified nor x/y-overruled. A least fixpoint of Fx/y can be constructed by the following iterative method[3] : 0 Jx/y =∅ α−1 α Jx/y = Fx/y (Jx/y ) for successor ordinal α  λ α for limit ordinal λ Jx/y = α<λ Jx/y λ0 λ0 Then there exists a least ordinal λ0 such that Fx/y (Jx/y ) = Jx/y = Jx/y .

3.2

Dialectical Proof Theory

Justified arguments can be dialectically determined from a set of arguments by the dialectical proof theory. We show the sound and complete dialectical proof theory for the abstract argumentation semantics JArgs,x/y . Definition 6. (x/y-Dialogue and x/y-Dialogue Tree). An x/y-dialogue is a finite nonempty sequence of moves movei = (P layeri , Argi ), (i ≥ 1) such that 1. P layeri = P (Proponent) iff i is odd; and P layeri = O (Opponent) iff i is even. = j) then Argi  = Argj . 2. If P layeri = P layerj = P (i  3. If P layeri = P (i ≥ 3) then (Argi , Argi−1 ) ∈ y; and if P layeri = O (i ≥ 2) then (Argi , Argi−1 ) ∈ x. An x/y-dialogue tree is a tree of moves such that every branch is an x/y-dialogue, and for all moves movei = (P, Argi ), the children of movei are all those moves (O, Argj ) such that (Argj , Argi ) ∈ x.

32

Takehisa Takahashi et al.

Definition 7. (Provably x/y-Justified). An x/y-dialogue D is a winning x/y-dialogue iff the termination of D is a move of proponent. An x/y-dialogue tree T is a winning x/y-dialogue tree iff every branch of T is a winning x/ydialogue. An argument Arg is a provably x/y-justified argument iff there exists a winning x/y-dialogue tree with Arg as its root. Theorem 1. Let Args be an abstract argument set. Assume an attack relation on Args is finite. Then Arg ∈ Args is Provably x/y-justified iff Arg is x/yjustified. (refer to [9] for the proof. )

4

Basic Argumentation

We describe our basic argumentation (BA) framework for EGAP based on the abstract argumentation framework. 4.1

Arguments

We extend the notion of reductant which is described in GAP [4]. Definition 8. (Reductant and Minimal Reductant). Suppose P is an EGAP, and Ci (1 ≤ i ≤ k) are rules in P of the form: i A : ρi ← B1i : µi1 & . . . & Bni i : µini & not(Bni i +1 : µini +1 ) & . . . & not(Bm : µimi ) i i For simplicity, we write Body1···n for B1i : µi1 & . . . & Bni i : µini and i i i i i : µimi ). Let not(Bodyni +1···mi ) for not(Bni +1 : µni +1 ) & . . . & not(Bm i ρ = {ρ1 , . . . , ρk }. Then, 1 k & . . . & Body1···n & A : ρ ← Body1···n 1 k k . . . & not(Bodynk +1···mk )

not(Bodyn1 1 +1···m1 ) &

is a reductant of P . If there exists no non-empty proper subset S ⊂ {ρ1 , . . . , ρk } such that ρ = S then it is a minimal reductant. Definition 9. (Arguments). Let P be an EGAP. An argument in P is a finite sequence Arg = [r1 , . . . , rn ] of minimal reductants in P such that: 1. for every i (1 ≤ i ≤ n) and for every annotated atom Aj : µj in the body of ri , there exists a minimal reductant rk such that Aj : µk (µk ≥ µj , n ≥ k > i) is head of rk . 2. There exists no proper subsequence of [r1 , . . . , rn ] which meets the first condition, and includes the minimal reductant r1 . A subargument of Arg is a subsequence of Arg which is an argument. The heads of minimal reductants in Arg are called conclusions of Arg, and the annotated default atoms in the body of minimal reductants in Arg are called assumptions of Arg. We write concl(Arg) for the set of conclusions and assm(Arg) for the set of assumptions of Arg. We denote the set of all arguments in P by ArgsP .

Formal Argumentation Frameworks for EGAP

33

Example 2. Suppose T = FOUR and P is an EGAP such that P = { recommend(I, X) : t ← f amous(X) : t & not(disreputable(X) : t), recommend(I, X) : f ← expensive(X) : t, recommend(I, X) : ⊥ ← read(I, X) : f, f amous(book A) : t ←, expensive(book A) : t ←, read(I, book A) : f ←}

where X is an object variable3 . Then, Arg = [ recommend(I, book A) :  ← f amous(book A) : t & expensive(book A) : t & not(disreputable(book A) : t), f amous(book A) : t ←, expensive(book A) : t ←

]

is an argument which says that the book A has both sides of recommendable and unrecommendable. To make an argument with the conclusion recommend (I, book A) : , we need the minimal reductant with the head in Arg. Reductants which are not minimal are unnecessary (or redundant ) in the sense that we can construct arguments with the same conclusion only from minimal reductants, and occasionally tends to be irrelevant. recommend(I, book A) : t ← f amous(book A) : t & read(I, book A) : f & not(disreputable(book A) : t)

is a reductant which is not minimal. It says “I recommend the book A because it is famous, I have never read it, and have never heard a disrepute of it”. read(I, book A) : f may be not relevant as a reason for asserting recommend (I, book A) : t. Thus the introduction of minimal reductants has a role of excluding redundant arguments with excessive ground. 4.2

Attack Relation for BA

The undercut and rebut are typical attack relations employed very often in argument systems [6, 7]. The undercut invalidates assumptions of arguments, and the rebut contradicts conclusions of arguments. However, in EGAP, there is no conflict among conclusions by paraconsistency of EGAP. To see this, let us consider T = [0, 1], EGAP P = {p : 0.3 ←, p : 0.6 ←}. Then the ideal assigned to p is [0, 0.6] in the model of P , and we have both p : 0.3 and p : 0.6 as the logical consequences. Therefore, two arguments ArgsP = {[p : 0.3 ←], [p : 0.6 ←]} can not be viewed as being in conflict with each other within one program. Therefore, we will take in to account the undercut only in BA. Definition 10. (Undercut). Arg1 undercuts Arg2 iff there exists A : µ1 ∈ concl(Arg1 ) and not(A : µ2 ) ∈ assm(Arg2 ) such that µ1 ≥ µ2 . Note that that undercut relation is not always one-directional. That is, it happens that Arg1 undercut Arg2 and vice versa. P is called f initary if an attack relation on ArgsP is finite. 3

We assume that a rule containing (object or annotation) variables represents any ground instance of it.

34

Takehisa Takahashi et al.

In BA, we use undercut/undercut-acceptable (u/u-acceptable), and define the BA semantics for an EGAP P to be JArgsP ,u/u (JP for short). u/u-justified arguments can be dialectically determined by the dialectical proof theory specified in the subsection 3.2. We omit the notation u/u when BA is obvious. 4.3

Equivalence of the BA Semantics to WFS

In ELP, the equivalence between the argumentation semantics Ju/a (a is undercut or rebut) and the well-founded semantics with explicit negation (WFSX) was shown in [7]. Similarly, in EGAP, we can show that Ju/u coincides with WFS. Definition 11. (BA Semantics). Let P be an EGAP. Then, 1. BA(P ) |= A : µ iff there exists a justified argument Arg ∈ JP such that for some ρ ≥ µ, A : ρ ∈ concl(Arg); 2. BA(P ) |= not(A : µ) iff for every arguments Arg ∈ ArgsP , if there exists ρ ≥ µ such that A : ρ ∈ concl(Arg), Arg is overruled. Theorem 2. Let P be an EGAP. Then, – WF S(P ) |= A : µ iff BA(P ) |= A : µ; – WF S(P ) |= not(A : µ) iff BA(P ) |= not(A : µ). (refer to [9] for the proof. ) Example 3. Consider T = FOUR, and let P be the following EGAP. P = { a : f ← not(b : f), a : t ← not(b : t), b : f ← not(a : t), b : f ← }.

Then, I0 Γ (I0 ) I1 = Γ Γ (I0 ) Γ (I1 ) I2 = Γ Γ (I1 )

= {a → ∅, b → ∅} = {a → {⊥, t, f, }, b → {⊥, t, f, }} = {a → ∅, b → {⊥, f}} = {a → {⊥, t}, b → {⊥, t, f, }} = {a → ∅, b → {⊥, f}} = I1

WFS(P ) |= b : ⊥, b : f, not(a : f), not(a : ). Arg2 : [a : t ← not(b : t)], ArgsP = { Arg1 : [a : f ← not(b : f)], Arg4 : [b : f ←], Arg3 : [b : t ← not(a : t)], Arg5 : [a :  ← not(b : f) & not(b : t)], Arg6 : [b :  ← not(a : t)] } JP = JP1 = {Arg4 } overruled arguments = {Arg1 , Arg5 } BA(P ) |= b : ⊥, b : f, not(a : f), not(a : ).

Thus, BA(P ) coincides with WF S(P ).

Formal Argumentation Frameworks for EGAP

5

35

Multi-agent Argumentation

In this section, we describe the multi-agent argumentation (MAA) framework for distributed knowledge bases. Like in [5], we identify each distributed EGAP with an agent with an individual view of the world, and arguments as view of agents. Definition 12. (Multi-agent System) Let each KBi (1 ≤ i ≤ n) be EGAP. Then the set KB = {KB1 , . . . , KBn } is a multi-agent system (MAS ). Our MAA is an argumentation framework in which agents argue each other about what each agent believes to be right (i.e., justified arguments for each agent in terms of BA). It is, however, not simply a distributed version of BA since each agent usually has a different recognition even for the same assertion from other agents, and hence there can be a new attack relation, differently from in BA. Let us take up an example. Suppose agent A has the confidence factors 0.5 and 0.8 about solution of some problem, depending on the different grounds. If agent B argues about it with a confidence factor 0.6, then agent B could rebut it by saying to agent A the two reasons. One is that agent A’s overall confidence is 0.8 (a maximum) according to the BA semantics, and the other is that agent A has too excessive confidence in view of B. The latter amounts to assuming sort of skepticism for the rebut relation. Note that agent A cannot rebut agent B since agent A’s overall confidence subsumes agent B’s one. Otherwise agent A would fall into self-contradiction. Taking into these considerations, we will have the following definitions. Definition 13. (Maximal Arguments). Let Args be a set of argument, and Arg be an argument in Args. We define the set of conclusions whose annotations are maximal as follows. max concl(Arg) = {A : µ ∈ concl(Arg) | for any ρ, if A : ρ ∈ concl(Arg), µ ≥ ρ}. Then Arg is called a maximal argument (margument ) if for all A : µ ∈ max concl(Arg), there is no Arg  ∈ Args such that for some ρ > µ, A : ρ ∈ max concl(Arg  ). Definition 14. (Rebut and Defeat). – Arg1 rebuts Arg2 iff there exists A : µ1 ∈ max concl(Arg1 ) and A : µ2 ∈ max concl(Arg2 ) such that µ1  ≥ µ2 . – Arg1 defeats Arg2 iff Arg1 undercuts Arg2 , or Arg1 rebuts Arg2 and Arg2 does not undercut Arg1 . Suppose KB = {KB1 , . . . , KBn } is a MAS. We denote the set of every margument in JKBi (i. e., the set JArgsKBi ,u/u  of justified arguments for KBi in BA) by mJKBi , and define ArgsKB = mJKBi . KB is called finitary if the attack relation on ArgsKB is finite. Then we define the MAA semantics for EGAP by JArgsKB ,d/u (JKB for short, and d = defeat, u = undercut). JKB also can be dialectically determined by the dialectical proof theory shown in the subsection 3.2 . We omit the notation d/u when MAA is obvious. Finally we define the semantics of MAA, similarly to BA.

36

Takehisa Takahashi et al. Arg33

m-Arguments of KB3

agree(death):(0.9, 1.0)

Arg32

desire(family, death):(0.9, 1.0)

Arg31

hate(family, muderer):(0.9, 0.2)

not(assuage(death, family):(0.7, 0.0)) u

r u

Arg12 agree(death):(0.8, 0.1) not(desire(family, death):(0.0, 0.6))

r

r

r

Arg22 Arg21 agree(death):(0.1, 0.4)

Arg11

hate(family, murderer):(1.0, 0.2)

atone(death,guilt):(0.2, 0.8) not(remorse(dead):(1.0, 0.0))

m-Arguments of KB1 r m-Arguments of KB2

Fig. 1. Relation among arguments of Example 4, where ‘u’ stands for undercut, and ‘r’ for rebut. The m-arguments framed in a thick line are justified arguments

Definition 15. (MAA Semantics). Let KB be a MAS. 1. MAA(KB) |= A : µ iff there exists a justified argument Arg ∈ JKB such that for some ρ ≥ µ, A : ρ ∈ concl(Arg); 2. MAA(KB) |= not(A : µ) iff for every argument Arg ∈ ArgsKB , if there exists ρ ≥ µ such that A : ρ ∈ concl(Arg), Arg is overruled: Example 4. We give an example of MAA about the pros and cons of the death penalty of murderers. Suppose a complete lattice T = [0, 1]2 where (µ1 , ρ1 ) ≤ (µ2 , ρ2 ) iff µ1 ≤ µ2 and ρ1 ≤ ρ2 , and in (µ, ρ) ∈ T , µ and ρ represent the degrees of an affirmation and a negation respectively. Let KB = {KB1 , KB2 , KB3 } be a MAS, and each KBi be a knowledge base such that: KB1 = { agree(death) : (0.8 × X, 0.5 × Y ) ← hate(f amily, murderer) : (X, Y ) & not(desire(f amily, death) : (0.0, 0.6)), hate(f amily, murderer) : (1.0, 0.2) ← } KB2 = { agree(death) : (0.5 × X, 0.5 × Y ) ← atone(death, guilt) : (X, Y ), atone(death, guilt) : (0.2, 0.8) ← not(remorse(dead) : (1.0, 0.0)) } KB3 = { agree(death) : (X, Y ) ← desire(f amily, death) : (X, Y ), desire(f amily, death) : (0.0, 1.0) ← not(assuage(death, f amily) : (0.7, 0.0)), desire(f amily, death) : (X, 0.3) ← hate(f amily, murderer) : (X, 0.0), hate(f amily, murderer) : (0.9, 0.2) ← },

where X and Y are annotation variables. Figure 1 shows every possible m-argument and every possible attack relation among them. The justified arguments are 1 2 = {Arg21 , Arg31 , Arg32 }, JKB = JKB = {Arg21 , Arg22 , Arg31 , Arg32 }. JKB

Formal Argumentation Frameworks for EGAP

37

An interpretation in the restricted semantics [4] is a mapping from the Herbrand base to T . The least interpretation ∆r is an interpretation assigning every atom to a least element (⊥), and hence ∆r |= A : ⊥ for every atom A. This means that an arbitrary interpretation satisfies any annotated atoms with an annotation ⊥. Therefore, we have to assume that there exists a rule A : ⊥ ← for every atom A in any EGAP to construct BA that coincides with WFS under the restricted semantics. With this assumption, let us consider Example 4 again. It is only in KB2 that there exists explicitly a knowledge about atone(death, guilt). Arg21 is supposed to be rebutted by an argument made from atone(death, guilt) : (0, 0) implicitly included in KB1 and KB3 . Likewise, Arg31 and Arg32 are rebutted in the same manner. The only justified arguments are ones of the form [A : ⊥ ←] for any atom A. We think that the rebuttal based on such unawareness is too skeptical. The general semantics allows to distinguish explicit knowledge from implicit knowledge by the existence of the empty ideal. Thus, we would say the general semantics is more relevant to the restricted one.

6

Concluding Remark

The main contribution of this paper is that we have given the formalization of two argumentation frameworks so that the argumentation semantics for BA coincides with the well-founded semantics for EGAP, under the general semantics of GAP. We have found that in MAA derived from BA, the general semantics is more adequate than the restricted semantics employed in [8, 2]. We conclude our paper with the remark that our argumentation frameworks with the expressive EGAP are more suitable to arguments with incomplete, inconsistent, or vague knowledge (for example, even with empirical and epistemic knowledge states such as ”it is and is not”, or ”it neither is nor is not”), and are flexible to further extension such as to more versatile attack relations, cooperation and learning through argumentation.

References [1] C. I. Chesnevar, A. G. Maguitman and R. P. Loui. Logical models of argument. ACM Computing Surveys, 32(4): 337-383, 2000. 28 [2] C. V. Damasio, L. M. Pereira and T. Swift. Coherent Well-founded Annotated Logic Programs. Proc. of the 5th Int. Conference on Logic Programming and Nonmonotonic Reasoning, pp. 262-276, 1999. 29, 37 [3] P. M. Dung. An Argumentation semantics for logic programming with explicit negation. Proc. of 10th Int. Conference on Logic Programming, pp. 616-630, 1993. 28, 30, 31 [4] M. Kifer and V .S. Subrahmanian. Theory of generalized annotated logic programmin and its applications. J. of Logic Programming, 12: 335-397, 1992. 28, 29, 32, 37 [5] I. Mora, J. J. Alferes and M. Schroeder. Argumentation and Cooperation for Distributed Extended Logic Programs. Working notes of the Workshop on Nonmonotonic Reasoning, 1998. 35

38

Takehisa Takahashi et al.

[6] H. Prakken and G. Sartor. Argument-based Extended Logic Programming with Defeasible Priorities. J. of Applied Non-Classical Logics, 7(1): 25-75, 1997. 28, 30, 33 [7] R. Schweimeier and M. Schroeder. Well-founded argumentation semantic for extended logic programming. Proc. of Int. Workshop on Non-monotonic Reasoning, 2002 30, 33, 34 [8] V. S. Subrahmanian. Amalgamating Knowledge Bases. ACM Transactions on Database Systems, 19(2): 291-331, 1994. 29, 30, 37 [9] T. Takahashi, Y. Umeda and H. Sawamura. Formal Argumentation Frameworks for the Extended Generalized Annotated Logic Programs. Technical Report http://www.cs.ie.niigata-u.ac.jp/~takehisa/, 2003. 32, 34

Specifying and Validating Reactive Systems with CommonKADS Methodology Maamar El-Amine Hamri, Claudia Frydman, and Lucile Torres Laboratoire des Sciences de l'Information et des Systèmes (LSIS) Campus Universitaire de St-Jérôme Avenue Escadrille Normandie-Niemen, 13397 Marseille cedex 20 France {amine.hamri,claudia.frydman,lucile.torres}@lsis.org http://www.lsis.org

Abstract. We present an extension of the CommonKADS methodology that is relevant to the specification and validation of reactive system behavior. CommonKADS is considered as one of the best-known methodology for specifying and designing knowledge systems. It provides a language for describing the system behavior, which is suitable for transformational systems. We introduce the elements useful for event-driven behavior specification in this language. The later translation of these elements into statecharts formalism allows us to validate the reactive behavior of the system by simulation.

1

Introduction

A reactive system is designed by a set of tasks that cooperate together in realizing a complex behavior. This is an open system that reacts to extern events by answering with actions. Like any system, the specification of a reactive system requires three points of view: structural, functional and behavioral views [1]. The behavior of reactive systems is really different from those of transformational systems and requires a different approach to be specified [2]. Using the CommonKADS methodology [3] for system specification gives some well-known advantages such as distinguishing specification from design or associating generic models with tasks. However, the methodology does not provide any concept for specifying the behavior of reactive system since constraints related to time and events cannot be introduced in the task layer in the CommonKADS expertise model [4]. We propose to extend the CommonKADS methodology by adding the concepts the expert should have to specify the behavior of a reactive system and to validate this behavior by simulation. These concepts are based on the statecharts formalism that is suitable for behavioral specification of reactive system.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 39-44, 2003.  Springer-Verlag Berlin Heidelberg 2003

40

2

Maamar El-Amine Hamri et al.

CommonKADS Task Behavior

In the CommonKADS knowledge model, task knowledge describes a hierarchical organization of tasks. So, any task is structurally defined by the set of its component subtasks; and the task behavior is described at each abstraction level, by specifying the running order of the components of the tasks. The hierarchical decomposition principle reduces system complexity, it makes possible to study the system at several abstraction levels and it makes easier the specification changes. Moreover, a hierarchical specification of behavior leads to obtain tasks free from their utilization domain and reusable. When describing the behavior of the task consists in ordering subtasks in time, it is a question of defining the internal control of the task and the task must be autonomous. Such a task is viewed as a function transforming inputs into outputs and relations between tasks are input/output relations. In the case of a reactive system, the control is external, driven by external events whose occurrences induce state changes of the system. The behavior of a reactive system is really the set of allowed sequences of events, conditions and actions with some timing constraints [2]. CommonKADS does not provide the means for specifying such an external control. We propose to complete the CommonKADS task description so as tasks can represent activities or states of the system (instead of functions) and their behavior is given in terms of events whose occurrences change the system state or trigger activities.

3

Reactive Behavior Specification

The reactive behavior is described by means of triplets (evt, task1, task2) where evt is an event name and the tasks task1 and task2 have the same abstraction level (they are component of the same parent task). This can be interpreted as follows: if task1 is running and evt happens, then task1 is interrupted and task2 is triggered. We introduce the initial task concept in order that a task triggers one of its components at least (the initial one). The initial task may be explicitly showed by this kind of specification: (init, task), where init is the name of the predefined event that makes task as initial one. Otherwise, the subtask given in first in the component list is the initial task, by default. Finally, the delay concept of a task expresses the maximum life of the task. It makes possible the task to be deactivated if no event occurs during the delay considered. The delay of a task can be expressed in the form of a positive floating number. Relations are not limited to tasks from one abstraction level. A relation (evt, task1, task2) can also be specified for tasks that are not at the same abstraction level, or that have not the same parent task. We propose to describe reactive behavior in the control structure of the task in the CommonKADS knowledge model, according to the following elementary grammatical rules:

Specifying and Validating Reactive Systems with CommonKADS Methodology

41

Reactive_Behavior := Orthogonal_Behavior | Orthogonal_Behavior Reactive_Behavior | Delay Orthogonal_Behavior | Delay Orthogonal_Behavior Reactive_Behavior Delay := Positive_Float_Number Orthogonal_Behavior := (‘ ‘init' ‘,' Id_Task ‘)'Triplet_Set Triplet_Set := ‘(' Id_Event ‘,' Id_Task ‘,' Id_Task ')' | ‘(' Id_Event ‘,' Id_Task ‘,' Id_Task ')' Triplet_Set Id_Event := Identifier Id_Task := Identifier Let us consider the example of the Citizens Quartz Multi-Alarm III watch [2]. This watch has a main display area and four smaller ones, a two-tone beeper, and four control buttons denoted here a, b, c and d. It can display the time or the date (day of month, month, day of week); it has a chime (beep on the hour if enabled), two independent alarms and a stopwatch (with lap and regular display modes). Time display is the default task. The time and the date display tasks are linked by d's, time shows again after two minutes in date display. Pressing the a button displays sequentially alarm1, alarm2, chime, stopwatch and return to time. Here, we specify just a part of the watch without refinement of the tasks. The components and the behavior of the display task are defined as follow: Task Display Decomposition: Time, Date, Alarm1, Alarm2, Chime, Stopwatch Control structure: (init, Time), (d, Time, Date), (d, Date, Time), (2, Date, Time), (a, Time, Alarm1), (a, Alarm1, Alarm2), (a, Alarm2, Chime), (a, Chime, Stopwatch), (a, Stopwatch, Time)

4

Operationalizing into Satecharts

The statechart formalism defined by David Harel [2] constitutes a standard for the specification of reactive systems. This formalism supports hierarchical descriptions and has an operative semantics that makes possible the specification to be validated by simulation [5]. A CommonKADS task can be interpreted as an entity provided with inputs/outputs, active or not, according to the event occurrences. With this point of view, we can traduce the reactive behavior of tasks described in terms of our elementary language into statecharts, by using the equivalencies below: Task layer of a CommonKADS Statechart knowledge model a task an initial task an event in a triplet a subtask the delay of a task

a state with the associated task as activity a default state a transition labeled with the event, between two states a substate a generic event: timeout

42

Maamar El-Amine Hamri et al.

Fig. 1. Statecharts for the reactive behavior of the Display task

The translation rules we defined allow the reactive behavior specification to be automatically traduced into the statechart formalism. Figure 1 gives the specification of the library management task obtained once translated.

5

Simulation

Statecharts are discrete event models. This kind of model is the most used for reactive system specification. In discrete event models, time is continuous and an event is an instantaneous value change of one or more descriptive variables. Events happen at dates called event occurring dates, represented by positive floating numbers. A way to implement a discrete event model simulation consists in using a set of discrete descriptive variables, a scheduler (for the chronological list of events or messages) and a global clock giving the simulation current date (that is the occurring date of the event being treated). Once these elements are given, the simulation algorithm is: while scheduler not empty do begin extracting the event of the nearest future: (e, s, d) (that is the event at the head of the scheduler); current_date = d (affecting the current date with the occurring date of the event); making the state change induced by the event; creating the future timeout or init events due to this state change and stocking them in the scheduler by chronological order end The timeout events must only be treated when the time past in the state is equal to the life of this state (that is the delay of the task corresponding to the state). If an external event occurs before this delay, the timeout event must be suppressed from the scheduler. If several events can occur simultaneously, treating at one go these events makes the system to pass from the current state to a new state. This new state cannot be

Specifying and Validating Reactive Systems with CommonKADS Methodology

43

always determined in a unique way. This happens with events leading the system from a common state to different states (in the case of conflicting transitions mentioned by [5]), when there exist simultaneous occurrences of these events. In the statechart of the Display task, the transitions labeled with events a and d are conflicting transitions while the time task is activate. Consequently, we choose to ignore the happening of simultaneous events and we suppose that at the most one event occurs at any given date. Finally, we consider that statecharts modeling nondeterministic behavior constitute underspecified models that must be corrected before simulation. We give now the results of the simulation of the display task for the following scheduler: Step

Actions

0 (initial)

1

2

3

4

5 6 7

extracting the first event : (init,time,50) current_date = 50 changing the state : activating the time task extracting the first event : (d,nil,60) current_date = 60 changing the state : activating the date task adding (time_out,date,62) in the scheduler extracting the first event : (time_out,date,62) current_date = 62 changing the state : activating the time task extracting the first event : (d,nil,65) current_date = 65 changing the state : activating the date task adding (time_out,date,67) in the scheduler extracting the first event : (time_out,date,67) current_date = 67 changing the state : activating the time task extracting the first event : (a,nil,70) current_date = 70 changing the state : activating the alarm1 task extracting the first event : (a,nil,72) current_date = 72 changing the state : activating the alarm2 task

Scheduler init,time,50 d,nil,60 d,nil,65 a,nil,70 a,nil,72 d,nil,60 d,nil,65 a,nil,70 a,nil,72 time_out,date,62 d,nil,65 a,nil,70 a,nil,72 d,nil,65 a,nil,70 a,nil,72 time_out,date,67 a,nil,70 a,nil,72 a,nil,70 a,nil,72 a,nil,72

44

6

Maamar El-Amine Hamri et al.

Conclusion

We provide experts with a simple language for specifying the behavior of a reactive system, in terms of events and tasks to be activated or interrupted. We have introduced translating rules to transform automatically such specifications into statecharts. These works have led us to develop software allowing the experts to operationalize the behavioral part of the specification established with the CommonKADS methodology in the form of statecharts, and to simulate this behavior. We propose now to enrich the concepts and the associated tools by defining conditions on events and communication between tasks.

References [1] [2] [3] [4]

[5]

Frydman C. and Torres L., Vérification et Validation de Modèles CommonKADS, Revue d'Intelligence Artificielle, 2000 Harel, D. 1987. Statecharts: A Visual Formalism for Complex Systems. Science of Computer Programming, 8, pp. 231-274 Schreiber G., Akkerman H., Anjerwierden A., De Hoog R., Shadbolt N., Van De Velde W. and Wielinga B. Knowlegde Engineering and Management The Commonkads Methodology, MIT press, London, England, 1999 Torres L., Frydman C. and Garrido de Ceita A. Adding Event-driven Control in CommonKADS Knowledge Model, KES–2001 5thInternational conference on knowledge-Based Intelligent Information Engineering Systems & Allied Technologies, Osaka et Nara, Japan, september 2001 Harel, D. and Naamad A. 1996. The Statemate Semantics of Statecharts. The Weizmann institute of science rehobot, Israel, October, 1995

TMMT: Tool Supporting Knowledge Modelling Guy Camilleri, Jean-Luc Soubie, and Joseph Zalaket IRIT CCI-CSC, Universit´e Paul Sabatier 118 route de Narbonne, 31062 Toulouse Cedex 4, France {camiller,soubie,zalaket}@irit.fr

Abstract. In this paper we propose a new tool for models design and maintenance. This tool was built to design cooperative knowledge based models. We first present the specifications of such a tool: it must produce easily executable models and meta-models; it has to be easy to use, with the possibility to handle drafts of models. Then we present the characteristics of TMMT (Task Method Modelling Tool) satisfying these specifications. We finally give some elements of methodology to build conceptual models with TMMT.

Introduction We could think today that the modelling of knowledge reached a stage of development that does not imply any more research about tools supporting modelling. In addition, the design of systems imbedding knowledge developed increasingly towards the use of the task/method paradigm like primitives of representation of the reasoning. However, force is to note the under-utilization of the functionalities of these tools and the difficulty of inserting them in the activity of design when this one is carried out within traditional environments of software development, specially based on UML. The main characteristic of cooperative systems containing knowledge (see Soubie [1]) is the multiplicity of the models, which constitute them. Thus, the model of the application can be compared to the traditional conceptual models, but the model of cooperation, although of comparable nature, has as model of its domain, the tasks of the preceding model. In the same manner, the model of communication operates on itself for the implementation of strategies of dialogue. It was thus a question of designing a tool of assistance to the construction and the operationalization of models also being able to operate on themselves (metamodels) implementing the recursivity and reflexivity. A specification of the tool was the facility of handling of the primitives and the readability of the produced models. Thus, TMMT is provided with a graphic interface from which it is possible to build and to modify the model in progress. Two formats of internal representation of the models allow a better portability.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 45–52, 2003. c Springer-Verlag Berlin Heidelberg 2003


46

1

Guy Camilleri et al.

The Task/Method Paradigm, Methods and Tools for Knowledge Modelling

The reasoning seen as a task is the result of research on the knowledge acquisition and modelling of the Eighties. At this period, the concept of expert knowledge, based on sets of heuristic knowledge of an expert, yielded the place to the modelling of knowledge of a domain. The effort was put on methodologies of acquisition (the term elicitation was very often used) of knowledge resulting from one or more experts, but also from other sources, like documents. The principle guiding these steps was the construction of a complete set of structured pieces of knowledge allowing the resolution of problems in a particular field of competences. The task constituted the container of a set of pieces of knowledge solving one problem or a set of problems. In this meaning, it constituted a granule of knowledge at the level where it described the resolution. Knowledge was then described as much as possible independently from the formal representation which would be used thereafter to implement them. Methodologies of knowledge acquisition were based on a top-down analysis of the problem solving resulting in a hierarchical decomposition of the whole task into sub-tasks being able to be alternate, describing various methods for the realization of the same task. These principles are the basis of the European project KADS, follow-up of the CommonKADS methodology, which constitutes the reference in this field. Until 2000, date of publication of the handbook ”Knowledge Engineering and Management - The CommonKADS Methodology ” [2], few methodologies were based on completely executable models, since the support of modelling induced constraints which one wanted to free oneself. One will however see with MACAO II [3] an attempt to facilitate the operationalization of the conceptual models. Same manner DSTM [4] with its language ZOLA allowed the use of executable primitives task and method. However, it seems that the movement towards the supports of modelling aiming to build models easily operationalizable is not also powerful, whereas work on the constitution of ontology [5] on the one hand and the re-use of components of models of problem solving developed. First of all, the facility of use is provided by a graphical and declarative representation of the knowledge implemented. Indeed, this characteristic allows at the same time a more significant implication of the carriers of knowledge in the systems design, but also facilitates the maintenance of the models produced by a fast reappropriation of the model a long time after its first construction. Then, a capacity to carry out syntactic and/or structural checks. For example, the checking of the illegal re-use of names, the consistency of the preconditions of a task or the detection of the lack of sub-task or loops within a hierarchical decomposition of tasks. This capacity is corollary of a possibility of automating thereafter the operationalization of the produced models. Lastly, it is necessary that the produced models can be handled simply by the tools for processing, not only in their operational form, but also in their declaratory form as a representation of the task.

TMMT: Tool Supporting Knowledge Modelling

47

Fig. 1. Truck world domain

We present in the second part TMMT (Task-Method Modelling Tool) a tool of knowledge modelling answering the preceding specifications, as well as a methodological approach of modelling with this kind of systems.

2

Modelling Primitives of TMMT

In this section we define some modelling primitives, on which the execution engine used in knowledge based systems and in the plan recognition processes for plan based approaches of the discourse (see Camilleri [6]) and in the action based planners are applicable. Execution and planning are two distinct activities requiring different models. A major difference between execution and planning is: planning builds hypothetical states of the world; therefore it is possible to come back (or cancel task application); while execution directly modifies the state of the world, the achievement of a task cannot be ignored even if it is not desired. Moreover, in planning it is necessary to isolate the salient objects of the world and to clone them; these two operations are not required for the execution processes. The main objective of our modelling primitives is to allow the representation of models being able to be handled by the execution, and by the plan recognition, and by the planning processes. 2.1

Domain Model

Definition 1 A domain model describes the objects of the world handled (directly or indirectly) by the reasoning model. A generic definition of high level modelling primitives for the domain model seems to be difficult, because it is too dependent on the application (cf [7]). We only chose to use the modelling object concept (inheritance, polymorphism, etc.) adopted from object-oriented languages. For example, the object model (described in UML) below constitutes a model fragment of the possible domain under TMMT of the “Truck” world. 2.2

Reasoning Model

Definition 2 The reasoning model describes how a task can be performed.

48

Guy Camilleri et al.

All the world objects and their relevant relations handled by the reasoning model must be described in the domain model. Therefore, a domain model can be defined only from the reasoning model. In the same way, the tasks described in the reasoning model handle the domain objects, and thus they depend on the domain model. The knowledge modelled in a conceptual model often results of a parallel construction of the domain model and the reasoning model. The Task/Method paradigm has been chosen to represent the reasoning model. The following modelling primitives are defined to support planning and execution processes. Definition 3 A task is a transition between two world state families. A task is defined as follows: Name Par Objective Methods

Task name Typified list of parameters handled by the task Goal state of the task Method list achieving the task

The field Name specifies the name of the task. The list of parameters Par represents the set of world objects (described in the domain model) handled by the task. The objective describes the goal of the task a state form. This Objective field can appear redundant, however a goal can be expressed in a natural language by to two different ways, by a verb (corresponding to the name of the task) or by a state (Objective of the task). All the defined methods during modelling are recorded in the list (Method of the task. As an example of a task model in our paradigm, the ”transport” task is defined as follows: ( Name: transport; Par : 0: Truck; 1:Package; 2:City; Objective: 1:getLocation().equals(2:); Methods: m )

Definition 4 A method describes one way (at only one level of abstraction) of performing a task. A method is characterized by the following fields: Heading App-cond Prec Effects Control Sub-task

Task achieved by the method Applicability conditions Preconditions which must be satisfied to be able to apply the method. Effects generated by the successfull application of the method achievement order of the sub-tasks sub-task set

The action performed by the method is indicated in the heading. The applicability conditions (as parameters of action) are used to constraint the instantiation of the method. For example, the parameters 0:getLocation() and 1: of the task transport (presented previously) must be different, this constraint will be modelled under the applicability conditions by 0:getLocation().equals(1:).equals(false). The preconditions are the conditions which must

TMMT: Tool Supporting Knowledge Modelling

49

be satisfied to apply the method. The difference between preconditions and applicability conditions is: an agent can satisfy the preconditions to apply the method, if an applicability condition is not satisfied then it will not try to satisfy it (cf Carberry [8]). The effects are caused by the application of the method. The task objectives belong necessarily to the effects, therefore all effects of all task’s methods contain the task objective. All the methods performing a task must thus generate a state of the world containing the objective; nevertheless the effects can be different. The execution order of sub-tasks is described in the Control field, the sub-tasks are recorded in the field Sub-task . For example, the following method carries out the task ”transport”: ( Heading : transport(Truck, Package, City); App-cond : 1:getLocation().equals(2:).equals(false); Prec: 0:getLocation().equals(1:getLocation(); 0:canLoad(1:); Effects: 1:setLocation(2:); 0:setLocation(2:); Control : Load(0:,1:); Move(0:,2:); Sub-task : {Load ; Move} )

Definition 5 A terminal task is a directly executable task. Its execution does not require decomposition (or planning). The terminal tasks are tasks having only one method. The field sub-task is empty and the Control can point on an executable code. 2.3

The Language of Constraint

To be used with TMMT, a domain model must be an object model. The constraints (preconditions, applicability conditions, effects, objective . . . ) of the tasks and the methods correspond to object methods of the modelled domain. The domain objects handled by a task are specified in the list of parameters. Therefore, a task cannot reach an object that is not a part (or that is not related to an object) of its parameters list. However, it is possible to access to a great number of world objects through object’s methods of the parameters list. The choice of the accessibility methods is often guided by the construction of the task model. In the previous example: 0:getLocation().equals(1:getLocation()), the number 0 corresponds to the parameter 0 of the list of parameters (Truck, Package, City). 0 indicates an object of the type (class) Truck. 2.4

The Language of Control

The field Control of Method allows the description of the execution order of method’s sub-tasks. Currently possible controls are: The sequence “;”, the conditionnals “If . . . Then and If . . . Then . . . Else” and the loop “While”. 2.5

TEE (Task Execution Engine)

TEE is an execution engine allowing the launching of a task execution. The main algorithm of this engine can be described roughly in the following way:

50

Guy Camilleri et al.

start(Task t) {if (t instanceof TerminalTask) startTerminal(t); else {choose the method to apply; found=false; for all methods m of t and if not found do {if ( m.getCondApp() is true ) {found=true; addMethodNode(t); startMethod(m,found);}} if (!found ) error("cannot execute");}}}

The procedure start(t) launches the execution of the task T. The task T must be instantiated to be carried out. If the task T is a terminal task, the engine launches the code execution attached to this task ( startTerminal(t)). If the task T is not terminal, the engine chooses a method from the current task. It traverses all methods; if a method is applicable, it applies it (startMethod(m, found)). The procedure startMethod(m, found) achieves a control analysis and launches the procedure start(ti) on sub-tasks of the method m. In the case where a subtask has not applicable methods, the procedure startMethod(m, found) assigns false to the found variable, which involves the search of another method. This property is significant because a method can be applicable (preconditions are true) to an abstraction level and is not longer on a lower level (which constitutes the principal interest of this kind of hierarchical modelling). As the tasks are represented in object form, a reasoning model can be regarded as a domain model by another reasoning model (meta-model). Moreover, the Control of the methods (as sequence, If . . . Then, etc.) is also modelled in the form of (terminal) tasks. A model (meta) of tasks can handle (build dynamically, modify . . . ) another model of tasks. Under TMMT, an execution meta-model was built. The execution of this model launches the task execution associated. Therefore, the execution engine TEE can itself modify the execution process through a meta-model. Moreover, This meta-model is a recursive model.

3

Methodological Aspects

Like the practice showed in the majority of cases of knowledge modelling, one cannot consider this activity like a sequential process. Indeed, the object structure of the model constraints the representation of the characteristics of the task (preconditions, conditions of applicability, inputs and outputs), and reciprocally, any reorganization of the control contained in the methods of the tasks or the decomposition itself. This is why it first of all seems preferable to build a rather summary domain model, from which one can carry out the first modelling of the tasks. The description of the method (the methods) of the preconditions and the conditions of applicability reveals new objects or properties which must be integrated in the domain model. One builds thereafter repeatedly the model of the field and the model of tasks, until the completion.

TMMT: Tool Supporting Knowledge Modelling

51

TMMT being intended for the construction of models of various levels, the modification at one level induces the modification of the others. For example, a questioning of the cooperation can involve a modification of the granularity of the tasks, forcing to introduce new characteristics into the objects of the domain. The methodology of modelling for the cooperative knowledge based systems is described in the life cycle of these systems suggested in [1]. It is this methodology which is at the base of the specifications of TMMT. Models can be experimentally validated on the most important cases (nominal ones) with hierarchical AI planners. However, it appears possible to validate formally some pieces of models.

Conclusion There is a requirement in terms for methodology and tools to build flexible and maintainable knowledge based systems, being able to contain several types of models. TMMT meets this requirement by proposing a framework easy to use and to conceive in a flexible way the models which are the subject of a rigorous syntactic checking and which are easily operationalizable. The form of the built models facilitates their use by processes of plan recognition and generation, useful in the situations not explicitly envisaged and for which the modelled knowledge can bring help. TMMT provides an environment of development of reasoning model based on a domain model described in the object paradigm. It appeared difficult to us to define primitives of modelling for the domain; the nature, objectives and the constraints (performance, robustness etc.) of the system are very dependent on the application. As the cooperative knowledge based systems are information processing systems, we chose to use the basic primitives of the object programming allowing, according to the application, to develop adapted tools of modelling of the domain. With TMMT, it is possible to describe meta-models (models handling a reasoning model or a domain model). The models of the domain and the reasoning are both object models; consequently it is possible to consider cross-references between these two models. This type of links must be managed by all the generic processes of handling of the knowledge models. Moreover, TMMT makes it possible to describe recursive models. The recursive models are often essential in the description of the meta-model (as in the model of the engine of execution presented). TMMT is currently used in an aeronautical project to build models for a cooperative knowledge based system aiming to help maintenance operator during his activity.

52

Guy Camilleri et al.

References [1] Soubie, J.L.: Coop´eration et syst`emes ` a base de connaissances. Master’s thesis, habilitation ` a diriger des recherches, Universit´e de Toulouse III, Toulouse (1996) 45, 51 [2] Schreiber, G., Akkermans, H., Anjewierden, A., de Hoog, R., Shadbolt, N., de Velde, W.V., Wielinga, B. In: Knowledge Engineering and Management The CommonKADS Methodology. The MIT Press, Cambridge, Massachusetts, London, England (2000) 46 [3] Matta, N., Aussenac-Gilles, N.: Expliciter Une M´ethode de R´esolution de Probl`emes Avec MACAO : Probl`emes M´ethodologiques. Aussenac-Gilles,N., Laublet,P. et Reynaud,C. Eds. Cepadues Editions , Toulouse, France (1996) 46 [4] Trichet, F., Tchounikine, P.: DSTM: A framework to operationalize and refine a problem-solving method modeled in terms of tasks and methods. International Journal of Expert Systems With Applications 16 (1999) 105–120 46 [5] Sowa, J.F.: Ontology, metadata, and semiotics. B. Ganter & G. W. Mineau, eds., Conceptual Structures: Logical, Linguistic, and Computational Issues, Lecture Notes in AI (1867), Springer-Verlag (2000) 55–81 46 [6] Camilleri, G.: A generic formal plan recognition theory. In: IEEE International Conference on Information, Intelligence and Systems ICIIS’99. (1999) 540–547 47 [7] Ist´enes, Z.: Zola a language to operationalise conceptual models of reasoning. Jounal of Computing and Information 2 (1996) 689–706 47 [8] Carberry, S.: Modeling the user’s plans and goals. Computational Linguistics 14 (1988) 23–37 49

State-Based Planning with Numerical Knowledge Joseph Zalaket and Guy Camilleri IRIT CCI-CSC, Universit´e Paul Sabatier 118 route de Narbonne, 31062 Toulouse Cedex 4, France {zalaket,camiller}@irit.fr

Abstract. The drawback of the STRIPS[4] encoding to support numbers in planning has motivated the extension to PDDL language the PDDL2.1 version [5] to allow numerical state variables. Many planning systems as FF-Metric [8] and SAPA [3] treat numbers in planning, but only resources and time are represented as numbers in their planning problems. In contrast, some real world problems require an enormous numerical representation. Therefore, we present in this paper a new planning framework allowing the application of arithmetic functions for numerical knowledge update. We propose a new action representation to improve numerical conditions and effects, which allow the application of symbolical planning techniques over numerical knowledge. As an application of our planning framework, we show the implementation of a numerical version of the well-known FF[7] planning system able to solve domains containing symbolical and/or numerical knowledge.

1

Introduction

In the recent years the development of efficient automated planning algorithms such as GRAPHPLAN [1], FF [7], HSP and HSP-r [2] has enhanced the planning process and made it applicable on a large number of symbolical domains. But problems closer to real world often contain numerical knowledge. Many recent planners as [6],[8] has been extended to support numerical handling, but often this planners are interested in treating the problem of time and resources added to the classical planning domains as an attached aspect. In this article we suggest the non-separation of the numerical aspect from the symbolical one. This integration of numerical and symbolical aspects within a planning problem gives us the possibility of sharing the results already obtained by the symbolical planning with the numerical planning. We propose the extension of STRIPS[4] language to represent numerical objects and numerical effects; in a way that numerical knowledge can be seen and handled as the symbolical ones. This integration within the STRIPS language will allow the planning to solve new types of problems. For example, to solve a planning problem consisting of producing some final products according to what is available in the stock as raw material products. The domain objects of this problem can be viewed as a set of object’s categories (products) (product r1, product r2, product F1,. . . ). Suppose we want V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 53–60, 2003. c Springer-Verlag Berlin Heidelberg 2003


54

Joseph Zalaket and Guy Camilleri

to produce an item of final product F1, which requires the use of 300 items of raw material product r1 from 5000 existing items in the stock. In classical STRIPS modeling we have to create 5000 different constant symbols to represent the items of product r1. And using 300 of these items can be done one by one in the best case. So the classical STRIPS language is not qualified to model this kind of problems. Indeed, an extension of STRIPS to integrate numerical knowledge modeling seems to be crucial. In our paradigm the objects of this example can be represented by numbers (a number for each product), which yields to a simpler problem resolving. Also in our paradigm, we have the advantage of representing symmetrical domain objects by number (e.g. the ferry domain Table 1) instead of multiple constant objetcs. An extension of the STRIPS language to support numbers will be detailed in section-1. To illustrate our approach, we will present in section-2 a numerical version of FF[7] planning system, which we call it NFF (Numerical Fast Forward). The difference between NFF and the numerical version of FF introduced by Hoffman Metric-FF [8] is that: NFF treats numerical knowledge by the same way as the symbolical ones. In contrast, Metric-FF treats numerical knowledge as a separate part of the symbolical planning problem. Some empirical results will be shown in section-3 before the conclusion of this paper.

2

Extending STRIPS to Support Numbers

In the following we use a sub-set of the first order logic to introduce an extension of the STRIPS language to deal with the numerical knowledge. Definition 1 The language L=(V, C, F , P) is defined over sets of variable symbols V, constant symbols C, function symbols F , and predicate symbols P = Pp ∪ Pt in the following way: 1. Variable symbols x ∈ V are terms. 2. Constant symbols c ∈ C are terms. 3. If f ∈ F is a function symbol with an arity n and t1 , . . . , tn are terms, then f (t1 , . . . , tn ) is a term. 4. All terms are obtained by applying the above rules (1, 2 and 3) in a finite time. 5. If p ∈ Pp is a predicate symbol with an arity m and t1 , . . . , tm are terms then p(t1 , . . . , tm ) is an atom. 6. If p ∈ Pt is a predicate symbol with an arity l and t1 , . . . , tl are terms then p(t1 , . . . , tl ) is an atom. Pt = {<, <=, =, =, >=, >} ∪ D where the first set is the set of comparator predicates and D is the set of the definition domains. These predicates are associated to a domain theory T . are atoms then the set 7. If p1 (. . .), p2 (. . .), . . . , pk (. . .) {p1 (. . .), p2 (. . .), . . . , pk (. . .)} represents the conjunction of atoms, which is the formula p1 (. . .) ∧ p2 (. . .) ∧ . . . ∧ pk (. . .).

State-Based Planning with Numerical Knowledge

55

Remarks : – All the language sets V, C, F and P can be infinite. – All constants c ∈ C are typed, that means it exists a predicate p ∈ D such as T
p(c). Definition 2 An action α (or an operator) represents a state transition and is defined in a STRIPS style by: – Param(α) is the list of parameters specified by a list of variable symbols. – Pre(α), Add(α) and Del(α) are respectively the precondition list, the add list and the delete list. Each one of these three lists is represented by a set of atoms.

Table 1. The ferry problem that consists of transporting a number n of cars from a place r1 to a place r2 can be modeled by introducing cars as number instead of constant symbols in the following way: Load(R,X) Prec: Add : Del :

Rive(R), Integer(X), > (X, 0) emptyF, nbC(R, X), atF (R) f ullF, nbC(R, −(X, 1)) emptyF, nbC(R, X)

UnLoad(R,X) Prec : Rive(R), Integer(X) f ullF, nbC(R, X), atF (R) Add : emptyF, nbC(R, +(X, 1)) Del : f ullF, nbC(R, X) Move(R1 ,R2 ) Prec: Rive(R1 ), Rive(R2 ) = (R1 , R2 ), atF (R1 )  Add : atF (R2 ) Del : atF (R1 )

Note: As in STRIPS, all variables are present in the parameter list. Therefore, numerical variables (like X) should also be present in the parameter list. In this example, the numerical parameter X represents the number of cars in a place R. The functions + and - are used to carry out calculations of the numerical terms. Definition 3 A world state is described by a set of atoms without function symbols or variable symbols. Definition 4 Substitution. The following two particular substitutions are used:

56

Joseph Zalaket and Guy Camilleri

The first substitution noted σ correponds to substitute variable symbols by constant symbols and function terms without variable symbols by constant symbols. The second substitution noted θ such as θ(X) is a substitution of X where: – X = {p1 (...); ...; pn (...)} is a set of atoms without variable symbols then θ(X) = {θ(p1 (...)); ...; θ(pn (...))} – X = P (t1 , . . . , tn ) is an atom without variable symbols then θ(X) = P (θ(t1 ), . . . , θ(tn )) – X = t is a term without variable symbols then • if t is a constant symbol then t = c and θ(t) = c. • if t = f (c1 , ... , cn ) with c1 , ... , cn constant symbols then it exists a constant symbol c such as {c/f (c1 , ... , cn )} ⊆ θ and θ(f (c1 , ... , cn )) = c. • (recursive part) if t = f (t1 , ... , tn ) with t1 , ..., tn are terms without variable symbols then θ(t) = θ(f (t1 , ... , tn )) = θ(f (θ(t1 ), ..., θ(tn ))). Remarks : – These two substitutions are used to replace variables by constants and to evaluate the functions (by replacing functions by constants). – As in the presented language L all terms are finite, so the recursive definition of θ stops at a certain time. – The substitution θ(f (c1 , ... , cn )) corresponds to the application of the function f . For example if θ(+(1, 2)) = 3 then in the domain theory T the result of 1 + 2 is 3. – If an infinite domain definition is used for the numerical variables then the C set can be infinite. – In the domain theory T , a function must only return a constant value and it does not have any side effects. Definition 5 An action α can be instantiated by two substitutions σ and θ (defined as above) iff ∀p(t1 , ... , tn ) ∈ prec(α), p ∈ Pt then T
θ(σ(p(t1 , ... , tn ))). In the ferry example (Table 1) Load(R,X) can be instantiated by σ={r1 /R; 1/X} and {0/ − (1, 1)} ⊆ θ because the theory T uses the classical interpretation of the comparator > then θ(σ(>(X, 0)) = >(1, 0), θ(σ(Rive(R)))=Rive(r1 ), θ(σ(Integer(X)))=Integer(1) and T
Rive(r1) ∧ Integer(1)∧ >(1, 0). Definition 6 a is a ground action of α iff it exists two substitutions θ and σ such as α can be instantiated by θ and σ. a is defined by: – Param(a)=σ(Param(α)). – Prec(a)={p(t1 , ..., tn ) such as it exists p’(t’1 , ... , t’n ) ∈ Pre(α), p’ ∈ Pp and p(t1 , ..., tn )=θ(σ(p’(t’1 , ... , t’n )))}. – Add(a)={p(t1 , ..., tn ) such as it exists p’(t’1 , ... , t’n ) ∈ Add(α), p’ ∈ Pp and p(t1 , ..., tn )=θ(σ(p’(t’1 , ... , t’n )))}. – Del(a)={p(t1 , ..., tn ) such as it exists p’(t’1 , ... , t’n ) ∈ Del(α), p’ ∈ Pp and p(t1 , ..., tn )=θ(σ(p’(t’1 , ... , t’n )))}.

State-Based Planning with Numerical Knowledge

57

Remark : Each variable symbol and each function symbol is replaced by the application of θ o σ. Therefore, the generated ground actions are identical to the ground actions generated in a pure STRIPS paradigm. They don’t contain any atom of Pt .

3

Numerical Planning

Usually planners generate the ground actions from the domain of action variables. In our framework, numerical knowledge can lead to an infinite number of ground actions. For example: the action Load(r1 ,1) defined below is a ground action of the Load(R,X) action. Load(r1 ,1) Prec: emptyF, nbC(r1 , 1), atF (r1 ) Add : f ullF, nbC(r1 , 0) Del : emptyF, nbC(r1 , 1)

3.1

Planning Graph

We present an approach based on the relaxation of the planning problem through a planning graph building for heuristic calculation. In this numerical paradigm, the generation of ground actions has to be completed progressively during the building of the graph. The idea is to instantiate actions from the current level during the graph building in a forward pass. In this way, the use of bijective (inversible) function is not required. The resulting ground actions being identical to pure STRIPS ground actions, which make possible the use of symbolical algorithms based on a planning graph building like GRAPHPLAN [1], FF [7] and HSP[2] to solve numerical problems. However, the search completeness cannot be guaranteed as the use of functions and numbers makes the planning process undecidable. To avoid an infinite flow during the planning process we give the modeler a possibility to add a lower limit and an upper limit for each numerical type depending on the planning problem to be solved. For example in the ferry domain of Table 1, if the problem consists of transferring 50 cars from place r1 to place r2 and the initial state contains 50 cars in place r1 (nbC(r1 , 50)) and 0 cars in place r2 (nbC(r2 , 0)), the interval of upper and lower limits for the integer variable X in the Load and UnLoad actions definition is [0, 50] . And thus, extra preconditions could be added to the list of preconditions ; < (X, 50) for the Load(R, X) action and > (X, 0), < (X, 50) for the U nLoad(R, X) action. 3.2

Numerical Fast Forward (NFF)

In contrary to FF, the ground actions in NFF are not computed at the beginning from the definition domain of variables.

58

Joseph Zalaket and Guy Camilleri

Fig. 1. The relaxed planning graph of the problem P

The NFF algorithm can be roughly described in the following way: 1. Only the symbolical part of actions is instantiated. 2. The planning graph is built by completing the actions instantiation (numerical part), then by applying the instantiated actions. 3. The relaxed plan is extracted from planning graph; the length of this relaxed plan constitutes the heuristic value. 4. The heuristic value previously calculated is used to guide the search of an algorithm close to the FF’s Hill Climbing algorithm. The heuristic calculation (steps: 2)and 3) and 4)) is done for each state in the main algorithm. Relaxed Planning Graph Building Let consider the following planning problem P=(A,I,G) where the action set A={Move; Load; UnLoad} (see Table 1), the initial state I={atF (r1 ); emptyF; nbC(r1 , 1); nbC(r2 , 0)} and the goal G={nbC(r2 , 1); atF (r1 )}.The relaxed planning graph of the problem P is discribed in figure 1. The graph starts at the level 0 by the initial state I. The actions Move(r1 ,r2 ) and Load(r1 ,1) are applicable at this level. The partially instantiated action Load(r1 ,X) is completed by σ={r1 /R; 1/X} because {nbC(r1 , 1); atF (r1 ); emptyF } ⊆ Level 0 and T
>(1,0). Moreover, θ according to the domain theory T replaces -(1,1) by 0 , that is {0/-(1,1)} ⊆ θ. The action Load(r1 ,1) is defined by: {emptyF ; nbC(r1 , 1); atF (r1 )}, Add(Load(r1 ,1))= Pre(Load(r1,1))= {f ullF ; nbC(r1, 0)} and Del(Load(r1 ,1))= {emptyF ; nbC(r1 , 1)}. The two actions Move(r1 ,r2 ) and Load(r1 ,1) are applied and their add lists are added to the level 1. Remark : In the relaxed graph building algorithm, we only defined the σ substitution. It seems reasonable that the θ substitution be common for all the actions in a planning problem. In our implementation, all functional calculus are carried out in the interpretation domain The actions completion algorithm is described in Algorithm1

State-Based Planning with Numerical Knowledge

59

Algorithm 1 Action completion algorithm for all p(...) ∈ Pre(α) and p ∈ Pp do for all predicate symbols p’ in the current level for which p=p’ and for all c’i ∈ Param(p’) and ci ∈ Param(p) such as c’i =ci do for all variable terms Xi ∈ Param(p) and ci ∈ Param(p’) do {ci /X} is added to the set S of σ end for end for end for for all σ ∈ S do if α can be instantiated by σ and θ then generate the corresponding ground action end if end for

4

Empirical Results

We have implement the NFF in Java, our main objective was the test of feasibility of certain type of numerical problems, little effort is made for code optimization. The machine used for tests is Intel celeron Pentium III with 256MB of RAM. The water jug problem: Consists of 3 jugs j1 ,j2 and j3 where Capacity(j1 ,36), Capacity(j2 ,45), Capacity(j3 ,54) The initial state is: Volume(j1 ,16), Volume(j2 ,27), Volume(j3 ,34) The goal is : Volume(j1 ,25), Volume(j2 ,0), Volume(j3 ,52)− > time = 8.36 s. The ferry domain: The initial state is: atF (r1 ), emptyF , nbC(r1 , 50), nbC(r2 , 0) For goals: nbC(r2 , 5)− > time= 0.571 s; nbC(r2 , 10)− > time= 1.390 s; nbC(r2 , 30)− > time= 21.937 s.

5

Conclusion

We have presented a STRIPS extension to support numerical and symbolical domains definition. We have proposed the way of substituting actions progressively during the planning process to reduce the ground action generation for numerical domains. The main goal of introducing numbers in the planning by the presented way is to allow the definition of domains closer to the real world. In our framework world objects can be retrieved from numerical variables by functions evaluation instead of only being constant symbols as in pure STRIPS. We haven’t interested in our work in the resources optimization (time or metrics), but we believe this task can be added easily to (or retrieved from) our planning framework.

60

Joseph Zalaket and Guy Camilleri

References [1] Avrim L. Blum and Merrick L. Furst. Fast planning through planning graph analysis. Proceedings of the 14th International Joint Conference on AI (IJCAI95), pages 1636–1642, 1995. 53, 57 [2] B. Bonet and H. Geffner. Planning as heuristic search. Artificial Intelligence, 129:5–33, 2001. 53, 57 [3] Minh B. Do and Subbarao Kambhampati. Sapa: A domain-independent heuristic metric temporel planner. European Conference on Planning, 2001. 53 [4] Fikes R. E. and Nilsson N. STRIPS: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2:189–208, 1971. 53 [5] M. Fox and D. Long. PDDL2.1 : An extention to PDDL for expressing temporal planning domains. AIPS, 2002. 53 [6] P. Haslum and H. Geffner. Heuristic planning with time and resources. Proc. IJCAI-01 Workshop on Planning with Resources, 2001. To appear., 2001. 53 [7] J. Hoffman. FF: The fast-forward planning system. AI Magazine, 22:57 – 62, 2001. 53, 54, 57 [8] J. Hoffmann. Extending FF to numerical state variables. to appear in: Proceedings of the 15th European Conference on Artificial Intelligence, Lyon, France, 2002. 53, 54

KAMET II: An Extended Knowledge-Acquisition Methodology* Osvaldo Cairó and Julio César Alvarez Instituto Tecnológico Au tónomo de México (ITAM) Department of Computer Science, Río Hondo 1, Mexico City, Mexico [email protected] [email protected]

Abstract. The Knowledge-Acquisition (KA) community necessities demand more effective ways to elicit knowledge in different environments. Methodologies like CommonKADS [8], MIKE [1] and VITAL [6] are able to produce knowledge models using their respective Conceptual Modeling Languages (CML). However, sharing and reuse is nowadays a must-have in knowledge engineering (KE) methodologies and domain-specific KA tools in order to permit Knowledge-Based System (KBS) developers to work faster with better results, and give them the chance to produce and utilize reusable Open Knowledge Base Connectivity (OKBC)-constrained models. This paper presents the KAMET II1 Methodology, which is the diagnosis-specialized version of KAMET [2,3], as an alternative for creating knowledge-intensive systems attacking KE-specific risks. We describe here one of the most important characteristics of KAMET II which is the use of Protégé 2000 for implementing its CML models through ontologies.

1

Introduction

The KAMET [2,3] methodology was born during the last third of the nineties. The life-cycle of KAMET's first version was constituted mainly by two phases. The first one, analyzes and acquires the knowledge from different sources involved in the system. The second one, models and processes this knowledge. The mechanism proposed in KAMET for KA was based on progressive improvements of models, which allowed to refine knowledge and to obtain its optimal structure in an incremental fashion. KAMET was inspired by two basic ideas: the spiral model proposed by Boehm and the essence of cooperative processes. Both ideas are closely related to the concept of risk reduction. *

1

This project has been founded by CONACyT, as project number 33038-A, and Asociación Mexicana de Cultura, A.C. KAMET II is a project that is being carried out in collaboration with the SWI Group at Amsterdam University and Universidad Politécnica de Madrid.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 61-67, 2003.  Springer-Verlag Berlin Heidelberg 2003

62

Osvaldo Cairó and Julio César Alvarez

Fig. 1. KAMET II's Methodological Pyramid

Fig. 1 shows the Methodological Pyramid of KAMET II. KAMET-II's life cycle: it represents the methodology itself; it is the beginning to reach the goal. Auxiliary tools provide an automated or semi-automated support for the process and the methods. A KBS Model represents a KBS detailed description, in form of models. This has led KAMET to successful applications in the medicine, telecommunications and human resources areas. The integration of diagnosis specialization and the KAMET methodology ideas converges in the KAMET II methodology which is a specialized methodology in the diagnosis area by focusing on KA from multiple sources. Although KAMET provides a framework for managing the phase of knowledge modeling process in a systematic and disciplined way, the manner to obtain additional organizational objectives like predictability, consistency and improvement in quality and productivity is still not well defined within the KAMET Life-Cycle Model (LCM). KAMET II supports most aspects of a KBS development project, including organizational analysis, KA from multiple sources, conceptual modeling and user interaction. We will present in this section a brief description of Project Management (PM) in KAMET II. We want a PM activity that helps us measure as much as possible with the purpose of making better plans and reaching commitments. We believe that the best way to deal with these problems is to analyze not only the “what” and the “how” of a software system, but also “why” we are using it. This is best done by starting the KA process with the early requirements, where one analyzes the domain within which the system will operate, by studying how the use of the system will modify the environment, and by progressively refining this analysis down to the actual implementation of single knowledge modules. The purpose of KAMET II PM is the elimination of the project mismanagement and its consequences in KA projects [4, 5]. This new approach intends to build reliable KBS that fulfill customer and organization quality expectations by providing the project manager with effective knowledge and PM techniques. The purpose of this paper is not to present the PM dimension in detail, but it is important to emphasize this new key piece in KAMET II.

2

Diagnosis within KAMET II

Diagnosis differs from classification in the sense that the desired output is a malfunction of the system. In diagnosis the underlying knowledge typically contains knowledge about system behavior, such as a causal model. The output of diagnosis can take many forms: it can be a faulty component, a faulty state, or a causal chain. Diagnosis tasks are frequently encountered in the area of technical and medical systems.

KAMET II: An Extended Knowledge-Acquisition Methodology

63

Fig. 2. KAMET II's Environment

KAMET II tries to mitigate the problem of constructing solutions for diagnosis problems with a complete Problem-Solving Method (PSM) library. This will be carried out by substituting the elaboration of the answer from the beginning for tailoring and refinement of PSMs with predefined tasks. KAMET II will have as part of his assets a PSM library for technical diagnosis in order to carry out diagnosis task modeling by selecting a subset of assumptions and roles from this library. The objective is to convert KAMET II into a methodology that concentrates as much experience as possible in the form of a PSM library, so that whenever a diagnosis problem needs to be treated, the steps to tackle it are always available. Fig. 2 shows how the PSM Diagnosis Library is the base for the creation of knowledge models by KAMET II. The philosophy of KAMET II is sharing and reuse. But, not only PSMs are the reusable KE artifacts in KAMET II, but the knowledge models as well, as shown next.

3

KAMET II Models and Protégé 2000 Ontologies

Constructing knowledge systems is viewed as a modeling activity for developing structured knowledge and reasoning models. To ensure well-formed models, the use of some KE methodology is crucial. Additionally, reusing models can significantly reduce time and cost of building new applications [2]. The goal is to have shareable knowledge, by encoding domain knowledge using a standard vocabulary based on the ontology. The KAMET II CML is presented after a discussion of what the potential problems are in KA. Pure rule representation as well as an object-modeling language, data dictionaries, entity-relationship diagrams, among other methods are considered no longer sufficient neither for the purpose of system construction nor for that of knowledge representation. Knowledge is too rich to be represented with notations like the Unified Modeling Language (UML). This requires stronger modeling facilities. A knowledge modeling method should provide a rich vocabulary in which the expertise can be expressed in an appropriate way. Knowledge and reasoning should be modeled in such a way that models can be exploited in a very flexible fashion [2]. The KAMET II CML has three levels of abstraction. The first corresponds to structural constructors and components. The second level of abstraction corresponds to nodes and composition rules. The third level of abstraction corresponds to the global model [3].

64

Osvaldo Cairó and Julio César Alvarez

Fig. 3. KAMET II' implemented in a Protégé 2000 ontology

In the following lines, it will be described how the Protégé 2000 model can be used to implement KAMET II knowledge models visually by means of Protégé 2000 frame-based class model (or metaclass model, although it is not necessary for KAMET II as it is for CommonKADS [9]) and the Diagram_Entity implementation. The structural constructors and the structural components of the KAMET II CML are mapped in Protégé 2000 in the following way. An abstract subclass Construct of the abstract class: THING is created as a superclass of the Problem, Classification and Subdivision classes that implement the correspondent concepts in the CML. The same is done for the structural components creating an abstract subclass Component as the superclass of Symptom, Antecedent, Solution, Time, Value, Inaccuracy, Process, Formula and Examination. However, with the purpose of creating diagrammatic representations of knowledge models, composition rules need to be subclasses of the abstract class Connector in order to be used in a KAMETII Diagram instance. [9] should be consulted for a complete description of the Protégé 2000 knowledge model and its flexibility of adaptation to other knowledge models like KAMET II's. Nevertheless, the importance of adapting KAMET II to Protégé 2000 is because of the potential visual modeling available in the tool. Diagrams are one way to accomplish the latter goal. A diagram is, visually, a set of nodes and connectors that join the nodes. Underlying these, there are instances of classes. Nodes map to domain objects and connectors map to instances of Connector (a class the diagram widget uses to

KAMET II: An Extended Knowledge-Acquisition Methodology

65

represent important relationships). When users see a diagram, they see a high-level summary of a large number of instances, without seeing a lot of the details of any particular instance. That is why a Diagram Widget is presented. This widget has the representativeness above mentioned. The mapping permits the use of an automated aid to construct KAMET II models. Fig. 3 (left) shows the KAMET II in a Protégé 2000 ontology [9]. In Fig. 3 (right) it is presented the Diagram Widget for KAMET II, that is nothing but a special kind of form [9] to obtain instances for the classes (or concepts) involved in diagnosis-specific problems. This is the way to edit the KAMET II models. As it is visible in the left hierarchy, all the KAMET II diagrams will be instances of the KAMETIIDiagram class, which is a direct subclass of the Network class. Fig. 3 (right) shows the structural components and constructors and the composition rules. Developers used to the original graphical notation [2] will not have any problem getting used to the symbols implemented in Protégé 2000. It is important to mention that inaccuracies are represented as entities and not as probabilistic slots. So, better intrinsic notation and representation are necessary to have better working models. Probabilistic networks [10] need a more elaborate analysis before they could be expressed in these graphical terms. Fig. 4 shows a simple model in KAMET II CML modeled in Protégé 2000. The model expresses that the problem P1 can occur due to two different situations. In the first one, the model expresses that if the symptoms S1 and S2 are known to be true then we can deduce the problem P1 is true with probability 0.70. In the second one, the model shows that if symptoms S1 and S5 are observed then we can conclude that the problem is P1 with probability 0.70. On the other hand, we can deduce that the problem P3 is true with probability 0.40 is symptoms S1 and S4 are known to be true. Finally, we can reach a conclusion that the problem is P2 with probability 0.90 if problems P1 and P3 and the symptom S3 are observed.

Fig. 4. Electrical Diagnosis modeled in Protégé 2000

66

Osvaldo Cairó and Julio César Alvarez

Fig. 4 showed an example implemented in Protégé 2000 using the facilities it provides for diagram construction with the Diagram Widget in the Instances tab.

4

Conclusions

In this paper we have described various aspects of the new KE methodology KAMET II. The reasons why KAMET II is a complete diagnosis methodology through its PSM Diagnosis Library for the procedural knowledge part of KBS construction were discussed. Not only simple cause problems can be tackled but also a great variety of them can be solved using a PSM Library. It was also shown how the modeling phase of declarative knowledge in KAMET II can be carried out using the Protégé 2000 automated tool through ontologies. The purpose was to map the KAMET II CML models to the Protégé 2000 frame-based model in order to provide sharing and reuse to knowledge engineers. These objectives can be achieved in the tool thanks to the API it provides in order to reuse the knowledge stored in the knowledge representation schemes facilitated and by the OKBC compliance of the tool. It was also presented the PM dimension of the methodology in order to diminish KE-specific risks in KBS development.

References [1] [2] [3]

[4]

[5] [6] [7] [8]

Angele, J., Fensel, D., and Studer, R.: Developing Knowledge-BASED Systems with MIKE. Journal of Automated Software Engineering. Cairó, O.: A Comprehensive Methodol ogy for Knowledge Acquisition from Multiple Knowledge Sources. Expert Systems with Applications, 14(1998), 116. Cairó, O.: The KAMET Methodology: Content, Usage and Knowledge Modeling. In Gaines, B. and Musen, M., editors, Proceedings of the 11th Banff Knowledge Acquisition for Knowledge-Based Systems Workshop, pages 1-20. Department of Comp uter Science, University of Calgary, SRGD Publications Cairó, O., Barreiro, J., and Solsona, F.: Software Methodologies at Risk. In Fensel, D. and Studer, R., editors, 11th European Workshop on Knowledge Acquisition, Modeling and Management, volume 1621 of LNAI, pages 319-324. Springer Verlag. Cairó, O., Barreiro, J., and Solsona, F.: Risks Inside-out. In Cairó, O., Sucar, L. and Cantu, F., editors, MICAI 2000: Advances in Artificial Intelligence, volume 1793 of LNAI, pages 426-435. Springer Verlag. Domingue, J., Motta, E., and Watt, S.: The Emerging VITAL Workbench. Medsker, L., Tan, M., and Turban, E.: Knowledge Acquisition from Multiple Experts:Problems and Issues. Expert Systems with Applications, 9(1), 35-40. Schreiber, G., and Akkermans, H.: Knowledge Engineering and Management: the CommonKADS Methodology. MIT Press, Cambridge, Massachusetts, 1 999.

KAMET II: An Extended Knowledge-Acquisition Methodology

67

Schreiber, G., Crubézy, M., and Musen, M.: A Case Study in Using Protégé2000 as a Tool for CommonKADS. In Dieng, R. and Corby, O., editors, 12th International Conference, EKAW 2000, Juan-les-Pins, France. [10] van der Gaag, L., and Helsper, E.: Experiences with Modeling Issues in Building Probabilistic Networks. In Gómez-Pérez, A. and Benjamins, R., editors, 13th International Conference, EKAW 2002,volume 2473 of LNAI, pages 21-26. Springer Verlag.

[9]

Automated Knowledge Acquisition by Relevant Reasoning Based on Strong Relevant Logic* Jingde Cheng Department of Information and Computer Sciences, Saitama University Saitama, 338-8570, Japan [email protected]

Abstract. Almost all existing methodologies and automated tools for knowledge acquisition are somehow based on classical mathematical logic or its various classical conservative extensions. This paper proposes a new approach to knowledge acquisition problem: automated knowledge acquisition by relevant reasoning based on strong relevant logic. The paper points out why any of the classical mathematical logic, its various classical conservative extensions, and traditional relevant logics is not a suitable logical basis for knowledge acquisition, shows that strong relevant logic is a more hopeful candidate for the purpose, and establishes a conceptional foundation for automated knowledge acquisition by relevant reasoning based on strong relevant logic.

1

Introduction

From the viewpoint of knowledge engineering, knowledge acquisition is the purposive modeling process of discovering and learning knowledge about some particular subject from one or more knowledge sources, and then abstracting, formalizing, representing, and transferring the knowledge in some explicit and formal forms suitable to computation on computer systems [10]. Automated knowledge acquisition is concerned with the execution of computer programs that assist in knowledge acquisition. As Sestito and Dillon pointed out: The main difficulty in extracting knowledge from experts is that they themselves have trouble expressing or formalizing their knowledge. Experts also have a problem in describing their knowledge in terms that are precise, complete, and consistent enough for use in a computer program. This difficulty stems from the inherent nature of knowledge that constitutes human expertise. Such knowledge is often subconscious and may be approximate, incomplete, and inconsistent [15]. Therefore, the intrinsically characteristic task in knowledge *

This work is supported in part by The Ministry of Education, Culture, Sports, Science and Technology of Japan under Grant-in-Aid for Exploratory Research No. 09878061 and Grant-in-Aid for Scientific Research (B) No. 11480079.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 68-80, 2003.  Springer-Verlag Berlin Heidelberg 2003

Automated Knowledge Acquisition

69

acquisition is discovery rather than justification, and the task have to be performed under the condition that working with approximate, incomplete, inconsistent knowledge is the rule rather than the exception. As a result, if we want to establish a sound methodology for knowledge acquisition in knowledge engineering practices, we have to consider the issue how to discover new knowledge from one or more knowledge sources where approximateness, incompleteness, and inconsistency are existent in some degree. Until now, almost all existing methodologies and automated tools for knowledge acquisition are somehow based on classical mathematical logic (CML for short) or its various classical conservative extensions. This approach, however, may be suitable to searching and describing a formal proof of a previously specified statement, but not necessarily suitable to forming a new concept and discovering a new statement because the aim, nature, and role of CML is descriptive and non-predictive rather than prescriptive and predictive. This paper proposes a new approach to knowledge acquisition problem: knowledge acquisition by relevant reasoning based on strong relevant logic. The paper points out why any of the classical mathematical logic, its various classical conservative extensions, and traditional relevant logics is not a suitable logical basis for knowledge acquisition, shows that strong relevant logic is more a hopeful candidates for the purpose, and establishes a conceptional foundation for automated knowledge acquisition by relevant reasoning based on strong relevant logic.

2

Reasoning and Proving

Reasoning is the process of drawing new conclusions from given premises, which are already known facts or previously assumed hypotheses (Note that how to define the notion of “new” formally and satisfactorily is still a difficult open problem until now). Therefore, reasoning is intrinsically ampliative, i.e., it has the function of enlarging or extending some things, or adding to what is already known or assumed. In general, a reasoning consists of a number of arguments (or inferences) in some order. An argument (or inference) is a set of declarative sentences consisting of one or more sentences as its premises, which contain the evidence, and one sentence as its conclusion. In an argument, a claim is being made that there is some sort of evidential relation between its premises and its conclusion: the conclusion is supposed to follow from the premises, or equivalently, the premises are supposed to entail the conclusion. Therefore, the correctness of an argument is a matter of the connection between its premises and its conclusion, and concerns the strength of the relation between them (Note that the correctness of an argument depends neither on whether the premises are really true or not, nor on whether the conclusion is really true or not). Thus, there are some fundamental questions: What is the criterion by which one can decide whether the conclusion of an argument or a reasoning really does follow from its premises or not? Is there the only one criterion, or are there many criteria? If there are many criteria, what are the intrinsic differences between them? It is logic that deals with the validity of argument and reasoning in general.

70

Jingde Cheng

A logically valid reasoning is a reasoning such that its arguments are justified based on some logical validity criterion provided by a logic system in order to obtain correct conclusions (Note that here the term “correct” does not necessarily mean “true.”). Today, there are so many different logic systems motivated by various philosophical considerations. As a result, a reasoning may be valid on one logical validity criterion but invalid on another. For example, the classical account of validity, which is one of fundamental principles and assumptions underlying CML and its various conservative extensions, is defined in terms of truth-preservation (in some certain sense of truth) as: an argument is valid if and only if it is impossible for all its premises to be true while its conclusion is false. Therefore, a classically valid reasoning must be truth-preserving. On the other hand, for any correct argument in scientific reasoning as well as our everyday reasoning, its premises must somehow be relevant to its conclusion, and vice versa. The relevant account of validity is defined in terms of relevance as: for an argument to be valid there must be some connection of meaning, i.e., some relevance, between its premises and its conclusion. Obviously, the relevance between the premises and conclusion of an argument is not accounted for by the classical logical validity criterion, and therefore, a classically valid reasoning is not necessarily relevant. Proving is the process of finding a justification for an explicitly specified statement from given premises, which are already known facts or previously assumed hypotheses. A proof is a description of a found justification. A logically valid proving is a proving such that it is justified based on some logical validity criterion provided by a logic system in order to obtain a correct proof. The most intrinsic difference between reasoning and proving is that the former is intrinsically prescriptive and predictive while the latter is intrinsically descriptive and non-predictive. The purpose of reasoning is to find some new conclusion previously unknown or unrecognized, while the purpose of proving is to find a justification for some specified statement previously given. Proving has an explicitly given target as its goal while reasoning does not. Unfortunately, until now, many studies in Computer Science and Artificial Intelligence disciplines still confuse proving with reasoning. Discovery is the process to find out or bring to light of that which was previously unknown. For any discovery, both the discovered thing and its truth must be unknown before the completion of discovery process. Since reasoning is the only way to draw new conclusions from given premises, there is no discovery process that does not invoke reasoning. As we have mentioned, the intrinsically characteristic task in knowledge acquisition is discovery rather than justification. Therefore, the task is concerning reasoning rather than proving. Below, let us consider the problem what logic system can satisfactorily underlie knowledge discovery.

3

The Notion of Conditional as the Heart of Logic

What is logic? Logic is a special discipline which is considered to be the basis for all other sciences, and therefore, it is a science prior to all others, which contains the

Automated Knowledge Acquisition

71

ideas and principles underlying all sciences [9, 16]. Logic deals with what entails what or what follows from what, and aims at determining which are the correct conclusions of a given set of premises, i.e., to determine which arguments are valid. Therefore, the most essential and central concept in logic is the logical consequence relation that relates a given set of premises to those conclusions, which validly follow from the premises. In general, a formal logic system L consists of a formal language, called the object language and denoted by F(L), which is the set of all well-formed formulas of L, and a logical consequence relation, denoted by meta-linguistic symbol |−L, such that for P ⊆ F(L) and c ∈ F(L), P |−L c means that within the framework of L, c is a valid conclusion of premises P, i.e., c validly follows from P. For a formal logic system (F(L), |−L), a logical theorem t is a formula of L such that φ |−L t where φ is the empty set. We use Th(L) to denote the set of all logical theorems of L. Th(L) is completely determined by the logical consequence relation |−L. According to the representation of the logical consequence relation of a logic, the logic can be represented as a Hilbert style formal system, a Gentzen natural deduction system, a Gentzen sequent calculus system, or other type of formal system. A formal logic system L is said to be explosive if and only if {A, ¬A} |−L B for any two different formulas A and B; L is said to be paraconsistent if and only if it is not explosive. Let (F(L), |−L) be a formal logic system and P ⊆ F(L) be a non-empty set of sentences (i.e., closed well-formed formulas). A formal theory with premises P based on L, called a L-theory with premises P and denoted by TL(P), is defined as TL(P) =df Th(L) ∪ ThLe(P), and ThLe(P) =df {et | P |−L et and et ∉ Th(L)} where Th(L) and ThLe(P) are called the logical part and the empirical part of the formal theory, respectively, and any element of ThLe(P) is called an empirical theorem of the formal theory. A formal theory TL(P) is said to be directly inconsistent if and only if there exists a formula A of L such that both A ∈ P and ¬A ∈ P hold. A formal theory TL(P) is said to be indirectly inconsistent if and only if it is not directly inconsistent but there exists a formula A of L such that both A ∈ TL(P) and ¬A ∈ TL(P). A formal theory TL(P) is said to be consistent if and only if it is neither directly inconsistent nor indirectly inconsistent. A formal theory TL(P) is said to be explosive if and only if A ∈ TL(P) for arbitrary formula A of L; TL(P) is said to be paraconsistent if and only if it is not explosive. An explosive formal theory is not useful at all. Therefore, any meaningful formal theory should be paraconsistent. Note that if a formal logic system L is explosive, then any directly or indirectly inconsistent L-theory TL(P) must be explosive. In the literature of mathematical, natural, social, and human sciences, it is probably difficult, if not impossible, to find a sentence form that is more generally used for describing various definitions, propositions, and theorems than the sentence form of “if ... then ... .” In logic, a sentence in the form of “if ... then ...” is usually called a conditional proposition or simply conditional which states that there exists a relation of sufficient condition between the “if” part and the “then” part of the sentence. Scientists always use conditionals in their descriptions of various definitions, propositions, and theorems to connect a concept, fact, situation or conclusion to its sufficient conditions. The major work of almost all scientists is to discover some sufficient condition relations between various phenomena, data, and laws in their research

72

Jingde Cheng

fields. Indeed, Russell 1903 has said, “Pure Mathematics is the class of all propositions of the form ‘p implies q,' where p and q are propositions containing one or more variables, the same in the two propositions, and neither p nor q contains any constants except logical constants” [14]. In general, a conditional must concern two parts which are connected by the connective “if ... then ... ” and called the antecedent and the consequent of that conditional, respectively. The truth of a conditional depends not only on the truth of its antecedent and consequent but also, and more essentially, on a necessarily relevant and conditional relation between them. The notion of conditional plays the most essential role in reasoning because any reasoning form must invoke it, and therefore, it is historically always the most important subject studied in logic and is regarded as the heart of logic [1]. In fact, from the age of ancient Greece, the notion of conditional has been discussed by the ancients of Greek. For example, the extensional truth-functional definition of the notion of material implication was given by Philo of Megara in about 400 B.C. [11, 16]. When we study and use logic, the notion of conditional may appear in both the object logic (i.e., the logic we are studying) and the meta-logic (i.e., the logic we are using to study the object logic). In the object logic, there usually is a connective in its formal language to represent the notion of conditional, and the notion of conditional, usually represented by a meta-linguistic symbol, is also used for representing a logical consequence relation in its proof theory or model theory. On the other hand, in the meta-logic, the notion of conditional, usually in the form of natural language, is used for defining various meta-notions and describing various meta-theorems about the object logic. From the viewpoint of object logic, there are two classes of conditionals. One class is empirical conditionals and the other class is logical conditionals. For a logic, a conditional is called an empirical conditional of the logic if its truth-value, in the sense of that logic, depends on the contents of its antecedent and consequent and therefore cannot be determined only by its abstract form (i.e., from the viewpoint of that logic, the relevant relation between the antecedent and the consequent of that conditional is regarded to be empirical); a conditional is called a logical conditional of the logic if its truth-value, in the sense of that logic, depends only on its abstract form but not on the contents of its antecedent and consequent, and therefore, it is considered to be universally true or false (i.e., from the viewpoint of that logic, the relevant relation between the antecedent and the consequent of that conditional is regarded to be logical). A logical conditional that is considered to be universally true, in the sense of that logic, is also called an entailment of that logic. Indeed, the most intrinsic difference between various different logic systems is to regard what class of conditionals as entailments, as Diaz pointed out: “The problem in modern logic can best be put as follows: can we give an explanation of those conditionals that represent an entailment relation?” [7]

Automated Knowledge Acquisition

4

73

The Logical Basis for Automated Knowledge Acquisition

Any science is established based on some fundamental principles and assumptions such that removing or replacing one of them by a new one will have a great influence on the contents of the science and even lead to creating a completely new branch of the science. CML was established in order to provide formal languages for describing the structures with which mathematicians work, and the methods of proof available to them; its principal aim is a precise and adequate understanding of the notion of mathematical proof. Given its mathematical method, it must be descriptive rather than prescriptive, and its description must be idealized. CML was established based on a number of fundamental assumptions. Some of the assumptions concerning with our subject are as follows: The classical abstraction: The only properties of a proposition that matter to logic are its form and its truth-value. The Fregean assumption: The truth-value of a proposition is determined by its form and the truth-values of its constituents. The classical account of validity: An argument is valid if and only if it is impossible for all its premises to be true while its conclusion is false. The principle of bivalence: There are exactly two truth-values, TRUE and FALSE. Every declarative sentence has one or other, but not both, of these truth-values. The classical account of validity is the logical validity criterion of CML by which one can decide whether the conclusion of an argument or a reasoning really does follow from its premises or not in the framework of CML. However, since the relevance between the premises and conclusion of an argument is not accounted for by the classical validity criterion of CML, a reasoning based on CML is not necessarily relevant, i.e., its conclusion may be not relevant at all, in the sense of meaning and context, to its premises. In other words, in the framework of CML, even if a reasoning is classically valid, the relevance between its premises and its conclusion cannot be guaranteed necessarily. Note that this proposition is also true in the framework of any classical conservative extension or non-classical alternative of CML where the classical account of validity is adopted as the logical validity criterion. On the other hand, taking the above assumptions into account, in CML, the notion of conditional, which is intrinsically intensional but not truth-functional, is represented by the truth-functional extensional notion of material implication (denoted by → in this paper) that is defined as A→B =df ¬(A∧¬B) or A→B =df ¬A∨B. This definition of material implication, with the inference rule of Modus Ponens for material implication (from A and A→B to infer B), can adequately satisfy the truthpreserving requirement of CML, i.e., the conclusion of a classically valid reasoning based on CML must be true (in the sense of CML) if all premises of the reasoning are true (in the sense of CML). This requirement is basic and adequate for CML to be used as a formal description tool by mathematicians. However, the material implication is intrinsically different from the notion of conditional in meaning (semantics). It is no more than an extensional truth-function of its antecedent and consequent but does not require that there is a necessarily relevant

74

Jingde Cheng

and conditional relation between its antecedent and consequent, i.e., the truth-value of the formula A→B depends only on the truth-values of A and B, though there could exist no necessarily relevant and conditional relation between A and B. It is this intrinsic difference in meaning between the notion of material implication and the notion of conditional that leads to the well-known “implicational paradox problem” in CML. The problem is that if one regards the material implication as the notion of conditional and regards every logical theorem of CML as an entailment or valid reasoning form, then a great number of logical axioms and logical theorems of CML, such as A→(B→A), B→(¬A∨A), ¬A→(A→B), (¬A∧A)→B, (A→B)∨(¬A→B), (A→B)∨(A→¬B), (A→B)∨(B→A), ((A∧B)→C)→((A→C)∨(B→C)), and so on, present some paradoxical properties and therefore they have been referred to in the literature as “implicational paradoxes” [1]. Because all implicational paradoxes are logical theorems of any CML-theory TCML(P), for a conclusion of a reasoning from a set P of premises based on CML, we cannot directly accept it as a correct conclusion in the sense of conditional, even if each of the given premises is regarded to be true and the conclusion can be regarded to be true in the sense of material implication. For example, from any given premise A ∈ P, we can infer B→A, C→A, ... where B, C, ... are arbitrary formulas, by using the logical axiom A→(B→A) of CML and Modus Ponens for material implication, i.e., B→A ∈ TCML(P), C→A ∈ TCML(P), ... for any A ∈ TCML(P). However, from the viewpoint of scientific reasoning as well as our everyday reasoning, these inferences cannot be regarded to be valid in the sense of conditional because there may be no necessarily relevant and conditional relation between B, C, ... and A and therefore we cannot say “if B then A,” “if C then A,” and so on. Obviously, no scientists did or will reason in such a way in their scientific discovery. This situation means that from the viewpoint of conditional or entailment, the truth-preserving property of reasoning based on CML is meaningless. Note that any classical conservative extension or non-classical alternative of CML where the notion of conditional is directly or indirectly represented by the material implication has the similar problems as the above problems in CML. Consequently, in the framework of CML and its various classical or non-classical conservative extensions, even if a reasoning is classically valid, neither the necessary relevance between its premises and conclusion nor the truth of its conclusion in the sense of conditional can be guaranteed necessarily. From the viewpoint to regard reasoning as the process of drawing new conclusions from given premises, any meaningful reasoning should be ampliative but not circular and/or tautological, i.e., the truth of conclusion of the reasoning should be recognized after the completion of the reasoning process but not be invoked in deciding the truth of premises of the reasoning. As an example, let us see the most typical human logical reasoning form, Modus Ponens. The natural language representation of Modus Ponens may be “if A holds then B holds, now A holds, therefore B holds.” When we reason using Modus Ponens, what we know? We know “if A holds then B holds” and “A holds.” Before the reasoning is performed, we do not know whether or not “B holds.” If we know, then we would not need to reason at all. Therefore, Modus Ponens should be ampliative but not circular and/or tautological. While, how can we know “B holds” by using Modus Ponens? Indeed, by using Modus Ponens, we can

Automated Knowledge Acquisition

75

know “B holds,” which is unknown until the reasoning is performed, based on the following reasons: (i) “A holds,” (ii) “There is no case such that A holds but B does not hold,” and (iii) we know (ii) without investigating either “whether A holds or not” or “whether B holds or not.” Note that the Wright-Geach-Smiley criterion (see below) for entailment is corresponding to the above (ii) and (iii). From this example, we can see that the key point in ampliative and non-circular reasoning is the primitive and intensional relevance between the antecedent and consequent of a conditional. Because the material implication in CML is an extensional truth-function of its antecedent and consequent but not requires the existence of a necessarily relevant relation between its antecedent and consequent, a reasoning based on the logic must be circular and/or tautological. For example, Modus Ponens for material implication is usually represented in CML as “from A and A→B to infer B.” According to the extensional truth-functional semantics of the material implication, if we know “A is true” but do not know the truth-value of B, then we cannot decide the truth-value of “A→B.” In order to know the truth-value of B using Modus Ponens for material implication, we have to know the truth-value of B before the reasoning is performed! Obviously, Modus Ponens for material implication is circular and/or tautological if it is used as a reasoning form, and therefore, it is not a natural representation of Modus Ponens. Moreover, in general, our knowledge about a domain as well as a scientific discipline may be incomplete and inconsistent in many ways, i.e., it gives us no evidence for deciding the truth of either a proposition or its negation, and it directly or indirectly includes some contradictions. Therefore, reasoning with incomplete (and some time inconsistent) information and/or knowledge is the rule rather than the exception in our everyday real-life situations and almost all scientific disciplines. Also, even if our knowledge about a domain or scientific discipline seems to be consistent at present, in future, we may find a new fact or rule that is inconsistent with our known knowledge, i.e., we find a contradiction. In these cases, we neither doubt the “logic” we used in our everyday logical thinking nor reason out anything from the contradictions, but we must consider that there are some wrong things in our knowledge and will investigate the causes of the contradictions, i.e., we do reason under inconsistency in order to detect and remove the causes of the contradictions. Indeed, in scientific research, the detection and explanation of an inconsistency between a new fact and known knowledge often leads to formation of new concepts or discovery of new principles. How to reason with inconsistent knowledge is an important issue in scientific discovery and theory formation. For a paraconsistent logic with Modus Ponens as an inference rule, the paraconsistence requires that the logic does not have “(¬A∧A)→B” as a logical theorem where “A” and “B” are any two different formulas and “→” is the relation of implication used in Modus Ponens. If a logic is not paraconsistent, then infinite propositions (even negations of those logical theorems of the logic) may be reasoned out based on the logic from a set of premises that directly or indirectly include a contradiction. However, CML assumes that all the information is on the table before any deduction is performed. Moreover, it is well known that CML is explosive but not paraconsistent, and therefore, any directly or indirectly inconsistent CML-theory TCML(P) must be explosive. This is because CML uses Modus Ponens for material implication as its inference rule, and has “(¬A∧A)→B”

76

Jingde Cheng

as a logical theorem, which, in fact, is the most typical implicational paradox. In fact, reasoning under inconsistency is impossible within the framework of CML. Through the above discussions, we have seen that a reasoning based on CML is not necessarily relevant, the classical truth-preserving property of a reasoning based on CML is meaningless in the sense of conditional, a reasoning based on CML must be circular and/or tautological but not ampliative, and reasoning under inconsistency is impossible within the framework of CML. These facts are also true to those classical or non-classical conservative extensions of CML. What these facts tell us is that CML and its various classical or non-classical conservative extensions are not suitable logical basis for automated knowledge acquisition. Traditional relevant logics ware constructed during the 1950s in order to find a mathematically satisfactory way of grasping the elusive notion of relevance of antecedent to consequent in conditionals, and to obtain a notion of implication which is free from the so-called “paradoxes” of material and strict implication [1, 2, 8, 12, 13]. Some major traditional relevant logic systems are “system E of entailment”, “system R of relevant implication”, and “system T of ticket entailment”. A major feature of these relevant logics is that they have a primitive intensional connective to represent the notion of conditional and their logical theorems include no implicational paradoxes. Von Wright, Geach, and Smiley suggested some informal criteria for the notion of entailment, i.e., the so-called “Wright-Geach-Smiley criterion” for entailment: “A entails B, if and only if, by means of logic, it is possible to come to know the truth of A→B without coming to know the falsehood of A or the truth of B” [1]. However, it is hard until now to know exactly how to formally interpret such epistemological phrases as “coming to know” and “getting to know” in the context of logic. Anderson and Belnap proposed variable-sharing as a necessary but not sufficient formal condition for the relevance between the antecedent and consequent of an entailment. The underlying principle of these relevant logics is the relevance principle, i.e., for any entailment provable in E, R, or T, its antecedent and consequent must share a sentential variable. Variable-sharing is a formal notion designed to reflect the idea that there be a meaning-connection between the antecedent and consequent of an entailment [1, 2, 12, 13]. It is this relevance principle that excludes those implicational paradoxes from logical axioms or theorems of relevant logics. However, although the traditional relevant logics have rejected those implicational paradoxes, there still exist some logical axioms or theorems in the logics, which are not so natural in the sense of conditional. Such logical axioms or theorems, for instance, are (A∧B)⇒A, (A∧B)⇒B, (A⇒B)⇒((A∧C)⇒B), A⇒(A∨B), B⇒(A∨B), (A⇒B)⇒(A⇒(B∨C)) and so on, where ⇒ denotes the primitive intensional connective in the logics to represent the notion of conditional. The present author named these logical axioms or theorems ‘conjunction-implicational paradoxes' and ‘disjunction-implicational paradoxes' [3, 4, 6]. For example, from any given premise A⇒B, we can infer (A∧C)⇒B, (A∧C∧D)⇒B, and so on by using logical theorem (A⇒B)⇒((A∧C)⇒B) of T, E, and R and Modus Ponens for conditional. However, from the viewpoint of scientific reasoning as well as our everyday reasoning, these inferences cannot be regarded as valid in the sense of conditional because there may be no necessarily relevant and conditional relation between C, D, ... and B and therefore we cannot say ‘if A and C then B', ‘if A and C and D then B', and so on.

Automated Knowledge Acquisition

77

In order to establish a satisfactory logic calculus of conditional to underlie relevant reasoning, the present author has proposed some strong relevant logics (or strong relevance logics), named Rc, Ec, and Tc [3, 4, 6]. The logics require that the premises of an argument represented by a conditional include no unnecessary and needless conjuncts and the conclusion of that argument includes no unnecessary and needless disjuncts. As a modification of traditional relevant logics R, E, and T, strong relevant logics Rc, Ec, and Tc rejects all conjunction-implicational paradoxes and disjunction-implicational paradoxes in R, E, and T, respectively. What underlies the strong relevant logics is the strong relevance principle: If A is a theorem of Rc, Ec, or Tc, then every sentential variable in A occurs at least once as an antecedent part and at least once as a consequent part. Since the strong relevant logics are free of not only implicational paradoxes but also conjunction-implicational and disjunctionimplicational paradoxes, in the framework of strong relevant logics, if a reasoning is valid, then both the relevance between its premises and its conclusion and the validity of its conclusion in the sense of conditional can be guaranteed in a certain sense of strong relevance. The strong relevant logics are hopeful candidates for the fundamental logic to satisfactorily underlie automated knowledge acquisition. First, the strong relevant logics can certainly underlie relevant reasoning in a certain sense of strong relevance. Second, a reasoning based on the strong relevant logics is truth-preserving in the sense of the intensional primitive semantics of the conditional. Third, since the WrightGeach-Smiley criterion for entailment is accounted for by the notion of conditional, a reasoning based on the strong relevant logics is ampliative but not circular and/or tautological. Finally, the strong relevant logics can certainly underlie paraconsistent reasoning.

5

Automated Knowledge Acquisition by Relevant Reasoning

Knowledge in various domains is often represented in the form of conditional. From the viewpoint of logical calculus of conditional, the problem of automated knowledge acquisition can be regarded as the problem of automated (empirical) theorem finding from one or more knowledge sources. Since a formal theory TL(P) is generally an infinite set of formulas, even though premises P are finite, we have to find some method to limit the range of candidates for “new and interesting theorems” to a finite set of formulas. The strategy the present author adopted is to sacrifice the completeness of knowledge representation and reasoning to get the finite set of candidates. This is based on the author's conjecture that almost all “new and interesting theorems” of a formal theory can be deduced from the premises of that theory by finite inference steps concerned with finite number of low degree entailments. Let (F(L), |−L) be a formal logic system and k be a natural number. The kth degree fragment of L, denoted by Thk(L), is a set of logical theorems of L which is inductively defined as follows (in the terms of Hilbert style formal system): (1) if A is a jth (j ≤ k) degree formula and an axiom of L, then A ∈ Thk(L), (2) if A is a jth (j ≤ k) degree formula which is the result of applying an inference rule of L to some mem-

78

Jingde Cheng

bers of Thk(L), then A ∈ Thk(L), and (3) Nothing else are members of Thk(L), i.e., only those obtained from repeated applications of (1) and (2) are members of Thk(L). Obviously, the definition of the kth degree fragment of logic L is constructive. Note that the kth degree fragment of logic L does not necessarily include all kth degree logical theorems of L because it is possible for L that deductions of some kth degree logical theorems of L must invoke those logical theorems whose degrees are higher than k. On the other hand, the following holds obviously: Th0(L) ⊂ Th1(L) ⊂ ... Thk−1(L) ⊂ Thk(L) ⊂ Thk+1(L) ⊂ ... Let (F(L), |−L) be a formal logic system, P ⊂ F(L), and k and j be two natural numbers. A formula A is said to be jth-degree-deducible from P based on Thk(L) if and only if there is an finite sequence of formulas f1, ..., fn such that fn = A and for all i (i ≤ n) (1) fi ∈ Thk(L), or (2) fi ∈ P and the degree of fi is not higher than j, or (3) fi whose degree is not higher than j is the result of applying an inference rule to some members fj1, ..., fjm (j1, ..., jm < i) of the sequence. If P ≠ φ, then the set of all formulas which are jth-degree-deducible from P based on Thk(L) is called the jth degree fragment of the formal theory with premises P based on Thk(L), denoted by T j th k Thk(L)(P). A formula is said to be j -degree-deductive from P based on Th (L) if and only if it is jth-degree-deducible from P based on Thk(L) but not (j−1)th-degreededucible from P based on Thk(L). Note that in the above definitions, we do not require (j ≤ k). The notion of jth-degree-deductive can be used as a metric to measure the difficulty of deducing an empirical theorem from given premises P based on logic L. The difficulty is relative to the complexity of problem being investigated as well as the strength of underlying logic L. Based on the above definitions, we have the following result. Let TSRL(P) be a formal theory where SRL is a strong relevant logic, and k and j be two natural numbers. If P is finite, then T jThk(SRL)(P) must be finite. This is also true even if T jThk(SRL)(P) is directly or indirectly inconsistent. This means that there exists a fixed point P′ such that P ⊆ P′ and T jThk(SRL)(P′) = P, even if T jThk(SRL)(P) is directly or indirectly inconsistent. The above result about SRL does not hold for those paradoxical logics such as CML and its various classical or non-classical conservative extensions, and traditional (weak) relevant logics T, E and R because these logics accept implicational, conjunction-implicational, or disjunction-implicational paradoxes as logical theorems. Let TSRL(P) be a formal theory where SRL is a strong relevant logic and k be a natural number. SRL is said to be kth-degree-complete for TSRL(P) if and only if for any empirical theorem et of ThSRLe(P), there is a finite natural number j such et is jthdegree-deducible from P based on Thk(SRL), i.e., all empirical theorems of ThSRLe(P) are some how deducible from P based on Thk(SRL). Having SRL where SRL is a strong relevant logic as the fundamental logic, and constructing, say the 3rd, degree fragment of SRL previously, for any given premises P, we can find the fixed point T jTh3(SRL)(P). Since SRL is free of implicational, conjunction-implicational, and disjunction-implicational paradoxes, as a result, we can obtain finite meaningful empirical theorems as candidates for “new and interesting theorems” of T jTh3(SRL)(P). Moreover, if SRL is 3rd-degree-complete for TSRL(P),

Automated Knowledge Acquisition

79

then we can obtain all candidates for “new and interesting theorems” of TSRL(P). These are also true even if TSRL(P) is inconsistent. Of course, maybe SRL is not 3rddegree-complete for TSRL(P). In this case, a fragment of SRL whose degree is higher then 3 must be used in order to find more empirical theorems. Until now, we have established a conceptional foundation for the empirical theorem finding within the framework of strong relevant logic. The next problem is how to develop programs to find empirical theorems automatically. Since any backward and/or refutation deduction system cannot serve as an autonomous reasoning mechanism to form and/or discover some completely new things, what we need is an autonomous forward reasoning system. We are developing an automated forward deduction system for general-purpose entailment calculus, named EnCal, which provides its users with the following major facilities [5]. For a logic L which may be a propositional logic, a first-order predicate logic, or a second-order predicate logic, a non-empty set P of formulas as premises, a natural number k (usually k < 5), and a natural number j all specified by the user, EnCal can do the following tasks: (1) reason out all logical theorem schemata of the kth degree fragment of L, (2) verify whether or not a formula is a logical theorem of the kth degree fragment of L, if yes, then give the proof, (3) reason out all empirical theorems of the jth degree fragment of the formal theory with premises P based on Thk(L), and (4) verify whether or not a formula is an empirical theorem of the jth degree fragment of the formal theory with premises P based on Thk(L), if yes, then give the proof. Now, for one or more given knowledge sources, we can represent the explicitly known knowledge in logical formulas at first, and then regard the set of logical formulas as the set of premises of a formal theory for the subject under investigation, use relevant reasoning based on strong relevant logic and EnCal to find new knowledge from the premises.

References [1] [2] [3] [4]

[5]

Anderson, A.R., Belnap Jr., N.D.: Entailment: The Logic of Relevance and Necessity, Vol. I. Princeton University Press, Princeton (1975) Anderson, A.R., Belnap Jr., N.D., Dunn, J. M.: Entailment: The Logic of Relevance and Necessity, Vol. II. Princeton University Press, Princeton (1992) Cheng, J.: Logical Tool of Knowledge Engineering: Using Entailment Logic rather than Mathematical Logic. Proc. ACM 19th Annual Computer Science Conference. ACM Press, New York (1991) 228-238 Cheng, J.: The Fundamental Role of Entailment in Knowledge Representation and Reasoning. Journal of Computing and Information, Vol. 2, No. 1, Special Issue: Proceedings of the 8th International Conference of Computing and Information. Waterloo (1996) 853-873 Cheng, J.: EnCal: An Automated Forward Deduction System for GeneralPurpose Entailment Calculus. In: Terashima, N., Altman, E. (eds.): Advanced IT Tools, Proc. IFIP World Conference on IT Tools, IFIP 96 − 14th World Computer Congress. CHAPMAN & HALL , London Weinheim New York Tokyo Melbourne Madras (1996) 507-514

80

Jingde Cheng

[6]

Cheng, J.: A Strong Relevant Logic Model of Epistemic Processes in Scientific Discovery. In: Kawaguchi, E., Kangassalo, H., Jaakkola, H., Hamid, I.A. (eds.): Information Modelling and Knowledge Bases XI. IOS Press, Amsterdam Berlin Oxford Tokyo Washington DC (2000) 136-159 Diaz, M.R.: Topics in the Logic of Relevance. Philosophia Verlag, Munchen (1981) Dunn, J.M., Restall, G.: Relevance Logic. In: Gabbay, D., Guenthner, F. (eds.): Handbook of Philosophical Logic, 2nd Edition, Vol. 6. Kluwer Academic, Dordrecht Boston London (2002) 1-128 Godel, K.: Russell's Mathematical Logic. In: Schilpp (ed.): The Philosophy of Bertrand Russell. Open Court Publishing Company, Chicago (1944) Hayes-Roth, F., Waterman, D.A., Lenat, D.B. (eds.): Building Expert Systems. Addison-Wesley, Boston (1983) Kneale, W., Kneale, M.: The Development of Logic. Oxford University Press, Oxford (1962) Mares, E.D., Meyer, R.K.: Relevant Logics. In: Goble, L. (ed.): The Blackwell Guide to Philosophical Logic. Blackwell, Oxford (2001) 280-308 Read, S.: Relevant Logic: A Philosophical Examination of Inference. Basil Blackwell, Oxford (1988) Russell, B.: The Principles of Mathematics. 2nd edition. Cambridge University Press, Cambridge (1903, 1938). Norton Paperback edition. Norton, New York London (1996) Sestito, S., Dillon, T.S.: Automated Knowledge Acquisition. Prentice Hall, Upper Saddle River (1994) Tarski, A.: Introduction to Logic and to the Methodology of the Deductive Sciences. 4th edition, Revised. Oxford University Press, Oxford (1941, 1946, 1965, 1994)

[7] [8] [9] [10] [11] [12] [13] [14] [15] [16]

CONCEPTOOL: Intelligent Support to the Management of Domain Knowledge Ernesto Compatangelo and Helmut Meisel Department of Computing Science, University of Aberdeen AB24 3UE Scotland, UK {compatan,hmeisel}@csd.abdn.ac.uk

Abstract. We have developed ConcepTool, a system which supports the modelling and the analysis of expressive domain knowledge and thus of different kinds of application ontologies. The knowledge model of ConcepTool includes semantic constructors (e.g. own slots, enumerations, whole-part links, synonyms) separately available in most framebased, conceptual, and lexical models. Domain knowledge analysis in ConcepTool explicitly takes into account the differences between distinct categories of concepts such as classes, associations, and processes. Moreover, our system uses lexical and heuristic inferences in conjunction with logical deductions based on subsumption in order to improve the quantity and the quality of analysis results. ConcepTool can also perform approximate reasoning by ignoring in a controlled way selected parts of the expressive power of the domain knowledge under analysis.

1

Context, Background and Motivations

Knowledge technologies are based on the concept of a “lifecycle” that encompasses knowledge acquisition, modelling, retrieval, maintenance, sharing, and reuse. Domain knowledge is currently a major application area for the lifecycleoriented approach to knowledge management [15]. This area includes most ontologies, which are formal shared specifications of domain knowledge concepts [6]. Domain knowledge is continuously created and modified over time by diverse actors. The effective management of this knowledge thus needs its continuous Verification and Validation (V&V). Verification analyses whether the modelling intentions are met and thus the intended meaning matches the actual one. Validation analyses whether there is consistency, and thus lack of contradictions. In practice, sizeable domain knowledge bodies cannot be properly verified and validated (i.e. they cannot be semantically analysed ) without a computerbased support. However, semantic analysis requires computers to understand the structure and the lexicon of this knowledge, thus making its implications explicit (verification) and detecting inconsistencies in it (validation). In recent years, Description Logics (DLs) were proposed as a family of languages for representing domain knowledge in a way that enables automated reasoning about it [1]. However, DL reasoners (e.g. FaCT [10], (Power)Loom [12], V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 81–88, 2003. c Springer-Verlag Berlin Heidelberg 2003


82

Ernesto Compatangelo and Helmut Meisel

CLASSIC [3], RACER [9]) and, consequently, the analysis tools based on them (e.g. i•COM [8], OilEd [2]) do not fully support domain knowledge V&V. There are two main reasons why DL-based systems do not fully support the verification and the validation of domain knowledge (and application ontologies): – They do not really operate at the conceptual level, as DL reasoning algorithms fail to distinguish between different deductive mechanisms for diverse categories of concepts. For instance, the differences between hierarchies of classes and hierarchies of functions are not taken into account [4]. – They do not complement taxonomic deductions with other logical, lexical and heuristic inferences in order to maximise conceptual analysis. Therefore, they do not detect further (potential) correlations between concepts, attributes, associative links, and partonomic links [5]. We have developed ConcepTool, the Intelligent Knowledge Management Environment (IKME) described in this paper, in order to overcome the above drawbacks of current DL-based tools for domain knowledge analysis.

2

The Knowledge Model of CONCEPTOOL

Existing modelling-oriented environments aim to provide an adequate expressive power to represent and combine a rich variety of knowledge. Modelling-oriented focus on expressiveness is not influenced by the potential undecidability or by the potential complexity of semantic analysis. Conversely, existing analysis-oriented environments aim to provide an adequate set of specialised reasoning services within a decidable, complete and computationally tractable framework. However, current analysis focus on worst-case decidability, completeness and tractability actually limits the allowed knowledge expressiveness. Research on ontologies highlighted that expressive domain knowledge models are needed to enable sharing and reuse in distributed heterogeneous contexts [7]. This implies allowing incomplete (approximate) deductions whenever decidable, complete and tractable reasoning is not theoretically or practically possible [14]. ConcepTool provides a “reasonable framework” for domain knowledge which supports expressive modelling and “heterogeneous” analysis. This framework is based on a modular approach that (i) expresses domain concepts by composing basic semantic constructors and (ii) combines diverse kinds of specialised inferences about concepts. The constructors that characterise the knowledge model of ConcepTool are described below, while the heterogeneous approach to reasoning used in our system is discussed in the following section. In devising the knowledge model (and the expressiveness) of ConcepTool, we focused on the following aspects which enhance its semantic interoperability: – Allow the analyst to specify domain knowledge at the “conceptual level”, using different concept types such as classes, associations, functions, goals. – Include constructors used in frame-based conceptual, and lexical knowledge models (e.g. own slots, enumerations, whole-part links, synonyms).

ConcepTool: Intelligent Support to the Management of Domain Knowledge

83

– Introduce the notion of individual, which generalises the notion of instance to denote an element that is not necessarily a member of an explicitly defined concept in the considered domain. We describe the ConcepTool knowledge model using the resource ontology in Figure 1, which integrates semantic constructors from frame-based, DL-based, conceptual, and lexical models. This ontology includes three kinds of elements, namely concepts (i.e. class and individual frames), roles (i.e. slots) and properties of inheritable concept-role links (i.e. facets of inheritable slots). All elements have non-inheritable properties (own slots). Concepts also have inheritable properties (template slots), which constrain them to other concepts as specified by facets. Concepts can be interpreted either as individuals or as sets of individuals (i.e. classes, associations, functions, etc.). Roles are interpreted as sets of binary links. Concept-role links (slot attachments) are interpreted as constraints of inheritable concept properties (e.g. attributes, parts, associative links, global axioms). Concept and role constructors (slots and facets) are interpreted according to the standard set-theoretic semantics used in description logics [1]. Non-inheritable concept properties (own slots) include, among others, the following constructors — the first three ones are not applicable to individuals. – The concept type specifies whether the concept is a class, an association, a process, or an individual belonging to one of these three categories. – The concept definition states whether the concept is either totally or only partially specified by its description. In the latter case, the concept is a generic subset of the (dummy) set fully specified by the description. – The instantiability states whether the concept can have any instances. – An enumeration explicitly defines a concept as the set characterised by the listed individuals. Note that concepts defined in this way (i.e. starting from the empty set — bottom — and adding individuals) cannot be compared with concepts defined starting from the universe (Top or Thing) and adding constraints. In other words, no subsumption relationship can be derived between concepts belonging to these two complementary groups. Each labelled field defines a couple (key name and corresponding value) that can be used to introduce a new own property such as author, creation date, version. All these user-definable properties can be used for different knowledge management purposes (e.g. versioning, annotations, retrieval). Inheritable concept properties (template slots) include the following constructors. Attributes are individual properties shared by all the individuals that belong to the same set concept. Parts are either individual components in structured individuals or set concept components in structured set concepts. Associative links are bi-directional connections between different types of concepts (e.g. classes and associations). Global axioms are mutual constraints between set concepts (e.g. disjointness or full coverage in terms of some subclasses). Non-inheritable role properties include inverse roles, used to define bidirectional associative links, and synonym roles, used to specify property name aliases.

84

Ernesto Compatangelo and Helmut Meisel

Restrictions on inheritable concept-role links (i.e. slot facets) include the following constructors. Minimum and maximum cardinalities constrain the multiplicity of a property (i.e. its lower and upper number of allowed values). Fillers constrain all individuals belonging to a concept to share the same value(s) for a property. The domain of an attribute constrains all its values to be selected from the class (or combination of classes) specified as domain. A domain enumeration explicitly lists all the allowed values (individuals) in the domain.

(0,N)

(0,N)

(0,N)

Concept (Frame)

(0,N)

Role (Slot Frame)

Concept-Role Link (Frame-Slot Attachment)

Role (Slot Frame)

Role Properties (Slots on Slot Frames)

Local Restrictions on Roles (Facets)

Global Restrictions on Roles

(a) Resource ontology overview Non-Inheritable Properties (Own Slots) Concept Type

1:1 {Class | . . . }

Name

1:1 String

Definition

1:1 {Partial | Total}

Superconcepts

0:M Concept

Instances

0:M Concept Instance

Instantiability

1:1 {Concrete | Abstract}

Enumeration

0:M Individual

Labelled Fields

0:M (Key, Value) Pair

Inheritable Properties (Template Slots) Attributes

0:M Concept Attribute

Parts

0:M Concept Part

Associative Links 0:M Concept Global Axioms

0:M Assertion

(b) Concepts Restrictions on inheritable properties (Facets)

Non-Inheritable Properties ( Own Slots)

Min Cardinality

1:1 Integer

Name

1:1 String

Max Cardinality

1:1 Integer

Superroles

0:M Global Role

Fillers

0:M Individual

Inverse Role

0:1 Global Role

Domain

0:1 Class (and|or Class)∗

Synonym Roles 0:M Global Role

Domain Enumeration 0:M Individual

(c) Concept-role links

Labelled Fields 0:M Key Value Pair

(d) Roles

Fig. 1. The ConcepTool resource ontology

ConcepTool: Intelligent Support to the Management of Domain Knowledge

85

Individuals in ConcepTool either explicitly or implicitly commit to the structure of some set concepts. Individuals directly belonging to a set concept are denoted as concept instances, while those belonging to any of its subsets are denoted as concept elements. This interpretation of individuals extends the one in DL-based and in frame-based approaches.

3

Knowledge Management Services in CONCEPTOOL

In devising the services to be provided by ConcepTool, we focused on a wide range of inferences and management support functionalities for very expressive domain knowledge. Our previous research highlighted the benefits gained (i) by combining different kinds of inferences and/or (ii) by including incomplete deductions and/or (iii) by reducing the expressiveness of the knowledge subject to analysis [5, 11]. We have incorporated all these features in ConcepTool, which currently combines functionalities from three different levels: – At the epistemological level, deductions from DL engines are used as the fundamental logic-based analysis services. These deductions are considered as a kind of reference ontology that can be compared with the domain knowledge under analysis. However, DL engines do not correctly compute subsumption (and thus the analysis results based on it) between concepts with “non-standard” hierarchical rules, such as associations (relationships) and functions. Therefore, conceptual emulators [4] are used to transform DL deductions into the result expected at the conceptual level whenever needed. – At the conceptual level, deductions include (i) the generation of different concept hierarchies (one for each concept type), (ii) the identification of inconsistent concepts, and (iii) the explicitation of new concept constraints and properties which are not explicitly stated in a domain knowledge body. The possibility of setting a specific level of expressive power to be used in conceptual reasoning allows the generation of further (and / or different) analysis results. This is performed by selecting those semantic constructors (e.g. attribute multiplicities, domains, or fillers) that can be ignored while reasoning with a domain knowledge body. The reduction of the expressive power subject to analysis is particularly useful for aligning and articulating domain knowledge during the early stages of ontology sharing and reuse [5]. – At the lexical level, deductions include lexical subsumption, synonymity, antonymity, and partonymity provided by lexical databases such as WordNet [13]. These deductions are complemented by a set of lexical heuristic inferences based on string containment. Inferential results from the three levels are presented to the analyst who can decide which ones to accept and which ones to reject. Both accepted and rejected inferences can be incorporated into the knowledge body under analysis by adding new concept constraints or by removing existing ones. The different kinds of epistemological, conceptual and lexical inferences can be performed in any user-defined sequence in order to maximise analysis results.

86

Ernesto Compatangelo and Helmut Meisel

In other words, the user (i) can decide which inferential services to invoke, (ii) compares the results of each invoked services, and (iii) decides which step to perform next. However, (partial) results from a specific inferential service (e.g. a constraint satisfaction check) could be automatically passed to another inferential service (e.g. a subsumption computation algorithm). In this way, a reasoning system could derive results about expressive knowledge that cannot be completely interpreted and processed by each single inferential service alone. The combination of different kinds of inferences has been performed with a prototypical version of ConcepTool which dynamically fuses taxonomic deductions from a DL engine with inferences from a constraint satisfaction reasoner [11]. In this approach, constraints on (numeric) concrete domains are transformed into a hierarchy of concrete domain classes which are then used to compute subsumption on the whole domain knowledge base. Preliminary results showed how this “inference fusion” mechanism can successfully extend the automated reasoning capabilities of a DL engine. Because the inferences provided by the constraint satisfaction reasoner are in no way complete, this same approach shows that incomplete deductions are nonetheless useful to derive meaningful (although potentially incomplete) analysis results. ConcepTool also provides an environment where different domain knowledge bodies can share the lexical namespace of their concept properties (attributes, associative links, partonomic links). Synonyms (e.g. last name and surname) are first proposed by lexical databases and then validated by ConcepTool on the basis of their mutual structural features. Full synonymity between two properties with lexical synonymity is rejected if their structural features (e.g. super-properties, inverse) are different. This approach, where distinct domain knowledge bases can share the same project space, and possibly the namespace associated to the project, explicitly supports both cooperative domain knowledge development and the alignment of existing domain knowledge bodies.

4

Discussion and Future Work

ConcepTool introduces three major novelties with respect to existing intelligent knowledge management environments. Firstly, its expressive knowledge model contains semantic components, such as individuals, which are neither jointly nor separately available in other models. Secondly, ConcepTool performs deductions at the conceptual level, thus providing correct analysis results for non-standard concepts like associations and functions. It also allows to ignore selectable properties and restrictions while reasoning. Moreover, it complements conceptual deductions with both lexical and heuristic inferences of different kinds. Thirdly, the ConcepTool interface explicitly supports cooperative domain knowledge development by providing an environment where new knowledge bodies can be directly linked to (and later decoupled from) existing ontologies or other reference knowledge bases.

ConcepTool: Intelligent Support to the Management of Domain Knowledge

87

The modelling and analysis functionalities provided in ConcepTool have been successfully tested using a number of sizeable domain ontologies. A preliminary version that demonstrates some relevant features of our system can be downloaded from http://www.csd.abdn.ac.uk/research/IKM/ConcepTool/. Future releases are expected to include (i) functionalities to import from / export to frame-based and DL-based environments using XML and RDF(S), (ii) a more general version of the articulation functionality already available in a previous version of ConcepTool that uses an entity-relationship knowledge model [5], and (iii) inference fusion for approximate reasoning with complex constraints [11].

Acknowledgements This work is supported by the British Engineering & Physical Sciences Research Council (EPSRC) under grants GR/R10127/01 and GR/N15764 (IRC in AKT).

References [1] F. Baader et al., editors. The Description Logic Handbook. Cambridge University Press, 2003. 81, 83 [2] S. Bechhofer et al. OilEd: a Reason-able Ontology Editor for the Semantic Web. In Proc. of the Joint German/Austrian Conf. on Artificial Intelligence (KI’2001), number 2174 in Lecture Notes in Computer Science, pages 396–408. SpringerVerlag, 2001. 82 [3] R. J. Brachman et al. Reducing CLASSIC to practice: knowledge representation meets reality. Artificial Intelligence, 114:203–237, 1999. 82 [4] E. Compatangelo, F. M. Donini, and G. Rumolo. Engineering of KR-based support systems for conceptual modelling and analysis. In Information Modelling and Knowledge Bases X, pages 115–131. IOS Press, 1999. Revised version of a paper published in the, Proc. of the 8th European-Japanese Conf. on Information Modelling and Knowledge Bases. 82, 85 [5] E. Compatangelo and H. Meisel. EER-ConcepTool: a “reasonable” environment for schema and ontology sharing. In Proc. of the 14th IEEE Intl. Conf. on Tools with Artificial Intelligence (ICTAI’2002), pages 527–534, 2002. 82, 85, 87 [6] A. Duineveld et al. Wondertools? a comparative study of ontological engineering tools. In Proc. of the 12th Banff Knowledge Acquisition for Knowledge-based Systems Workshop (KAW’99), 1999. 81 [7] R. Fikes and A. Farquhar. Distributed Repositories of Highly Expressive Reusable Ontologies. Intelligent Systems, 14(2):73–79, 1999. 82 [8] E. Franconi and G. Ng. The i•com Tool for Intelligent Conceptual Modelling. In Proc. of the 7th Intl. Workshop on Knowledge Representation meets Databases (KRDB’00), 2000. 82 [9] V. Haarslev and R. M¨ oller. RACER System Description. In Proc. of the Intl. Joint Conf. on Automated Reasoning (IJCAR’2001), pages 701–706, 2001. 82 [10] I. Horrocks. The FaCT system. In Proc. of the Intl. Conf. on Automated Reasoning with Analytic Tableaux and Related Methods (Tableaux’98), number 1397 in Lecture Notes In Artificial Intelligence, pages 307–312. Springer-Verlag, 1998. 81

88

Ernesto Compatangelo and Helmut Meisel

[11] B. Hu, I. Arana, and E. Compatangelo. Facilitating taxonomic reasoning with inference fusion. In Knowledge Based Systems – To appear. Revised version of the paper published in the Proc. of the 22nd Intl. Conf. of the British Computer Society’s Specialist Group on Artificial Intelligence (ES’2002). 85, 86, 87 [12] R. MacGregor. A description classifier for the predicate calculus. In Proc. of the 12th Nat. Conf. on Artificial Intelligence (AAAI’94), pages 213–220, 1994. 81 [13] G. A. Miller. WordNet: a Lexical Database for English. Comm. of the ACM, 38(11):39–41, 1995. 85 [14] H. Stuckenschmidt, F. van Harmelen, and P. Groot. Approximate terminological reasoning for the semantic web. In Proc. of the 1st Intl. Semantic Web Conference (ISWC’2002), 2002. 82 [15] The Advanced Knowledge technologies (AKT) Consortium. The AKT Manifesto, Sept. 2001. http://www.aktors.org/publications http://www.aktors.org/publications. 81

Combining Revision Production Rules and Description Logics Chan Le Duc and Nhan Le Thanh Laboratoire I3S, Universit´e de Nice - Sophia Antipolis, France [email protected] [email protected]

Abstract. Knowledges of every application area are heterogeneous and they develop in time. Therefore, a combination of several formalisms on which revision operations are defined, is required for knowledge representation of such a hybrid system. In this paper, we will introduce several revision policies in DL-based knowledge bases and show how revision operators are computed for DL language ALE. From this, we define revision production rules consequent of which is a revision operator. Such rules allow us to represent context rules in translation between DL-based ontologies. Finally, we will introduce formal semantics of the formalism combining revision production rules and Description Logics (ALE).

1

Introduction

Description Logics can be used as a formalism for design of ontologies of an application domain. In order to reduce the size of ontologies, different ontologies for different subdomains or user profiles are derived from a shared common ontology. The shared concepts in the common ontology can be redefined in a derived ontology with the aim of fitting an adaptable context. This is necessary to determine more sufficient meaning of a shared concept in the context of current users. The redefining can be performed in run-time owing to context rules. The antecedent of a context rule is a predicate which describes context condition and its consequent is an operation which translates a concept definition into another. For this reason, we need an other formalism to capture the semantics of such ontologies. It is specific production rules, namely revision production rules, whose consequent is an operation allowing to change a shared concept definition. In previous hybrid languages, for example CARIN in [4], it combines Horn rules and Description Logics to obtain a formalism the expressiveness of which is significantly improved. However, it seems that these languages could not capture context rules since the consequent of a Horn rule is still a predicate. On the other hand, since the definition of a concept in knowledge base can be modified by the application of a revision production rule, revision operations must be defined on DL-based terminology. Previous works on how to define revision operations have concentrated on operations TELL (to add a concept description fragment to a concept definition), FORGET (to delete a concept description fragment V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 89–99, 2003. c Springer-Verlag Berlin Heidelberg 2003


90

Chan Le Duc and Nhan Le Thanh

from the concept definition), and on simple languages which do not permit existential restrictions [1]. Moreover, operation FORGET requires that a deleted fragment must be a part of the concept definition to be revised. Thus, the main contributions of this paper are, first, to propose several revision policies some of which require the definition of a new operation, namely PROPAG, in order to conserve subsumption relationships. Second, we propose to extend revision operations to language ALE. This extension will allow us to define the semantics of a formalism combining revision production rules and language ALE.

2 2.1

Knowledge Base Revision and Revision Policies Preliminaries

Revision Operators Informally, TELL is interpreted as an operation which adds knowledge to knowledge base ∆, whereas FORGET deletes knowledge from knowledge base ∆. If we use a DL language allowing both to normalize concept descriptions and rewrite concept definitions as conjunctions of simple concepts, the revision operators are more formally written as follows : – TELL(C, Expr ) := C  Expr – FORGET(C  Expr, Expr ) := C where C is concept to be revised and Expr is a simple concept. In order to have reasonable definitions of the revision operators, it is paramount that several constraints are respected. However, if all of these constraints are to be respected, certain problems are inevitable. In the following subsection, we attempt to identify some of important constraints for the revision operators. Criteria for Revision The majority of these criteria is proposed and discussed in [1]. We only recall and reformulate some of them for the purposes of this paper. 1. Adequacy of the Request Language. The request language should be interpretable in the formalism the KB uses. 2. Closure. Any revision operation should lead to a state of the KB representable by the formalism used. 3. Success. The criterion requires that any revision operation must be successful, i.e after a TELL the new knowledge (Expr ) must be derivable from the KB and after a FORGET the deleted knowledge (Expr ) must not be derivable from the KB no longer. More formally, there exists an interpre/ tation I and an individual aI ∈ OI domain, such that aI ∈ C I and aI ∈ (T ELL(C, Expr))I . Similarly, there exists an interpretation I and an indi/ C I and aI ∈ (F ORGET (C, Expr))I . vidual aI ∈ OI domain, such that aI ∈ 4. Minimal Change. This criterion requires that revision operations cause a minimal change of the KB. It is obvious that it needs a definition concerning the distance between two KBs. Therefore, a pragmatic consideration is necessary in each real application.

Combining Revision Production Rules and Description Logics

91

5. Subsumption Conservation. The terminological subsumption hierarchy of the KB should be conserved by any revision operation. A priori, criteria 4 and 5 are difficult to guarantee simultaneously since subsumption conservations may change the majority of the hierarchy. 2.2

Revision Policies

In the previous subsection, we introduced the general principles for revision on DL-based knowledge base. In certain cases there may exist one principle that we cannot entirely adhere to. We now present pragmatic approaches in which some of these criteria are respected while the others are not. In order to facilitate the choice of policy made by the user, we will attempt to identify the advantages and disadvantages. The first policy takes into account criterion 5 (Subsumption Conservation) whereas the second policy allows us to change subsumption hierarchy. The third policy can be considered as a compromise between the first and the second. Conservative Policy This policy allows us to modify definition of a concept C (concept description) by adding or deleting a fragment to or from the definition. Concept C and all concepts defined via C now have a new definition. However, the modification of C should not violate any subsumption relationship. Otherwise, no operation is to be done. By that we mean, if A B holds before the modification for some concepts A, B in the Tbox, then A  B still holds after the modification. This requirement is in accordance with the viewpoint in which the subsumption relationships are considered as invariant knowledges of the KB. This condition is clearly compatible with the object-oriented database model provided that the hierarchical structure of classes is persistent, although class definitions may be changed. In this model, each time a class definition is changed, all existent instances of the old class must remain the instances of the new class. In fact, the evolution of our understanding about the world leads to more precise definitions. In other words, we need to change the definition of a concept C while all subsumption relationships and assertions C (a) in Abox remain held. This can lead to some incompatibilities between the considered criteria for definitions of inferences in the open world. Indeed, we suppose that revision operator TELL is defined as follows : TELL(C, Expr ) := C  Expr. According to the assertion conservation proposed by this policy, if C I (aI ) holds then (T ELL(C, Expr))I (aI ) holds as well for all interpretation I. This means that C  TELL(C, Expr ) for all interpretation I. Hence C ≡ TELL(C, Expr ) since TELL(C, Expr )  C for all interpretation I. This contradicts criterion 3 (success). In order to avoid this incompatibility, an addition of the epistemic operator can be required [5]. We now try to provide a framework for the revision operators. Suppose that Tbox is acycle and does not contain general inclusions. Since the subsumption relationship conservation is required in this policy, a modification on C can be propagated in all Tbox. In the following, we will investigate this propagation

92

Chan Le Duc and Nhan Le Thanh

for each operator TELL and FORGET. The definitions of TELL, FORGET and PROPAG with respect to criteria 3, 4 (success and minimal change) for a concrete language are investigated in the next section. TELL(C, Expr ). Obviously, this operation makes a concept D more specific where D contains concept C in its definition. On the other hand, concepts no containing concept C may be modified because of the conservation of the subsumption relationships. The unfolded definition of D containing C is written ˜ ) and the unfolded definition of D no containing C is written as D( ˜ C). ¯ as D(C ˜ ) and D( ˜ C) ¯ can be obtained by replacing all concept names by their defiD(C nitions except concept C. Let E be a direct super concept of F in the hierarchy. ˜ C) ¯ then relationship F  E is There are two cases as follows : i) if E = E( automatically conserved since E is not changed and F may be not changed or ¯ and E = E(C ˜ ) or F = F˜ (C ) become more specific. ii) if whether F = F˜ (C) ˜ ), we need a new operator, namely PROPAG TELL, that allows and E = E(C us to compute a new definition of F such that the subsumption relationship is conserved. Operator PROPAG TELL should be defined with respect to criterion 4 in some way. FORGET(C, Expr ). This operator is, in fact, computed via an other operator, namely MINUS, that can be interpreted as a deletion of Expr from C. In the case where C = C’  Expr, we must have MINUS(C, Expr ) = C’. Similar as TELL, the modification made by FORGET may be propagated in all Tbox. Revision of Literal Definition This revision policy had been proposed in [1]. The main idea of this policy is to allow us to modify the definition of a concept without taking into account subsumption relationships. In addition, it requires to revise the Abox with respect to the new Tbox because a modification in Tbox may cause inconsistencies in the Abox. This policy is convenient for the case in which we need a correction of a misunderstanding in terminology design. The individuals of a “bad” definition now belong to a better definition. In fact, in this approach concept definitions are considered as literal knowledges while subsumption relationships are derived from these definitions. As a result, a reclassification of the concept hierarchy can be required after a revision. Note that this policy does not apply operator PROPAG since subsumption conservation is omitted. Operators TELL(C, Expr ) and FORGET(C, Expr ) are defined owing to operator MINUS described as above. Revision on Abox. An important task in this policy is to revise the ABox when the Tbox is changed. All assertions D (a) where D is defined directly or indirectly via C must be revised. It is clear that operator FORGET does not cause any inconsistence in the Abox since if F (C )(a) holds where F (C ) is a concept description via C, then FORGET(F (C )(a), Expr ) holds as well. In contrast, there may be an inconsistence in the Abox after TELL(C, Expr ). In this case, the revision on Abox requires an inference which is based on the added part Expr. In particular, D (a) holds for some concept D = F (C ) = C, iff Expr (a) holds.

Combining Revision Production Rules and Description Logics

93

Mixed Policy As presented above, this policy can be considered as a compromise between two policies presented, namely mixed policy. It allows us to modify the definition of a concept while respecting all subsumption relationships. Moreover, following a revision on the Tbox, a revision on the Abox is necessary. Technically, this policy does not require any additional operator in comparison to the two policies above.

3 3.1

Revision Operators for Language ALE Preliminaries

Let NC be a set of primitive concepts and NR be a set of primitive roles. Language ALE uses the following constructors to build concept descriptions: conjunction (C  D ), value restriction (∀r.C ), existential restriction (∃r.C ), primitive negation (¬P ), concepts TOP and BOTTOM. Let ∆ be an non-empty set of individuals. Let .I be a function that transforms each primitive concept P ∈ NC into P I ⊆ ∆ and each primitive role r ∈ NR into rI ⊆ ∆× ∆. The semantics of a concept description are inductively defined owing to the interpretation I = (∆,.I ) in the table below.

Syntax  ⊥ C D ∀r.C, r ∈ NR ∃r.C, r ∈ NR ¬P , P ∈ NC

Semantics ∆ ∅ CI ∩ DI {x ∈ ∆|∀y:(x,y) ∈ r I → y ∈ C I } {x ∈ ∆|∃y:(x,y) ∈ r I ∧ y ∈ C I } ∆ \ PI

Least Common Subsumer (LCS) [8] Subsumption Let C, D be concept descriptions. D subsumes C, C  D, iff C I ⊆ DI for all interpretations I. Least Common Subsumer Let C1 , C2 be concept descriptions in a DL language. The least common subsumer C is a least common subsumer of C1 , C2 (for short lcs(C1 , C2 )) iff i) Ci  C for all i , and ii)C is the least concept description which has this property, i.e if C’ is a concept description such that Ci  C’ for all i, then C  C’. According to [8], lcs of two or more descriptions always exists for language ALE and the authors proposed an exponential algorithm for computing it.

94

Chan Le Duc and Nhan Le Thanh

Language ALE and Structural Subsumption Characterizing The idea underlying the structural characterization of subsumption proposed in [8] is to transform concept descriptions into a normal form. This normal form allows one to build a description tree the nodes of which are labeled by sets of primitive concepts. The edges of this tree correspond to value restrictions, namely ∀r-edge and existential restrictions, namely r-edge. Hence, the subsumption relationship of two concept descriptions can be translated into the existence of a homomorphism between the two corresponding description trees. The following theorem results from this idea. Theorem for structural subsumption characterization [8] Let C, D be ALEconcept descriptions. Then, C  D if and only if there exists a homomorphism  to GC . from GD Simple Concept Descriptions in Tbox In a Tbox, a concept definition can depend on other concept definitions. This dependence must be conserved by revision operations. For this reason, we cannot define simple concept as a conjunct in unfolded form. In what following, we propose a simple concept definition which fits the idea above. An ALE-concept description is called simple if it does not contain any conjunction on the top-level. A concept description is called normal if it is a conjunction of simple concept descriptions. We denote SC as a set of simple concept descriptions of concept C in a TBox. Note that a simple concept can be considered part of the meaning of a concept. As a consequence, the sets of simple concepts of the equivalent concepts can be different. A discussion in detail of this subject is given in [1]. 3.2

Operator TELL

Operator TELL Let C1  ...  Cn be of the normal form of C and Expr be simple concept. The operator TELL is defined as follows : If C  Expr then TELL(C, Expr ) := C Otherwise, TELL(C, Expr ) := C  Expr and SD = SC ∪ {Expr } If the knowledge to be added is derivable from C before the execution of TELL i.e C  Expr, no more remains to be done and the KB is not changed. Otherwise, we need to guarantee that the added knowledge is derivable from C after the execution of TELL. In fact, since TELL(C, Expr ) = C  Expr , we have TELL(C, Expr )  Expr. Moreover, if C  Expr, this definition of operator TELL will satisfy criteria 3 (success). Criterion 4 (minimal modification) will be semantically satisfied since C  Expr is the most concept description which is subsumed by C and Expr. Operator PROPAG TELL We will use the notions presented in section 2.2. ˜ In addition, denote E (C, D ), E(C, D ) as original and unfolded forms where each occurrence of C is replaced by D. Let E be a direct super concept of a concept

Combining Revision Production Rules and Description Logics

95

F in the hierarchy. Operator PROPAG TELL(E, F, Expr ) will compute a new concept description of F while taking into account criterion 4 (minimal change) and F  E. ˜ ). We denote S 0 (C ) as a set of all sE ∈ SE such Assume that E = E(C E 1 ¯ that sE depends on C, denoted by sE (C ), and SE (C) as a set of all sE ∈ SE ¯ We have : such that sE does not depend on C, denoted by sE (C). ˜ PROPAG TELL(E, F, Expr ):=F (C, C  Expr ) sE ∈SE0 (C) sE (C, C  Expr ) As a consequence, PROPAG TELL(E, F, Expr )  E. Note that PROPAG TELL(E, F, Expr ) is the most concept description ˜ which is less than F˜ (C, C  Expr ) and E(C, C  Expr ).



Example 1. Let T1 ={ C := ∃r.Q, E := ∃r.C, F := ∃r.∃r.(Q  R) } and Expr := ∃r.P where P , Q, R ∈ NC , r ∈ NR . ˜ ), F = F˜ (C); ¯ SE ={∃r.C }, SF ={∃r.∃r.(Q  R)} We have F  E and E = E(C ˜ ˜ ˜ Note that F (C, C  Expr )  E(C, C  Expr ) since E(C, C  Expr ) = ∃r.(∃r.Q ˜  ∃r.P ) and F (C, C  Expr ) = ∃r.∃r.(Q  R). We have PROPAG TELL(E, F, Expr ) = ∃r.∃r.(Q  R)  ∃r.(C  ∃r.P ) and SP ROP AG T ELL(E,F,Expr) = {∃r.∃r.(Q  R), ∃r.(C  ∃r.P )}.
3.3

Operator FORGET

This operator FORGET(C, Expr ) here defined, allows for Expr ∈ / SC . This extension is necessary because in some cases we need to forget a term Expr from C where C  Expr and Expr ∈ / SC . Such knowledge can be derived from C. Operator MINUS between Two ALE-Description Trees Let C, D be propagated normal ALE-concept descriptions and C  D. MINUS(C, D ) is defined as the remainder of GC description tree after the deletion of image ϕ(GD ) in GC for all homomorphisms ϕ from GD to GC . More formally, let GC =(NC ,  EC , n0 , lC ) and GD =(ND , ED , m0 , lD ). MINUS(C, D ) = (NC , EC , p0 , lC )  where lC (ϕ(m)) = lC (ϕ(m)) \ lD (m) for all m ∈ ND and all ϕ. Example 2. Let E, F be concept descriptions in the example1. We have MINUS(F, E ) = ∃r.∃r.R.




Proposition 3.1 Let C be a propagated normal ALE-concept description, D be a propagated normal ALE-simple concept description and D = D1  ...  Dn . Moreover, C  D. If C = X  D and X  Di for all i, then MINUS(C, D)≡ X. A proof of Proposition 3.1 can be found in [9]. 0 1 a set of all sC ∈ SC such that sC  Di for all i and SC−D We denote SC−D  0 a set of all sC as the remainder of sC ∈ / SC−D following the operation MINUS. 0 Proposition 3.1 guarantees that sC ∈ SC are not changed. We have SMIN US(C,D) 0 1 = SC−D ∪ SC−D .

96

Chan Le Duc and Nhan Le Thanh

We can now define operator FORGET. Let C1  ...  Cn be of the normal form of C and Expr be a simple concept. Operator FORGET is defined as follows : If C  Expr then FORGET(C, Expr ) := C Otherwise, FORGET(C, Expr ) := MINUS (C, Expr ) and, 0 1 SF ORGET (C,Expr) = SC−Expr ∪ SC−Expr If C  Expr, i.e we may leave out the fragment Expr as it does not contribute to the definition of C. In this case, there is nothing to do since C is not derivable from the knowledge to be forgotten. In fact, there exists an individual a such / ExprI where I is an interpretation. By that we mean, that aI ∈ C I and aI ∈ the knowledge Expr is not derivable from concept C following the execution of operation FORGET. Therefore, the operator which is defined in such a way satisfies criterion 3 (success). If C  Expr, operator MINUS ensures the unique existence of the result concept description. Hence, operator FORGET is well defined. Also, operator MINUS guarantees criteria 3 and 4 for operator FORGET owing to Proposition 3.1 [9]. Note that if C = X  Expr then FORGET(C, Expr) = X. Operator PROPAG FORGET Let E be a direct super concept of a concept F in the hierarchy. Operator PROPAG FORGET(E, F, Expr ) will compute a new concept description of E while taking into account criterion 4 (minimal change). Computing lcs is, therefore, necessary for this aim. Assume that F = F˜ (C ). We define : PROPAG FORGET(E, F, Expr ):= ˜ lcs{E(C, MINUS(C, Expr )), F˜ (C, MINUS(C, Expr ))} Computed in this way, PROPAG FORGET(E, F, Expr ) is the least concept ˜ description such that it is more than E(C, MINUS(C, Expr )) and ˜ F (C, MINUS(C, Expr )). We now compute SP ROP AG F ORGET (E,F,Expr) according to the definition above. ˜ MINUS(C, Expr )) and E2 = Thus abbreviated, we denote E1 = E(C, PROPAG FORGET(E, F, Expr ). Since E1  E2 , we can define an intersection of all images ϕ(GE2 ) in GE1 : K = ϕ ϕ(GE2 ) where ϕ is a homomorphism from GE2 0 1 to GE1 . We denote SE a set of all sE1 ∈ SE1 such that GsE1 ⊆ K and SE a set of 1 2 all sE2 ∈ SE2 such that ϕ(GsE2 )
K where ϕ is some homomorphism from GE2 0 1 to GE1 . We define SE
: = SE ∪ SE . It is not difficult to check that 1 2 s∈SE
s ≡ PROPAG FORGET(E, F, Expr ).





Example 3. Let T1 ={ C := ∃r.(P  Q )  ∀r.R, E := ∃r.S  ∃r.∃r.(P  Q ), F := ∃r.S  ∃r.C } and Expr := ∃r.P where P , Q, R, S ∈ NC , r ∈ NR . ˜ C); ¯ SE ={∃r.S, ∃r.∃r.(P  Q )}, We have F  E and F = F˜ (C ), E = E( SF ={∃r.S, ∃r.C }, MINUS(C, Expr ) = ∃r.Q  ∀r.R. ˜ Note that F˜ (C,MINUS(C, Expr ))  E(C,MINUS(C, Expr )) since ˜ F (C,MINUS(C, Expr ))=∃r.S  ∃r.(∃r.Q  ∀r.R) and ˜ E(C, MINUS(C, Expr ))= ∃r.S  ∃r.∃r.(P  Q ) We have PROPAG FORGET(E, F, Expr ) = ∃r.S  ∃r.∃r.Q; 0 1 ={∃r.S , ∃r.∃r.Q} and SE = ø. K = {∃r.S  ∃r.∃r.Q}; SE 1 2

Combining Revision Production Rules and Description Logics

As a consequence, SP ROP AG 3.4

F ORGET (E,F,Expr)

= {∃r.S, ∃r.∃r.Q }

97




Revision on Abox

In the previous subsection, a concept can be considered as a conjunction of simple concepts. The revision operators on Tbox add or delete some terms to/from these sets. Let E be a modified concept definition in the Tbox. Let SE and SE
be the sets of simple concepts of E, respectively, before and after a revision on the Tbox. Let I be a model of the KB. For each assertion E I (aI ), revision on the Abox performs a checking of all assertions sI (aI ) where s ∈ SE
\SE . If there is some sI (aI ) which is not verified, E I (aI ) becomes inconsistent and it is deleted from the Abox. Otherwise, if every sI (aI ) is verified, assertion E I (aI ) is conserved in the Abox.

4 4.1

Revision Production Rules and Description Logics Revision Production Rules

The rule to be defined below is a specific form of production rule, namely revision production rule . In fact, a consequent of these rules is only an revision operator on a terminology. The general form of these rules in a KB ∆ = (∆T , ∆A , ∆PR ) is given as follows : r ∈ ∆PR , r : p ⇒ TELL(Concept, Expr ) or r : p ⇒ FORGET(Concept, Expr ) where p is a predicate formed from assertions in ∆T with the constructors ¯ 1 ) ∧. . . ∧ pn (X ¯ n ), X ¯ i are tuples of variables or constants. The predip1 (X cates p1 ,. . . , pn are whether concepts names, role names or an ordinary predicate. Revision operators TELL and FORGET have been defined in section 3. An example which illustrates how to exploit these rules can be found in [9]. 4.2

Semantics of Combining Formalism

We can now define the semantics of the formalism combining revision production rules and description logic ALE based on the semantics of revision operators. We will define semantics of the formalism on the assumption that the propagation operators are not taken into account. It will not be too difficult to extend the following definition to a KB with these operators. Let ∆ be a KB composed of three components ∆ = (∆T , ∆A , ∆PR ). An interpretation I is a pair (O,.I ) where O is a non-empty set and a function .I maps every constant a in ∆ to an object aI ∈ O. An interpretation I is a model of ∆ if it is a model of each component of ∆. Models of terminological component ∆T are normally defined as what of description logics. An interpretation I is ¯ I ∈ pI . a model of a ground fact p(¯ a) in ∆A if a An interpretation I is a model of r : p ⇒ TELL(C, Expr ) if, whenever the ¯ i )I ∈ pI for function .I maps the variables of r to the domain O such that (X i every atom of the antecedent of r, then

98

Chan Le Duc and Nhan Le Thanh

1. if aI ∈ (C I ∩ ExprI ) then aI ∈ (T ELL(C, Expr))I 2. if aI ∈ C I and aI ∈ / ExprI then aI ∈ / (T ELL(C, Expr))I An interpretation I is a model of r : p ⇒ FORGET(C, Expr ) if, whenever ¯ i )I ∈ pI the function .I maps the variables of r to the domain O such that (X i for every atom of the antecedent of r, then if aI ∈ C I then aI ∈ (F ORGET (C, Expr))I

5

Conclusion and Future Work

We introduced several revision policies and a framework for the revision operators on DL-based knowledge base. In the first policy, the assertion conservation on Abox, despite modifications of concept definition on Tbox, focuses our attention on inferences with the epistemic operator in a closed world. However, the interaction between the epistemic operator and the revision operators was not investigated in this paper. Next, we showed how to compute revision operators for language ALE. A deeper study of the complexity of these operators is necessary for a possible improvement in the computing. An other question arises concerning the extension of the revision operators to languages allowing for disjunction (ALC). This extension requires a structural characterization of subsumption for these languages. On the other hand, if we allow for Horn rules as a third component, then which interactions will take place in the combining formalism ? In this case, the obtained formalism will be an extension of language CARIN [4]. Hence, an extension of the inference services of CARIN to the combining formalism deserves investigation.

References [1] B. Nebel. Reasoning and Revision in Hybrid Representation Systems, Thesis 19901995 90, 92, 94 [2] B. Nebel. Terminological reasoning is inherently intractable. Artificial Intelligence, 1990 [3] M. Schmidt-Schauβ, G. Smolka. Attributive concept descriptions with complements. Artificial Intelligence, 1991 [4] A. Y. Levy and M.C Rousset. Combining Horn Rules and Description Logics in CARIN, Artificial Intelligence, Vol 104, 1998. 89, 98 [5] F. M. Donini, M. Lenzerini, D. Nardi, W. Nutt, A. Schaerf. An Epistemic Operator for Description Logics, Artificial Intelligence, Vol 100, 1998. 91 [6] F. M. Donini, M. Lenzerini, D. Nardi, A. Schaerf. Reasoning in Description Logics, CSLI Publications, 1997 [7] R. K¨ usters. Non-Standard Inferences in Description Logics. Thesis, Springer 2001 [8] F. Baader, R. K¨ usters and R. Molitor. Computing least common subsumer in Description Logics with existential restrictions, Proc. of the 16th Int. Joint Conf. on Artificial Intelligence (IJCAI-99 ) 93, 94

Combining Revision Production Rules and Description Logics

99

[9] C. Le Duc, N. Le Thanh. Combining Revision Production Rules and Description Logics. Technical Report 2003. See http://www.i3s.unice.fr/~cleduc. 95, 96, 97

Knowledge Support for Modeling and Simulation ˇ cenko Michal Sevˇ Czech Technical University in Prague Faculty of Electrical Engineering, Department of Computer Science [email protected]

Abstract. This paper presents intermediate results of the KSMSA project (Knowledge Support for Modeling and Simulation Systems). The outcome of the project is to apply artificial intelligence and knowledge engineering techniques to provide support for users engaged in modeling and simulation tasks. Currently we are focused on offering search services on top of a knowledge base, supported by a domain ontology.

1

Introduction

System modeling and simulation is an important part of an engineer’s work. It is an important tool for e.g. analyzing behavior of systems, predicting behavior of new systems, or diagnosing discrepancies in faulty systems. However, modeling and simulation tasks are highly knowledge-intensive. They require lot of knowledge directly needed to perform modeling and simulation tasks, as well as lot of background knowledge, such as mathematics, physics, etc. A great deal of this knowledge is tacit, hard to express, and thus hard to transfer. The knowledge has often heuristic nature, which complicate implementation of such a knowledge directly into computer systems. We propose to design a knowledge-based system supporting modeling and simulation tasks. At present, we are focused on offering search services for the users. The prerequisite to our system is the development of a formal model of concepts in the domain of modeling and simulation, or of engineering in general. Such a model is commonly referred to as an ontology. Ontologies basically contain taxonomy of concepts related to each other using axioms. Axioms are statements that are generally recognized to be true, represented as logical formulas. Ontologies help us to represent the knowledge normally possessed only by humans in computer memory, and to perform simple reasoning steps that resemble reasoning of humans. Ontologies also help to structure large knowledge bases, and provide a means to search such knowledge bases with significantly better precision than conventional search engines. Ontology can be accompanied by a lexicon, which provides a mapping between concepts in the ontology and natural language. This allows users to refer to concepts in the ontology using plain English words or phrases, or to automatically process natural language texts and to estimate their logical content. We have developed a prototype of ontology and lexicon for our domain. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 99–103, 2003. c Springer-Verlag Berlin Heidelberg 2003


100

ˇ cenko Michal Sevˇ

The new generation of search engines are based on the concept of metadata. Metadata are information that is associated with documents being indexed and searched by search engines, but are invisible to the user when a document is normally displayed. They are used to focus the search, thus improving its precision. Metadata that refer to the document as a whole include the name of the document author, date of the last modification, application domain, bindings to other documents, etc. Such kind of metadata provide additional information about the document that is not explicitly specified in its content. However, metadata can be associated also with parts of documents, such as sections, paragraphs, sentences or even words, often for the purpose of associating them with certain semantics, i.e. meaning. For instance, a string ‘John Smith’ can be associated with metadata stating that this string expresses the name of some person. Such metadata in general do not provide additional information to humans, as they understand the meaning of words from the context. However, they help to compensate the lack of intelligence of indexing and search engines that do not have such capability. Of course, ontologies and metadata bear on each other. Ontologies provide a means of describing the structure of metadata, so that metadata can be authored in a uniform way. A resource (document) is said to commit to an ontology if its metadata are consistent with that ontology. Indexing engine can then search a group of documents committing to one ontology, and it can provide users with means for focusing their searches according to concepts in that ontology. This scenario demonstrates the need for standardized ontologies that are committed to by many resources, so that the ontology-based search engines can index all of them.

2

Our Approach

We propose to design an ontology-based indexing and search engine for engineering applications, and for modeling and simulation in particular. To do this, we need to design an ontology for this domain. This section describes the most fundamental design goals of our search engine, the next section presents an overview of the content of our prototype ontology. The architecture of our knowledge system is quite conventional. It consists of the following components: – – – – –

knowledge base knowledge base index ontology and lexicon user’s interface (search service) administrator’s interface

The knowledge base is a collection of documents on the Internet available through the HTTP protocol. These are mostly HTML documents, containing either unrestricted natural language text relevant to our domain (such as electronic textbooks), or semi-structured documents (collection of solved examples,

Knowledge Support for Modeling and Simulation

101

catalogs of system components, model libraries, etc). Documents in the knowledge base may be provided by different authors. Since the knowledge in these documents is expressed implicitly, it is the user’s responsibility to extract and interpret this knowledge. Our system merely retrieves documents that it considers useful to the user. This is done with the help of the metadata stored in the knowledge base index. The knowledge base index is a central repository that stores metadata associated with documents in the knowledge base. It is used by the search engine to determine documents relevant to users’ queries. For semi-structured documents in the knowledge base, metadata have database-like nature. For example, documents in a collection of solved examples are annotated with attributes like application domain, physical domain, modeling technique, or used simulation engine. Documents containing unrestricted text may be also annotated with keywords structured according to concepts in the ontology. Since the index is connected to the ontology which defines mutual couplings between concepts, the search engine can perform simple reasoning steps during searches, such as generalizations, specializations, and natural-language disambiguations. The design decision that the knowledge base index is centralized is pragmatic. Centralized index is easier to maintain and is more accessible for the search engine that needs it for its operation. One can think that the metadata might be better supplied by authors of the documents and stored directly with these documents. This is the idea behind the initiative called the Semantic Web. However, today’s practice is that only few documents on the web are annotated with metadata (at least in the engineering domain), as the web-document authors are conservative and are not willing to invest extra effort into the development of their web pages. Our centralistic approach allows us to annotate documents developed by third parties. The user’s interface allows users to express their queries and submit them to the search engine. We propose two kinds of search interfaces. For searching for semi-structured documents, an administrator can design an ad-hoc form for entering the values of attributes. This enables users to focus on attributes that are relevant to a group of documents they are interested in. The second kind of interface enables users to search according to keywords. This service is very similar to conventional keyword-based search engines, but thanks to its exploitation of ontology and lexicon, it should exhibit better precision and recall. Keywords entered by the user are looked up in the lexicon, and related to respective concepts in the ontology. In the case of ambiguity (for polysemous keywords), users are prompted to resolve this ambiguity by selecting a subset of meanings (ontology concepts) corresponding to the keyword. The lexicon can also contain multiple words (synonyms) for single ontology concept, reducing thus the risk of document loss during the search due to using synonymous words when asking for concepts. These simple techniques partially reduce the impact of ambiguity of the natural language, and ensure better quality of search. The final component of our knowledge server is the administrator’s interface. It is an integrated environment for maintaining the ontology, lexicon and

102

ˇ cenko Michal Sevˇ

knowledge base index. It contains an ontology and lexicon browser and editor, document annotation tool, and analytical tools supporting the maintenance and evolution of the knowledge server.

3

The Ontology

This section presents a brief overview of the ontology that makes the core of our knowledge system. Our ontology is based on a general purpose, or upper , ontology called Suggested Upper Merged Ontology (SUMO) [1]. This implies that our ontology is represented in the Knowledge Interchange Format (KIF). One of the unique features of SUMO is that it is mapped to a large general purpose lexicon WordNet [2]. This enable the user to refer to concepts in the SUMO ontology using plain English words or phrases. We started to create our domain ontology on top of SUMO, and added corresponding terms to the lexicon by embedding them in the ontology using special predicates. Unlike other engineering ontologies, like EngMath [3] or PhysSys [4], our ontology is intended to represent basic common-sense concepts from the domain, rather than formally represent engineering models. Our knowledge base should represent the knowledge about modeling, not the models themselves. Our ontology is divided into several sections. The first section describes a taxonomy of models and modeling methodologies that engineers use to describe modeled systems, e.g. differential equations, block diagrams, multipole diagrams, bond graphs, etc. Although these concepts could be defined quite formally, we decided to provide only basic categorization accompanied by loose couplings providing informal common-sense navigation among the concepts. The second section describes a taxonomy of engineering components (devices) and processes, i.e. entities that are subject to modeling and simulation. The taxonomy in the current version of our ontology is only a snapshot of the terminology used in engineering, but should provide a solid base for extending it by domain experts. The third section of the ontology contains a taxonomy of typical engineering tasks (problems) and methods that can be used to solve them. The last section describes structured metadata that can be associated with individual documents in the knowledge base.

4

Conclusions and Future Directions

In this paper, we have presented our approach to the design of the knowledge system supporting modeling and simulation tasks. We presented only the general idea, more technical details can be found in [5]. We have implemented a first prototype of the whole system, which can be accessed from the project homepage [6]. The system includes all mentioned components, i.e. the search interface, the knowledge base index, the ontology and lexicon, and the administration tool. In the following paragraphs we propose some future work that should be done within the project.

Knowledge Support for Modeling and Simulation

103

Evolution of the Ontology, Lexicon and Knowledge Base. Although the current version of the ontology and the knowledge base can already demonstrate some concepts of our approach, it is clear that these resources must be much bigger both to be useful in practice and to prove that the approach is correct. We must thus dedicate some time for evolution of these resources—we must add new documents to the knowledge base and amend the ontology and lexicon accordingly. This work should also provide useful feedback for improving our tools for the knowledge base maintenance. Enlarging the ontology and knowledge base will also raise some questions about the performance and scalability. Improving the Functionality of Interfaces. The administrator’s interface and user’s interface should be improved. Especially the administration tool provides wide space for improvement. We plan to add some analytical tools based on information extraction and statistical methods, which should help to evolve and validate the knowledge base. We also plan to develop a tool that will automatically evaluate the performance of our knowledge system based on data recorded by the search interface, i.e. questions asked by users and answers provided by the knowledge system. Formalizing the Task Knowledge. Besides the static knowledge implicitly contained in the knowledge base, we propose to explicitly formalize the task knowledge required to perform certain classes of typical tasks taking place in the modeling and simulation domain. This task knowledge should provide more direct support for the users than mere access to static knowledge. Integration with the Existing Environment. Another important goal is to integrate our prototypical tools to an existing modeling and simulation environment, DYNAST, which we developed within another project. We will seek for new userinterface and computer-human interaction paradigms that can directly exploit the ontology and knowledge structures developed within this project.

References [1] I. Niles and A. Pease, “Toward a standard upper ontology,” in Proceedings of the 2nd International Conference on Formal Ontology in Information Systems (FOIS-2001), 2001. 102 [2] “Wordnet homepage,” http://www.cogsci.princeton.edu/~wn/. 102 [3] T. R. Gruber and R. G. Olsen, “An ontology for engineering mathematics,” in Fourth International Conference on Principles of Knowledge Representation and Reasoning, Bonn, Germany, 1994. 102 [4] W. N. Borst, Construction of Engineering Ontologies for Knwoledge Sharing and Reuse, Ph.D. thesis, University of Twente, September 1997. 102 ˇ cenko, “Intelligent user support for modeling and simulation of dy[5] Michal Sevˇ namic systems,” Tech. Rep., Czech Technical University in Prague, January 2003, Postgraduate Study Report DC-PSR-2003-01. 102 [6] “Knowledge support for modeling and simulation (ksmsa) project homepage,” http://virtual.cvut.cz/ksmsa/index.html. 102

Expert System for Simulating and Predicting Sleep and Alertness Patterns Udo Trutschel, Rainer Guttkuhn, Anneke Heitmann, Acacia Aguirre, and Martin Moore-Ede Circadian Technologies, Inc. 24 Hartwell Avenue, Lexington, MA 02421, [email protected]

Abstract. The Expert System “Circadian Alertness Simulator” predicts employee sleep and alertness patterns in 24/7-work environments using rules of complex interactions between human physiology, operational demands and environmental conditions. The system can be tailored to the specific biological characteristics of individuals as well as to specific characteristics of groups of individuals (e.g., transportation employees). This adaptation capability of the system is achieved through a built-in learning module, which allows transferring information from actual data on sleep-wake-work patterns and individual sleep characteristics into an internal knowledge database. The expert system can be used as a fatigue management tool for minimizing the detrimental biological effects of 24-hour operations by providing feedback and advice for work scheduling and/or individualspecific lifestyle training. Consequently, it can help reduce accident risk and human-related costs of the 24/7 economy.

1

Introduction

Expert systems are proven tools for solving complex problems in many areas of everyday life, including fault analysis, medical diagnostic, scheduling, system planning, economic predictions, stock investments, sales predictions, management decisions and many more. Despite the numerous applications, expert systems that allow the prediction of human behaviour are relatively rare. One important application in this area is the prediction of sleep and alertness patterns of individuals and/or industry specific employee groups exposed to 24/7 operations. The last 40 years have seen the evolution of a non-stop 24/7 (24 hours a day, 7 days a week) world driven by the need to increase productivity, enhance customer service and utilize expensive equipment around the clock. However, humans are biologically designed to sleep at night. Millions of years of biological evolution led to the development of an internal biological clock in the brain (suprachiasmatic nucleus), which generates the human body's circadian (approximately 24 hour) rhythms of sleep and wakefulness, and synchronizes them with day and night using V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 104-110, 2003.  Springer-Verlag Berlin Heidelberg 2003

Expert System for Simulating and Predicting Sleep and Alertness Patterns

105

daylight as a resetting tool (Moore-Ede 1982). In this non-stop 24/7 world, people are constantly exposed to mismatches between the biological clock and the work demands, leading to compromised sleep and alertness, decreased performance and accidents, and an increase in human error-related costs. Circadian Technologies has developed the Expert System “Circadian Alertness Simulator” (CAS) that predicts employee sleep and alertness patterns in a 24/7 work environment. This expert system is based on complex rules and interactions between human physiology (e.g., light sensitivity, sleep pressure, circadian sleep and alertness processes), operational demands (e.g., work schedules) and environmental conditions (e.g., light effects). The system can be tailored to the specific biological characteristics of individuals as well as to specific characteristics of groups of individuals (e.g., specific employee populations). This adaptation capability of the system is achieved through a built-in learning module, which applies the system rules and algorithms to actual data on sleep-wake-work patterns and individual characteristics in a training process and stores the results into an internal knowledge database. The Expert System CAS is used as a fatigue management tool for minimizing the detrimental biological effects of 24-hour operations by providing advice for work scheduling or individual-specific lifestyle training. It can therefore help reduce the human-related costs of the 24/7 economy.

2

Expert System CAS – Structure and Learning Capabilities

The Expert System CAS is based on a general concept with model functions, specific rules and free parameters for storage of the information about the light sensitivity, circadian and homeostatic components of sleep/wake behavior and alertness and other individual features of a person, such as chronotype (morningness/eveningness), habitual sleep duration (long/short sleeper), napping capabilities and sleep flexibility (individual capability to cope with mismatches of internal clock and sleeping time constraints). The knowledge base of the system represents a complex interaction of model functions, free parameters and rules. The most important features will be described below. A two process model of sleep and wake regulation assumes an interaction of homeostatic (modeled by exponential functions and two free parameters) and circadian components (modeled by multiple sinus functions and ten free parameters) based on the following rules: (1) If ‘the homeostatic factor reaches the upper circadian threshold' then ‘the system switches from sleep to wake, and (2) If the homeostatic factor reaches the lower circadian threshold'; then ‘the system switches from wake to sleep', provided that sleep is allowed, otherwise the system switches always to sleep below the lower threshold as soon as sleep is permitted (Daan et al. 1984). The characteristics of the circadian component depend on the complex interaction of biological rhythms, individual light sensitivity (modeled by the Phase Response Curve (PRC)) and environmental light exposure strongly correlated to the 24-hour daily cycle which can be summarized by the next two rules: (3) If ‘light is applied before the minimum of the circadian component' then ‘the phase of the biological clock is delayed, and (4) If light is applied after the minimum of the circadian component' then ‘the phase of the biological clock is advanced' (Moore-Ede, et al.

106

Udo Trutschel et al.

1982). As a consequence the circadian curve is shifted to a new position over the next day, modifying all future sleep/wake behaviour. As a further consequence, the sleep/wake behaviour will determine the timing and intensity of light exposure expressed by the following rule: (5) If ‘the state of a person is sleep' then ‘the light exposure is blocked' leading to an unchanged phase of the circadian component. The expert knowledge for the sleep and alertness prediction is obtained through a training process, in which the expert system learns from actual data of individual sleep personality, sleep/wake-activity and light exposure (Fig 1.). Training Process Minimize Target Function

Interference Engine Input (Interface)

Output (Interface)

Sleep Personality Wake/Sleep Data Light Exposure

Sleep Times Alertness Level

Knowledge Base External Database of Sleep Personality Profiles of Shiftworkers

External Database of Wake/Work/Sleep Data of Shiftworkers During Work Times

Example-Database of Wake/Sleep Data of Shiftworkers during Vacation Times

Fig. 1. Training procedure for the Expert system CAS

The target of the training is to determine the free parameter that result in the best match of actual and simulated sleep and alertness data. After the training is complete, a specific data set is stored in the knowledge database of the Expert System CAS and can be used to simulate and predict sleep and alertness patterns for given specific work-rest schedules (Fig. 2).

Interference Engine Input (Interface)

Output (Interface)

Person/Group ID Wake/Work Data Light Exposure

Sleep/Alertness Patterns Fatigue Risk Score Risk Times

Knowledge Base Fig. 2. Sleep and alertness predictions of the trained Expert System CAS

Expert System for Simulating and Predicting Sleep and Alertness Patterns

107

The simulated and predicted sleep behavior of a person has direct consequences for the alertness level as a function of time, in this paper expressed by two overall easy to understand output parameters: (1) Fatigue Risk Score (0 = lowest fatigue, 100 = highest fatigue), which calculates the cumulative fatigue risk, and (2) Percentage of Work Time below a critical alertness threshold.

3

“Short-Lark”- and “Long-Owl”-Test Cases for Shift Scheduling

To illustrate the capabilities of the Expert System CAS, two artificial people were simulated in a shiftwork environment, working 8-hour backward rotating shift schedule (7 working days-3 shift start times). They are called “Short-Lark” (extreme morning type with short habitual sleep time) and “Long-Owl” (extreme evening type with long habitual sleep time). Without any external constraints such as work or other blocked times for sleep, the “Short-Lark” sleeps every day from 2200 to 0400 and the “Long-Owl” sleeps every day from 0100 to 1100. The expert system was trained with both sleep patterns, and the results were stored in the knowledge database as personspecific input vectors. The predicted sleep and alertness behavior of our test subjects when working 8-hour backward rotating shift schedule with the assumption that sleep is forbidden during the blue marked working times is shown in Fig. 3 and Fig. 4.

Fig. 3. Predicted sleep pattern for “Short-Lark” working 8-hour shift schedule

It is clear from Fig. 3 that the evening and especially the night shift interfere with the normal sleep behavior (black bars) of the “Short-Lark,” creating conflicts with the internal biological clock. This should have severe consequences for the alertness level

108

Udo Trutschel et al.

during work times (blue bars) as expressed by a high overall Fatigue Risk Score of 72, and 37.4% work performed at extreme risk times.

Fig. 4. Predicted sleep pattern for “Long-Owl” working 8-hour shift schedule

Completely different is the sleep pattern of the “Long-Owl” (Fig.4). Here the work schedule interferes with sleep (black bars) only during the day shift, and the disturbed night sleep is recovered immediately after the shift. Therefore, the conflict with the internal biological clock is only modest and reasonable alertness levels during work times are maintained, expressed by a lower overall Fatigue Risk Score of 57 and only 12.3% of work at extreme risk times.

4

Group-Specific Application in the Transportation Industry

Fatigue-related safety concerns are particularly relevant for safety-critical occupations, such as vehicle operators in the rail, road transport or bus industry. Accidents caused by driver fatigue are often severe, as the drowsy driver may not take evasive action (i.e. brake or steer) to avoid a potential collision. In recognition of this, the U.S. Department of Transportation has identified human fatigue as the “Number One” safety problem with a cost in excess of $12 billion per year. To address this need Circadian Technologies, Inc. over the past ten years in partnership with the rail & trucking industry has collected work/rest/sleep and alertness data from over 10,000 days of rail and truck operators in actual operating conditions. Using the training process depicted in Fig.1 an Expert System “CAS-Transportation” was created and applied to a specific trucking operation. “CAS-Transportation” was used to simulate

Expert System for Simulating and Predicting Sleep and Alertness Patterns

109

sleep and alertness behavior and predict fatigue risk from the work-rest data of truck drivers. The results were correlated with the actual accident rates and costs of a 500truck road transport operation. Using “CAS-Transportation” as feedback tool, nonexperts (managers, dispatchers) were provided with the relative risk of accidents due to driver fatigue from any planned sequence of driving and resting hours, and were ask to make adjustments to the working hours to minimize the overall Fatigue Risk Score. As a result of the intervention, the mean Fatigue Risk Score of the group was significantly reduced from 46.8 to 28.9 (dark line in Fig. 5a, 5b). The reduction of the Fatigue Risk Score was associated with a reduction in the frequency and severity of accidents (Moore-Ede et al., 2003). The total number of truck accidents dropped 23.3% and the average cost per accident was significantly reduced (65.8%).

Frequency in Percent

Before

8 7 6 5 4 3 2 1 0

0

10

20

30

40

50

60

70

80

90

100

Fatigue Index Fig. 5a. Fatigue Risk Score before feedback from Expert System CAS-Transportation

Frequency in Percent

After

8

p< 0.0001

7 6 5 4 3 2 1 0

0

10

20

30

40

50

60

70

80

90

Fatigue Index Fig. 5b. Fatigue Risk Score after feedback from Expert System CAS-Transportation

100

110

Udo Trutschel et al.

Using the feedback from the Expert System CAS in a risk-informed, performancebased safety program gives managers and dispatchers incentives to address some of the most important causes of driver fatigue, and therefore of fatigue-related highway accidents. This risk-informed, performance-based approach to fatigue minimization enables non-experts (managers and dispatchers) to make safety-conscious operational decisions while having sufficient flexibility to balance the specific business needs of their operation.

5

Conclusion

The Expert System CAS is able to acquire knowledge about individual sleep personalities or industry specific groups in order to predict Fatigue Risk Scores and/or alertness patterns for given shift schedule options or real world work/wake data. Missing sleep patterns can be replaced through simulated sleep patterns with considerable certainty. From a database of many shift schedules acceptable schedule options can be selected by exposing these options to different sleep personalities or industry specific groups and predicting possible fatigue risks. In addition, the simulation capabilities of Expert System CAS are supporting the decision process during schedule design. The examples (“Short-Lark”, “Long-Owl”, “Transportation Employees”) presented in this paper as well as the shift schedule design process in general can be considered as forward-chaining application of the Expert System CAS. Shift schedule design starts with the operational requirements, investigates the large number of possible solutions for shift durations, shift starting times, direction of rotation and sequence of days ‘on' and days ‘off', and makes the schedule selection based on the criteria fatigue risk and/or alertness levels specified by the user. In addition, the Expert System CAS can be used in the investigation of a fatigue related accident. This specific application represents an example of a backward chaining approach where fatigue as a probable cause of an accident is assumed and the Expert System CAS attempts to find the supporting evidence to verify this assumption.

References [1] [2] [3]

Daan S., Beersma DG., Borbely A. Timing of human sleep: recovery process gated by a circadian pacemaker. Am. J. Physiology 1984; 246: R161-R178. Moore-Ede M. Sulzman F., Fuller C. The clocks that time us. Harvard University Press, Cambridge, 1982. Moore-Ede M., Heitmann A., Dean C., Guttkuhn R. Aguirre A. Trutschel U. Circadian Alertness Simulator for Fatigue Risk Assessment in Transportation: Application to Reduce Frequency and Severity of Truck Accidents. Aviation, Space and Environmental Medicine 2003, in press.

Two Expert Diagnosis Systems for SMEs: From Database-Only Technologies to the Unavoidable Addition of AI Techniques Sylvain Delisle1 and Josée St-Pierre2 Institut de recherche sur les PME Laboratoire de recherche sur la performance des entreprises Université du Québec à Trois Rivières 1 Département de mathématiques et d'informatique 2 Département des sciences de la gestion C.P. 500, Trois-Rivières, Québec, Canada, G9A 5H7 Phone: 1-819-376-5011 + 3832 Fax: 1-819-376-5185 {sylvain_delisle,josee_st-pierre}@uqtr.ca www.uqtr.ca/{~delisle, dsge}

Abstract. In this application-oriented paper, we describe two expert diagnosis systems we have developed for SMEs. Both systems are fully implemented and operational, and both have been put to use on data from actual SMEs. Although both systems are packed with knowledge and expertise on SMEs, neither has been implemented with AI techniques. We explain why and how both systems relate to knowledgebased and expert systems. We also identify aspects of both systems that will benefit from the addition of AI techniques in future developments.

1

Expertise for Small and Medium-Sized Enterprises (SMEs)

The work we describe here takes place within the context of the Research Institute for SMEs—the Institute's core mission (www.uqtr.ca/inrpme/anglais/index.html) is to support fundamental and applied research to foster the advancement of knowledge on SMEs to contribute to their development. The specific lab in which we have conducted the research projects we refer to in this paper is the LaRePE (LAboratoire de REcherche sur la Performance des Entreprises: www.uqtr.ca/inrpme/larepe/). This lab is mainly concerned with the development of scientific expertise on the study and modeling of SMEs' performance, including a variety of interrelated subjects such as finance, management, information systems, production, technology, etc. The vast majority of research projects carried out at the LaRePE involves both theoretical and practical aspects, often necessitating in-field studies with SMEs. As a result, our research projects always attempt to provide practical solutions to real problems confronting SMEs. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 111-125, 2003.  Springer-Verlag Berlin Heidelberg 2003

112

Sylvain Delisle and Josée St-Pierre

In this application-oriented paper we briefly describe two expert diagnosis systems we have developed for SMEs. Both can be considered as decision support systems— see [15] and [18]. The first is the PDG system [5]: a benchmarking software that evaluates production and management activities, and the results of these activities in terms of productivity, profitability, vulnerability and efficiency. The second is the eRisC system [6]: a software that helps identify, measure and manage the main risk factors that could compromise the success of SME development projects. Both systems are fully implemented and operational. Moreover, both have been put to use on data from actual SMEs. What is of particular interest here, especially from a knowledge-based systems perspective, is the fact that although both the PDG and the eRisC systems are packed with knowledge and expertise on SMEs, neither has been implemented with Artificial Intelligence (AI) techniques. However, if one looks at them without paying attention to how they have been implemented, they qualify as “black-box” diagnostic expert systems. In the following sections, we provide further details on both systems and how they relate to knowledge-based and expert systems. We also identify aspects of both systems that could benefit from the addition of AI techniques in future developments.

2

The PDG System: SME Performance Diagnostic

2.1

An Overview of the PDG System

The PDG system evaluates a SME from an external perspective and on a comparative basis in order to produce a diagnosis of its performance and potential, complemented with relevant recommendations. Although we usually refer to the PDG system as a diagnostic system it is in fact a hybrid diagnostic-recommendation system as it not only identifies the evaluated SME's weaknesses but it also makes suggestions on how to address these weaknesses in order to improve the SME's performance. An extensive questionnaire is used to collect relevant information items on the SME to be evaluated. Data extracted from the questionnaire is computerized and fed into the PDG system. The latter performs an evaluation in approximately 3 minutes by contrasting the particular SME with an appropriate group of SMEs for which we have already collected relevant data. The PDG's output is a detailed report in which 28 management practices (concerning human resources management, production systems and organization, market development activities, accounting, finance and control tools), 20 results indicators and 22 general information items are evaluated, leading to 14 recommendations on short term actions the evaluated SME could undertake to improve its overall performance. As shown in Figure 1 (next page), the PDG expert diagnosis system is connected to an Oracle database which collects all the relevant data for benchmarking purposes— the PDG also uses the SAS statistics package, plus Microsoft Excel for various calculations and the generation of the final output report. The PDG reports are constantly monitored by a team of multidisciplinary human experts in order to ensure that recommendations are valuable for the entrepreneurs. This validation phase, which always takes place before the report is sent to the SME, is an occasion to make further im-

Two Expert Diagnosis Systems for SMEs

113

provements to the PDG system, whenever appropriate. It is also a valuable means for the human experts to update their own expertise on SMEs. Figure 1 also shows that an intermediary partner is part of the process in order to guarantee confidentiality: nobody in our lab knows to what companies the data are associated. data and results

Oracle Database

report

report

Lab questionnaire financial data

questionnaire, financial data

report

Entrepreneur (SME)

Intermediary Partner

PDG Expert Diagnosis System

information expertise

Multidisciplinary Team of Human Experts

Fig. 1. The PDG system: evaluation of SMEs, from an external perspective and on a comparative basis, in order to produce a diagnosis of their performance and potential

The current version of the PDG system has been in production for 2 years. So far, we have produced more than 600 reports and accumulated in the database the evaluation results of approximately 400 different manufacturing SMEs. A recent study was made of 307 Canadian manufacturing SMEs that have used the PDG report, including 49 that have done so more than once. Our results show that the PDG's expert benchmarking evaluation allows these organisations to improve their operational performance, confirming the usefulness of benchmarking but also, the value of the recommendations included in the PDG report concerning short term actions to improve management practices [17]. 2.2

Some Details on the PDG System

The PDG's expertise is located in two main components: the questionnaire and the benchmarking results interpretation module—in terms of implementation, the PDG uses an Oracle database, the SAS statistical package, and Microsoft Excel. The first version of the questionnaire was developed by a multidisciplinary team of researchers in the following domains: business strategy, human resources, information systems, industrial engineering, logistics, marketing, economics, and finance. The questionnaire development team was faced with two important challenges that quickly became crucial goals: 1) find a common language (a shared ontology) that would allow researchers to understand each other and, at the same time, would be accessible to entrepreneurs when answering the questionnaire, and 2) identify long-term performance indicators for SMEs, as well as problem indicators, while keeping contents to a minimum since in-depth evaluation was not adequate.

114

Sylvain Delisle and Josée St-Pierre

The team was able to meet these two goals by the assignment of a “knowledge integrator” role to the project leader. During the 15-month period of its development, the questionnaire was tested with entrepreneurs in order to ensure that it was easy to understand both in terms of a) contents and question formulation, and b) report layout and information visualization. All texts were written with a clear pedagogical emphasis since the subject matter was not all that trivial and the intended lectureship was quite varied and heterogeneous. Several prototypes were presented to entrepreneurs and they showed a marked interest for graphics and colours. Below, Figure 2 shows a typical page of the 10-page report produced by the PDG system.

Fig. 2. An excerpt from a typical report produced by the PDG system. The evaluated SME's performance is benchmarked against that of a reference group

The researchers' expertise was precious in the identification of vital information that would allow the PDG system to rapidly produce a general diagnosis of any manufacturing SME. The diagnosis also needed to be reliable and complete, while being

Two Expert Diagnosis Systems for SMEs

115

comprehensible by typical entrepreneurs as we pointed out before. This was pioneering research work that the whole team was conducting. Indeed, other SME diagnosis systems are generally financial and based on valid quantitative data. The knowledge integrator mentioned above played an important part in this information engineering and integration process. Each expert had to identify practices, systems, or tools that had to be implemented in a manufacturing SME to ensure a certain level of performance. Then, performance indicators had to be defined in order to measure to what extent these individual practices, systems, or tools were correctly implemented and allowed the enterprise to meet specific goals—the relationship between practices and results is a distinguishing characteristic of the PDG system. Next, every selected performance indicator was assigned a relative weight by the expert and the knowledge integrator. This weight is used to position the enterprise being diagnosed with regard to its reference group, thus allowing the production of relevant comments and recommendations. The weight is also used to produce a global evaluation that will be displayed in a synoptic table. Contrary to many performance diagnostic tools in which the enterprise's information is compared to norms and standards (e.g. [11]), the PDG system evaluates an enterprise relative to a reference group selected by the entrepreneur. Research conducted at our institute seriously questions this use of norms and standards: it appears to be dubious for SMEs as they simply are too heterogeneous to support the definition of reliable norms and standards. Performance indicators are implemented as variables in the PDG system—more precisely in its database, and in the benchmarking results interpretation module (within the report production module). These variables are defined in terms of three categories: 1) binary variables, which are associated with yes/no questions; 2) scale variables, which are associated with the relative ranking of the enterprise along a 1 to 4 or a 1 to 5 scale, depending on the question; and 3) continuous (numerical) variables, which are associated with numerical figures such as the export rate or the training budget. Since variables come in different types, they must also be processed differently at the statistical level, notably when computing the reference group used for benchmarking purposes. In order to characterize the reference group with a single value, a central tendency measure that is representative of the reference group's set of observations is used. Depending on the variable category and its statistical distribution, means, medians, or percentages are used in the benchmarking computations. Table 1 (next page) shows an example of how the evaluated enterprise's results are ranked and associated with codes that will next be used to produce the various graphics in the benchmarking report. The resulting codes (see CODE in Table 1) indicate the evaluated enterprise's benchmarking result for every performance indicator. They are then used by the report generation module to produce the benchmarking output report, which contains many graphical representations, as well as comments and recommendations. The codes are used to assign colours to the enterprise, while the reference group is always associated with the beige colour. For instance, if the enterprise performs better than its reference group, CODE = 4 means colour is green forest. However, in the opposite situation, CODE = 4 would mean colour is red. Other colours with other meanings are yellow, salmon, and green olive. Figure 2 above illustrates how these coloured graphics look like (although they appear only in black and white here).

116

Sylvain Delisle and Josée St-Pierre

Table 1. Some aspects of the representation of expertise within the PDG system with performance indicators implemented as variables. This table shows three (3) variables: one scale variable (participative management), one binary (remuneration plan), and one continuous numerical (fabrication cost). Legend: SME = variable value for the evaluated enterprise; MEA = mean value of the variable in the reference group; RG = reference group; MED = median value of the variable in the reference group; CODE = resulting code for the evaluated enterprise

Scale variable if SME >= (1.25 x MEA), then CODE = 4

Binary variable if SME = 1 and 10% of RG =1 then CODE = 4 if SME = 1 and 25% of RG = 1 then CODE = 3 if SME = 1 and 50% of RG = 1 then CODE = 2 if SME = 1 and 75% of RG = 1 then CODE = 1 if SME = 1 and 90% of RG = 1 then CODE = 0

if SME >= (1.10 x MEA), then CODE = 3 if SME >= (1.00 x MEA), then CODE = 2 if SME >= (0.90 x MEA), then CODE = 1 if SME >= (0.75 x MEA), then CODE = 0 Example: participative Example: remuneration plan management

Continuous (numerical) variable if SME >= (1.25 x MED), then CODE = 4 if SME >= (1.10 x MED), then CODE = 3 if SME >= (1.00 x MED), then CODE = 2 if SME >= (0.90 x MED), then CODE = 1 if SME >= (0.75 x MED), then CODE = 0 Example: fabrication cost

3

The eRisC System: Risk Assessment of SME Development Projects

3.1

An Overview of the eRisC System

SMEs often experience difficulties accessing financing to support their activities in general, and their R&D and innovation activities in particular—see [1], [4], [8], and [9]. Establishing the risk levels of innovation activities can be quite complex and there is no formalized tool to help financial analysts assess them and correctly implement compensation and financing terms that will satisfy both lenders and entrepreneurs. This situation creates a lot of pressure on the cash resources of innovating SMEs. Based on our team's experience with SMEs and expertise with the assessment of risk, and thanks to the contribution of several experts that constantly deal with SMEs development projects, we have developed a state-of-the-art Web-based software called eRisC (see Figure 3 next page). The eRisC (https://oraprdnt.uqtr.uquebec.ca/erisc/index.jsp) expert diagnosis system identifies, measures and allows to manage the main risk factors that could compromise the success of SMEs development projects including expansion, export and innovation projects, each of which is the object of a separate section of the software. An extensive dynamic, Web-based questionnaire is used to collect relevant information items on the SME expansion project to be evaluated.

Two Expert Diagnosis Systems for SMEs

Entrepreneur (SME) or other agent

data

117

Oracle Database

data and results

report

report

Internet data

eRisC Expert Diagnosis System

information expertise

Multidisciplinary Team of Human Experts

Fig. 3. The eRisC system: a Web-based software that helps identify, measure and manage the main risk factors involved in SME development projects

The contents of the questionnaire are based on an extensive review of literature in which we identified over 200 risk factors acting upon the success of SMEs development projects. For example, factors associated with the export activity are export experience, commitment/planning, target market, product, distribution channel, shipping and contractual/financial aspects. These seven elements are broken down into 21 subelements involving between 58 and 216 questions—the number of questions ranges from 59 to 93 for an expansion project, from 58 to 149 for an export project, and from 86 to 216 for an innovation project. Data extracted from the questionnaire is fed into an elaborated knowledge-intensive algorithm that computes risk levels and identifies main risk elements associated with the evaluated project. As shown in Figure 3, the eRisC expert diagnosis system is connected to an Oracle database which collects all the relevant data. Since eRisC was developed after the PDG system, it benefited from the most recent Web-based technologies (e.g. Oracle Java) and was right from the start designed as a fully automated system. More precisely, contrary to the PDG reports, there is no need to constantly monitor eRisC's output reports—thus the dotted arrows on the right-hand side of Figure 3 above. eRisC was developed for and validated by entrepreneurs, economic agents, lenders and investors, to identify the main risk factors of SMEs development projects in order to improve their success rates and facilitate their financing. As of now, various organizations are starting to put eRisC at use in real life situations, allowing us to collect precious information in eRisC's database on SMEs projects and their associated risk assessment. We have a group of 30 users, from various organizations and domains, who currently use eRisC for real-life projects and who provide us with useful feedback for marketing purposes.

118

Sylvain Delisle and Josée St-Pierre

3.2

Some Details on the eRisC System

eRisC's contents was developed by combining various sources of information, knowledge and expertise: the literature on business failure factors and the one on project management, our colleagues' expertise on SMEs, and invaluable information from various agents dealing with these issues on a day-to-day basis, such as lenders, investors, entrepreneurs, economic advisors and experts. Based on this precious and abundant information, we first assembled a long list of potential risk factors that could disturb or influence significantly the development of SMEs projects. In a second phase, we had to reduce the original list of risk factors which was simply too long to be considered in its entirety in real-life practical situations. In order to do that, we considered the relative importance and influence of risk factors on the failure of evaluated projects. Once this pruning was completed, and after we ensured that we had not discarded important factors, the remaining key factors were grouped into meaningful generic categories. We then developed sets of questions and subquestions that would support the measurement of the actual risk level of a project. This also allowed us to add a risk management dimension to our tool by inviting the user to identify with greater precision facets that could compromise the success of the project, and thus allowing a better control with the implementation of appropriate corrective measures. A relatively complex weight system was also developed in order to associate a quantitative measure to individual risk elements, to rank these elements, and to compute a global risk rating for the evaluated project—see Figure 4 below.

Fig. 4. An excerpt from the expansion project questionnaire. The only acceptable answers to questions are YES, NO, NOT APPLICABLE, DON'T KNOW

Two Expert Diagnosis Systems for SMEs

119

In a third and final phase, the contents of eRisC was validated with many potential users and their feedback was taken into consideration to make adjustments on several aspects such as question formulation, term definition, confidentiality of information, etc. At this point, the tool was still “on paper”, as an extensive questionnaire (grid), and had not been implemented yet. So an important design decision had to be made at the very beginning of the implementation phase: how to convert the on-paper, large, and static questionnaire into an adequate form to be implemented into the eRisC software? As we examined various possibilities, we gradually came to look at it more and more as an interactive and dynamic document. In this dynamic perspective, the questionnaire would be adaptable to the users' needs for the specific project at hand. In a sense, the questionnaire is at the meeting point of three complementary dimensions: the risk evaluation model as defined by domain experts, the user's perspective as a domain practitioner; and the computerized rendering of the previous two dimensions. Moreover, from a down-to-earth, practical viewpoint, users would only be interested in the resulting software if it proved to be quick, user friendly, and better than their current non-automated tool.

Fig. 5. Risk Assessment Results Produced by eRisC

With regard to the technological architecture, eRisC is based on the standard 3tiered Web architecture for which we selected Microsoft's Internet Explorer (Web browser) for the client side, the Tomcat Web server for the middleware, and, for the data server, the Oracle database server (Oracle Internet Application Server 8.1.7 Enterprise Edition) running on a Unix platform available at our University in a secured environment. All programming was done with JSP (JaveServer Page) and JavaScript. A great advantage of the 3-tiered model is that it supports dynamic Web applications

120

Sylvain Delisle and Josée St-Pierre

in which the contents of Web pages to be shown on the user's (client's) Web browser is computed “on the fly”, i.e. dynamically, from the Web server and the information it fetched from the database server in response to the user's (client's) request. The five (5) main steps of processing involved in a project risk evaluation with eRisC are: 1) dynamic creation of the questionnaire, according to the initial options selected by the user; 2) project evaluation (questions answering: see Figure 4) by the user; 3) saving of data (user's answers) to the database; 4) computation of results; and 5) presentation of results in an online and printable report. Once phases 1 to 3 are completed, after some 30 minutes on average, eRisC only takes a minute or so to produce the final results, all this taking place online. Final results include a numerical value representing the risk rating (a relative evaluation between 0 and 100) for the specific SME project just evaluated, combined with the identification of at least the five most important risk factors (to optionally perform risk mitigation) within the questionnaire's sections used to perform the evaluation, plus a graphical (pie) representation showing the risk associated with every section and their respective weight in the computation of the global project risk rating—see Figure 5 on the previous page and Figure 6 below. The user can change these weights to adjust the evaluation according to the project's characteristics, or to better reflect her/his personal view on risk evaluation. These “personal” weights can also be saved by eRisC in the user's account so that the software can reuse them the next time around.

Fig. 6. Mitigation Report and Risk Assessment Simulation in eRisC

Two Expert Diagnosis Systems for SMEs

121

When sufficient data will have been accumulated in eRisC's database, it will be possible to establish statistically-based weight models for every type of user. Amongst various possibilities, this will allow entrepreneurs to evaluate their projects with weights used by bankers, allowing them to better understand the bankers' viewpoint when entrepreneurs ask for financing assistance. Finally, mitigation elements are associated with many risk factors listed in eRisC's output report. Typically associated with the most important risk factors, these mitigation elements suggest ways to reduce the risk rating just computed. The user can even re-compute the risk level with the hypothesis that the selected mitigation elements have been put in place in order to assess the impact they may have on the project's global risk level. A new graphic will then be produced showing a comparison of the risk levels before and after the mitigation process.

4

Conclusion: AI-less Intelligent Decision Support Systems

A good deal of multi-domain expertise and informal knowledge engineering was invested into the design of the PDG and the eRisC expert diagnosis systems. In fact, at the early stage of the PDG project, which was developed before eRisC, it was even hoped that an expert-system approach would apply naturally to the task we were facing. Using an expert system shell, a prototype expert system was in fact developed for a subset of the PDG system dealing only with human resources. However, reality turned out to be much more difficult than anticipated. In particular, the knowledge acquisition, knowledge modelling, and knowledge validation/verification phases ([7], [12], [11], [16], [3]) were too demanding in the context of our resources constraints especially in the context of a multidisciplinary domain such as that of SME for which little formalized knowledge exists. Indeed, many people were involved, all of them in various specialization fields (i.e. management, marketing, accounting, finance, human resources, engineering, technical, information technology, etc.) and with various backgrounds (researchers, graduate students, research professionals and, of course, entrepreneurs). One of the main difficulties that hindered the development of the PDG as an expert system was the continuous change both the questionnaire and the benchmarking report were undergoing during the first three years of the project. So at the same time the research team was trying to develop a multidisciplinary model of SME performance evaluation, users' needs had to be considered, software development had to be carried out, and evaluation reports had to be produced for participating SMEs. This turned out to be a rather complicated situation. The prototype expert system mentioned above was developed in parallel with the current version, although only for the subset dealing with human resources—see [10] and [19] for examples of expert systems in finance. The project leader's knowledge engineer role was very difficult since several experts from different domains were involved and the extraction and fusion of these various fields of expertise had never been done before. Despite the experts' valuable experience, knowledge, and good will, they had never been part of a similar project before. The modelling of such rich, complex, and vast information, especially for SMEs, was an entirely new challenge both scientifically and technically. Indeed, because of their heterogeneous nature, and contrary to large enterprises, SMEs are

122

Sylvain Delisle and Josée St-Pierre

much more difficult to model and evaluate. For instance, the implementation of certain management practices may be necessary and usual for traditional manufacturing enterprises, but completely inappropriate for a small enterprise subcontracting for a large company or a prime contractor. These important considerations and difficulties, not mentioning the consequences they had on the project's schedule and budget, lead to the abandon of the expert system after the development of a simple prototype. As to the eRisC system, since it was another multi-domain multi-expert project, and thanks to our prior experience with the PDG system, it was quickly decided to stay away from AI-related approaches and techniques. During the development of eRisC's questionnaires, we realized how risk experts always tended to model risk assessment from their own perspective and from their own personal knowledge, as reported in the literature. This is why we built our risk assessment model from many sources, thanks to a comprehensive review of literature and the availability of several experts, in order to ensure we ended up with an exhaustive list of risk determining factors for SME projects. For instance, here are the main different perspectives (see e.g. [14]). • • •

Bankers and lenders care mostly about financial aspects and tend to neglect qualitative dimensions that indicate whether the enterprise can solve problems and meet challenges in risky projects. Entrepreneurs do not realize that their implication in the project can in fact constitute a major risk from their partners' viewpoint. Economic consultants and advisors have a specialized background that may prevent them from having a global perspective on the project.

Obviously, it is the fusion of all these diverse and complementary expertise sources that would have been used to develop the knowledge base of an expert-system version of the current eRisC system. However, this was simply impossible given the timetable and resources available to us. Of course, this does not mean that AI tools were inappropriate for those two projects. As a research team involved in an applied project, we made a rational decision based on our experience on a smaller scale experiment (i.e. the PDG prototype expert system on human resources), on our time and budget constraints, and on the welldocumented fact that multi-domain multi-expert knowledge acquisition and modelling constitutes a great challenge. Yet another factor that had great influence on our design decisions was the fact that both projects started out on paper as questionnaires which led naturally to database building and use of all the database-related software development. Thus, both the PDG and eRisC systems ended up as knowledge-packed systems built on database technology. However, as we briefly discuss in Section 5 below, we are now at a stage where we plan the addition of AI-related techniques and tools. The current versions of the PDG and eRisC systems, although not implemented with AI techniques, e.g. knowledge base of rules and facts, inference engine, etc. (see, e.g., [13], [18]), qualifies as “black-box” expert diagnosis systems. These unique systems are based on knowledge, information and algorithms that allow them to produce outputs that only a human expert, or in fact several human experts in different domains, would be able to produce in terms of diagnosis and recommendation quality. These reports contain mostly coloured diagrams and simple explanations that are formulated in plain English (or French) so that SMEs entrepreneurs can easily under-

Two Expert Diagnosis Systems for SMEs

123

stand them. The PDG is the only system that can be said to use some relatively old AI techniques. Indeed, the comments produced in the output report are generated via a template-based approach, an early technique used in natural language processing.

5

Future Work: Bringing Back AI Techniques into the Picture

The PDG and the eRisC systems are now at a stage where we can now reconsider the introduction of AI techniques in new developments. The main justification for this is the need to eliminate human intervention while preserving high quality outputs, based on rare highly-skilled knowledge and expertise. We have started to develop new modules that will increase even more the intelligence features of both systems. Here is a short, non-exhaustive list accompanied with brief explanations: •

•

•

•

•

Development of data warehouses and data mining algorithms to facilitate statistical processing of data and extend knowledge extraction capabilities. Such extracted knowledge will be useful to improve the systems' meta-knowledge level, which could be used in the systems' explanations for instance, and also to broaden human experts' domain knowledge. This phase is already in progress. The huge number of database attributes and statistical variables manipulated in both systems is overwhelming. A conceptual taxonomy, coupled with an elaborated data dictionary, has now become a necessary addition. For instance, the researcher should be able to find out quickly to what concepts a particular attribute (or variable) is associated, to what computations or results it is related, and so on. This phase has recently begun. Development of an expert system to eliminate the need for any human intervention in the PDG system. Currently, a human expert must revise all reports before they are sent to the SME. Most of the time, only minor adjustments are required. The knowledge used to perform this final revision takes into consideration individual results produced in various parts of the benchmarking report and analyze potential consequences of interrelationships between them in order to ensure that conclusions and recommendations of the evaluated SME are both valid and coherent. This phase is part of our future work. Augment current systems with case-based reasoning and related machine learning algorithms. In several aspects of both systems, evaluation of the problem at hand could be facilitated if it were possible to establish relationships with similar problems (cases) already solved before. Determining the problems' salient features to support this approach would also offer good potential to lessen the users' burden during the initial data collection phase. This phase is part of our future work. Study the potential of agent technology to reengineer some elements of both systems, especially from a decision support system perspective [2]. This could be especially interesting for the modelling and implementation of distributed sources of expertise that contribute to decision processing. For example, in the PDG system each source of expertise in the performance evaluation of a SME could be associated with a distinct agent controlling and managing its own knowledge

124

Sylvain Delisle and Josée St-Pierre

base. Interaction and coordination between these agents would be crucial aspects of a PDG system based on a community of cooperative problem-solving agents.

References [1] [2] [3] [4] [5]

[6]

[7] [8] [9] [10] [11] [12] [13] [14]

Beaudoin R. and J. St-Pierre (1999). “Le financement de l'innovation chez les PME”, Working paper for Développement Économique Canada, 39 pages. Available: http://www.DEC-CED.gc.ca/fr/2-1.htm Bui T. and J. Lee (1999). “An Agent-Based Framework for Building Decision Support Systems”, Decision Support Systems, 25, 225-237. Caulier P. and B. Houriez (2001). “L'approche mixte expérimentée (modélisation des connaissances métiers)”, L'Informatique Professionnelle, 32 (195), juin-juillet, 30-37. Chapman R.L., C.E. O'Mara, S. Ronchi and M. Corso (2001). “Continuous Product Innovation : A Comparison of Key Elements across Different Contingency Sets”, Measuring Business Excellence, 5(3), 16-23. Delisle S. and J. St-Pierre (2003). “An Expert Diagnosis System for the Benchmarking of SMEs' Performance”, First International Conference on Performance Measures, Benchmarking and Best Practices in the New Economy (Business Excellence '03), Guimaraes (Portugal), 10-13 June 2003, to appear. Delisle S and J. St-Pierre (2003). “SME Projects: A Software for the Identification, Assessment and Management of Risks”, 48th World Conference of the International Council for Small Business (ICSB-2003), Belfast (Ireland), 15-18 June 2003, to appear. Fensel D. and F. Van Harmelen (1994). “A Comparison of Languages which Operationalize and Formalize KADS Models of Expertise”, The Knowledge Engineering Review, 9(2), 105-146. Freel M.S. (2000). “Barriers to Product Innovation in Small Manufacturing Firms”, International Small Business Journal, 18(2), 60-80. Menkveld A.J. and A.R. Thurik (1999). “Firm Size and Efficiency in Innovation: Reply”, Small Business Economics, 12, 97-101. Nedovic L. and V. Devedzic (2002), “Expert Systems in Finance—A CrossSection of the Field”, Expert Systems with Applications, 23, 49-66. Matsatsinis N.F., M. Doumpos and C. Zopounidis (1997). “Knowledge Acquisition and Representation for Expert Systems in the Field of Financial Analysis”, Experts Systems with Applications, 12(2), 247-262. Rouge A., J.Y. Lapicque, F. Brossier and Y. Lozinguez (1995). “Validation and Verification of KADS Data and Domain Knowledge”, Experts Systems with Applications, 8(3), 333-341. Santos J., Z. Vale and C. Ramos (2002). “On the Verification of an Expert System: Practical Issues”, Lecture Notes in Artificial Intelligence #2358, 414424. Sarasvathy, D.K., H.A. Simon and L. Lave (1998). “Perceiving and Managing Business Risks : Differences Between Entrepreneurs and Bankers”, Journal of Economic Behavior and Organization, 33, 207-225.

Two Expert Diagnosis Systems for SMEs

125

[15] Shim J.P., M. Warkentin, J.F. Courtney, D.J. Power, R. Sharda and C. Carlsson (2002). “Past, Present, and Future of Decision Support Technology”, Decision Support Systems, 33, 111-126. [16] Sierra-Alonso A. (2000). “Definition of a General Conceptualization Method for the Expert Knowledge”, Lecture Notes in Artificial Intelligence #1793, 458469. [17] St-Pierre J., L. Raymond and E. Andriambeloson (2002). “Performance Effects of the Adoption of Benchmarking and Best Practices in Manufacturing SMEs”, Small Business and Enterprise Development Conference, The University of Nottingham. [18] Turban E. and J.E. Aronson (2001). Decision Support Systems and Intelligent Systems, Prentice Hall. [19] Wagner W.P., J. Otto and Q.B. Chung (2002). “Knowledge Acquisition for Expert Systems in Accounting and Financial Problem Solving”, Knowledge-Based Systems, 15, 439-447.

Using Conceptual Decision Model in a Case Study Miki Sirola Helsinki University of Technology Laboratory of Computer and Information Science P.O.Box 5400 , FIN-02015 HUT, Finland [email protected]

Abstract. Decision making is mostly based on decision concepts and decision models built in decision support systems. Type of decision problem determines application. This paper presents a case study analysed with a conceptual decision model that utilises rule-based methodologies, numerical algorithms and procedures, statistical methodologies including distributions, and visual support. Selection of used decision concepts is based on case-based needs. Fine tuning of the model is done during construction of the computer application and analysis of the case examples. A kind of decision table is built including pre-filtered decision options and carefully chosen decision attributes. Each attribute is weighted, decision table values are given, and finally total score is calculated. This is done with a many-step procedure including various elements. The computer application is built on G2 platform. The case example choice of career is analysed in detail. The developed prototype should be considered mostly as an advisory tool in decision making. More important than the numerical result of the analysis is to learn about the decision problem. Evaluation expertise is needed in the development process. The model constructed is a kind of completed multi-criteria decision analysis concept. This paper is also an example of using a theoretical methodology in solving a practical problem.

1

Introduction

There are aspects in decision making that need to be paid special attention to. Various decision concepts have been composed and many kind of decision models have been built to provide the decision support systems with best possible aid. Decision maker has to be able to integrate all valuable information available and ennoble out good enough decision in each particular decision case. Visual support is often very valuable for the decision maker. Visual support can mean several things of course. Visualisation of process data is a basic example of giving to the decision maker information that can be valuable. Visualisation of process variables begin with simple plots and time series, and may go into more and more complicated development. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 126-133, 2003.  Springer-Verlag Berlin Heidelberg 2003

Using Conceptual Decision Model in a Case Study

127

Such support methodologies as decision tables, decision trees, flow diagrams, and rule-based methods ought to be mentioned. Calculation algorithms e.g. for optimisation are often needed as well. Selection criteria formation and decision option generation are also important parts of the decision process, systematically used for example in multi-criteria decision analysis methodology. Statistical methodologies, distributions, object models, agents and fuzzy models are also introduced as parts of decision models. Simulation for tracking purposes and prediction should not be forgotten either. In system level applications several of these concepts are needed and utilised further. The decision types can be divided into long-term decisions and short-term decisions. The number of objectives, possible uncertainty, time dependence, etc. also affect to the decision problem perspective. Sometimes the decisions need to be made on-line and sometimes off-line. All these factors need to be kept in mind when the problem solving methodology is chosen. Comparing risk and cost is a common methodology in decision making. Choosing the preferences with competing priorities is an important point. Cumulative quality function [1] and chained paired comparisons [2] are examples of more specific methodologies used in decision making. Measures in decision making are also interesting. Deterministic decision making has its own measures mostly based on value and utility theory, while stochastic decision making uses statistical measures such as distributions. Decision concepts have been reviewed in reference [3]. Although decision making is applied in many areas, the literature seems to concentrate on economy and production planning. In used methodologies there exists more variation. For instance decision analysis approach and knowledge-based technologies are commonly used concepts. Decision making in handling sequences, resource allocation, network design, sorting problems and classification are examples of problem types studied in detail. These issues are also discussed in references [4], [5] and [6]. The author has dealt with other decision analysis case examples e.g. in references [7], [8] and [9]. In the conceptual decision model presented in this paper only carefully considered features have been used. Some of the earlier presented techniques have been selected into this model. This model utilises rule-based methodologies, numerical algorithms and procedures, statistical methodologies including e.g. distributions, and visual support. Rule-based methodologies are used for instance in preliminary elimination of decision options, algorithms and procedures e.g. in calculation of weight coefficients, and statistical distributions in evaluating the values in a kind of decision table. This utilisation is explained more in detail in the next chapter about the decision concept and model. The selection of the used decision concepts in the model is based on case-based needs. By analysing real decision problems the most suitable features and methodologies have been taken in use. If some feature is noticed to be unnecessary in the decision process, it has been left out from the final decision model. Also missing features have been completed on the way. The concept of the model is first planned on rough level, and then fine tuned during the examination of the case examples. The type of the decision problem determines the application. In a decision situation there often exist alternatives. Sometimes the alternatives appear in frequent sets. In each decision situation there is a certain history that can be more or less known.

128

Miki Sirola

Statistical support, production of solutions, filtering and selection are all needed in certain steps. Situation-based assessment is very important for instance in a case as if checkers. This game located on a draught-board may be analysed by using different area sizes or numbers of elements. The number of possible combinations increases very fast with the size. Other possible cases worth to mention are e.g. selection of career, buying decision (car is the classical example), or an optimisation of a going route. The selection of career may include such attributes as inclination, interest, economy, risk, etc. The buying decision has such possible attributes as price, technical qualities, colour, model, etc. In this paper the case choice of career is analysed more in detail. The computer application of this case example has been built with G2 expert system shell.

2

Decision Concept of the Model

The conceptual decision model is presented here first on a rough level, and then by going more and more into details when getting more into the examined case itself. It was found out that the rule-based methods, algorithms and procedures, statistical methods such as distributions, and visual support are the most suitable methodologies and give the most desired features for the decision model in question. The whole concept in use consists of these elements just mentioned. In the model a kind of decision table is built including decision options and decision attributes (see Figure 1). The decision attributes can also be called decision criteria. Each decision attribute has a weight coefficient, and each decision option can be valued in regard to each attribute. This far the table is quite similar that is used in quantitative analysis of multi-criteria decision analysis. In fact also the formula calculating the final numerical result is exactly the same: n

vtot ( ai ) = ∑ wjvj ( ai )

(1)

i =1

where ai is decision alternative i, wj is weight coefficient of criteria j and vj(ai) is a scaled score of the decision alternative i compared to criterion j. The decision problem shapes during the analysis process. First the decision attributes are defined for the case in question. Then the decision options are created. The decision options are filtered with a rule-based examination, and only the most suitable ones are selected for the final analysis. Similar procedure is possible to realise with the decision attributes as well. The weight coefficients are calculated with an algorithm based on pairwise comparisons and step by step adjustment through all attributes in the final analysis. This procedure is adjusted for each case separately. The use of statistical distributions comes into the figure when the values in the decision table are given. Some attributes with regard to their corresponding decision option are valued by using information in statistical tables. Historical data is one of the elements used in constructing these tables. This method and its realisation is explained more in detail in the next chapter about the computer application.

Using Conceptual Decision Model in a Case Study

129

The decision table helps also the decision maker to perceive the decision problem visually. It is of course possible to use many kinds of visualisation methodologies in addition, visualisations that help the decision maker to picture the decision problem better than just pure numbers in a decision table. But already the decision problem shaped into a decision table gives great help in understanding the decision problem better.

3

Computer Application

The computer application is realised in G2 expert system shell. The presented realisation scheme (see Figure 1) is for the case choice of career, but it can be easily generalised for any of the cases mentioned in this paper. Some key features of the realisation are also documented here. The decision table itself is realised as a freedom table. The changing values comes either from arrays such as choice of career (options), value of attribute (criteria) and weight of attribute (weight coefficients), or some of the functions explained later on (numerical values in the decision table). The result is calculated with formula (1) as it was mentioned in chapter 2. Also the array values are calculated as outputs of procedures or functions. An object class career has been defined, and four subclasses inclination, interest, economy and risk. The career candidates such as mathematician, natural scientist, linguist, lawyer, economist and trucker are defined as instances of the object class career. The rule base (not seen in the figure) takes care for instance of the filtering of the initial decision options. Minimum limit values are defined for the weight coefficients of the attributes, and based on these comparisons the final decision options are selected. As already mentioned there are procedures and functions calculating the weight coefficients and numerical values in the decision table. The procedures and functions calculating the numerical values in the decision table also utilises accessory tables including information from statistical distributions about the attributes in the decision table. The realisation scheme of the calculation of the weight coefficients and numerical table values is not described in every detail in this paper.

4

Case Example about the Choice of Career

In the case more than thirty persons choice of career is analysed. The selection process is more or less the same with each person, or at least so similar, that only one such example is discussed in detail. No statistical analysis is done for the whole random sample. The qualities of the decision model and the concept used are considered more important, and these features are best lighted through an illustrative example. The decision table for person number 17 is seen in Figure 1. For the person number 17 the decision option filtering gives out three options: mathematician, lawyer and trucker. So in the decision table of Figure 1 the career1 is mathematician, career2 is lawyer, and career3 is trucker. The decision attributes are

130

Miki Sirola

inclination, interest, economy and risk. Here inclination means the genetic feasibility of the person for such a career. Interest means the subjective willingness to choose the career. Economy means the statistical income of each career type, and risk the statistical possibility of getting unemployed in each profession.

Fig. 1. A window of the G2 application where the case choice of career is analysed for person number 17

As input for the procedure calculating weight coefficients are given such things as each person's subjective evaluation and a kind of general importance. Pairwise comparisons are done to each two attributes and the final values are calculated step by step. As already mentioned the attributes economy and risk are valued by using tables including information from statistical distributions, while the other attributes are valued by different means. The statistical distributions are taken from public databases. A subjective measure is used in valuing the attribute interest, while the attribute inclination is valued by combined technology including subjective, statistical and kind of quality measures. Person number 17 seems to have most inclination for the career of lawyer, and most interest for the career of mathematician. The statistics shows best income for the career of lawyer, and smallest risk for the career of trucker. Note that the attribute risk is in inverse scale, so a big number means a small risk (which is considered to be a good quality), while all other attributes are in normal scale (big number means big inclination, interest or economy). The scale of the values in the decision table is from 0 (worst possible) to 5 (best possible).

Using Conceptual Decision Model in a Case Study

131

The numerical result shows clearly that person number 17 should choose the option lawyer. The second best choice would be mathematician, and clearly last comes trucker. This result is so clear that sensitivity analysis is not needed with this person. In many cases the sensitivity analysis shows the week points of the analysis by making clear how different parameters affect to the final result. It must be noted though that the attribute risk was given very small weight. Still many people consider this attribute rather important in this context. In this case the inclination has been given rather high emphasis. The importance of attributes economy and interest are located somewhere between. This tool should be considered as some kind of advisory tool, and by no means as an absolute reference for the final choice. Although with the person number 17 two of the attributes clearly point to the choice of lawyer career, there are still two important attributes that disagree with this opinion. For instance many people think that one should follow the voice of interest in such things as choosing career. Still this kind of analysis can be very informative for the decision maker to find out the other motives for such choices that are often more hidden.

5

Discussion

Conceptual decision model has been built by first collecting the desired decision concept by combining in a new way existing decision support methodology. Then the decision model has been built by iterating the details while the computer application is being built and the case examples analysed. The computer application is programmed on G2 platform. The case example of choice of career is analysed more in detail. Also other case examples have been used in the development of the computer application, and also in the fine tuning of the concept and model itself. The developed prototype should be considered mostly as an advisory tool in decision making. The features have been selected with thorough care, and therefore the desired qualities are mostly found from the final application. Also the realisation puts up some restrictions in the possibilities of course, but in general I think that we can be rather satisfied with the results achieved. The decision model includes rule-based methodologies, numerical algorithms and procedures, statistical methodologies such as distributions, and visual support. The use of these methodologies in the model has been explained more in detail in previous chapters. The role of visual support is not very remarkable in this application, although it can be considered generally quite important and also an essential part of the concept. There exist many possibilities in making errors, for instance in calculating the weight coefficients and in producing the values in the decision table, but mostly the sensitivity analysis helps to find out the week points in the analysis. Plausible comparison of different attributes is also a problem. By giving the weight coefficients it is tried to put the attributes in a kind of order of importance, but to justify the comparison itself is often problematic. For instance comparison of risk and cost is not always correct by common opinion. The numerical result of the analysis is not the most important result achieved. The most important result is to find out what are the most important decisive factors in the

132

Miki Sirola

whole decision making process, and therefore to learn more about the problem itself. Already the better understanding of the problem helps in finding out a good solution, even if the decision maker does not agree with the numerical result given by the tool. The decisive factors and their order of importance are found from the decision attributes (criteria) of the formulated decision problem. Sometimes the revelation of hidden motives helps the decision maker. This often happens during the long procedure of playing with the decision problem, from the beginning of the problem formulation to the final analysis and even documentation. Better understanding by learning during the decision problem is one of the key issues that this paper is trying to present. Evaluation expertise is also very important to include in the solution procedure of the decision problem. Otherwise we are just playing with random numbers without a real connection to the decision itself that we are tying to solve. This paper is an example of using a theoretical methodology in solving a practical problem. The model constructed is a kind of completed multi-criteria decision analysis (MCDA) concept. The decision table itself is very similar, but the difference is in the way that different components and values of the table are produced. In ordinary MCDA method more handwork is done, while this system is rather far automated. The case about choice of career is a typical one time decision. The case of buying a new car is very similar in this regard. On the other hand the game checkers is completely different decision problem type. In that case rather similar decision situations come out repeatedly. The role of historical data, retrieval and prediction becomes more important. The case of optimising a going route is again different, third type of problem. Only the first one of the previous problems was analysed in this paper, although also other cases have been calculated with the tool. The others were just very shortly introduced here. Reporting large amount of analysis results would be impossible and not appropriate in a short paper, and also better focus on the chosen particular problem has been possible. With different type of decision problem the methodology used changes from many parts. Such case could be a topic of another paper. In this paper the used methodology has been reflected through a single application to be able to point out findings quite in detail. Although many persons were analysed in the choice of career case example experiment, no statistical analysis for the whole random sample was made. To concentrate more in this issue is one clear future need. As a tool capable of handling rather large amounts of data in a moderate time has been built, a natural way to proceed is to widen the scope into this direction. In the analysis of other persons choosing their career numerous variations in the results, and also in different phases of the analysis were found out. Because from the methodological point of view the procedure is similar, reporting this part of analysis has been ignored. The realisation platform for the computer application is G2 expert system shell. This environment is quite suitable for this kind of purpose. G2 is very strong in heuristics, and not so good in numerical calculation. Although both of them are needed in this application, the need of very heavy numerical calculation is not so essential. As a kind of combinatorial methodology has been developed, also other

Using Conceptual Decision Model in a Case Study

133

platforms could be considered. G2 was just a natural choice to make the first prototype to test the ideas presented in this paper. I owe an acknowledgement for the Laboratory of Automation Technology of our university for the possibility of using G2 expert system shell as a platform of the computer application, which made the analysis of the case examples possible.

References [1] [2] [3] [4] [5] [6] [7] [8]

[9]

Zopounidis C., M. Doumpos. Stock evaluation using preference disaggregation methodology. Decision sciences. Atlanta (1999) Ra J. Chainwise paired comparisons. Decision sciences. Atlanta (1999) Sirola M. Decision concepts. To be published. Ashmos D., Duchon D., McDaniel R. Participation in strategic decision making: the role of organizational predispositions and issue interpretation. Decision sciences. Atlanta (1998) Santhanam R., Elam J. A survey of knowledge-based systems research in decision sciences (1980-1995). The Journal of the Operational Research Society. Oxford (1998) Rummel J. An empirical investigation of costs in batching decisions. Decision sciences. Atlanta (2000) Sirola M. Computerized decision support systems in failure and maintenance management of safety critical processes. VTT Publications 397. Espoo, Finland (1999) Laakso K., Sirola M., Holmberg J. Decision modelling for maintenance and safety. International Journal of Condition Monitoring and Diagnostic Engineering Management. Birmingham (1999) Vol. 2, No. 3, ISSN 1363 – 7681, pp. 13-17 Sirola M. Applying decision analysis method in process control problem in accident management situation. Proceedings of XIV International Conference on System Science. Wroclaw, Poland (2001)

Automated Knowledge Acquisition Based on Unsupervised Neural Network and Expert System Paradigms Nazar Elfadil and Dino Isa Division of Engineering, University of Nottingham in Malaysia No. 2 Jln Conlay, 50450 Kuala Lumpur, Malaysia Tel: 60321408137, Fax: 603-27159979 [email protected]

Abstract. This is paper presents an approach for automated knowledge acquisition system using Kohonen self-organizing maps and k-means clustering. For the sake of illustrating the system overall architecture and validate it, a data set represent world animal has been used as training data set. The verification of the produced knowledge based had done by using conventional expert system.

1

Problem Background

In our daily life, we cannot avoid making decisions. Decision-making may be defined as making a conclusion or determination upon a problem at hand. However, in recent years, problems to be solved have become more complex. Consequently, knowledge based decision-making systems have been developed to aid us in solving complex problems. Nevertheless, the knowledge base itself has become the bottleneck, as it is the part of the system that is still developed manually. Nonetheless, as the size and complexity of the problems increase, and experts become scarce, the manual extraction of knowledge becomes very difficult. Hence, it is imperative that the task of knowledge acquisition be automated. The demand for automated knowledge acquisition system has increased dramatically. The previous approaches for automated knowledge acquisition are based on decision trees, progressive rule generation, and supervised neural networks [1]. All the above-mentioned approaches are supervised learning methods, requiring training examples combined with their target output values. In the real world cases, target data is not always known so as to provide to the system before start training the data set [2]. This paper is organized as follows: Section II presents an overview of the automated knowledge acquisition. Section III demonstrates illustrative application. Finally section IV illustrates the conclusion and future work.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 134-140, 2003.  Springer-Verlag Berlin Heidelberg 2003

Automated Knowledge Acquisition Based on Unsupervised Neural Network

2

135

Automated Knowledge Acquisition

The paper proposes an automated knowledge acquisition method in which, knowledge (connectionist) is extracted from data that have been classified by a Kohonen self-organizing map (KSOM) neural network. This knowledge (at this stage) is of the intermediate-level concept rule hierarchy. The final concept rule hierarchy is generated by applying a rule generation algorithm that is aided by an expert system inference engine. The resulting knowledge (symbolic) may be used in the construction of the symbolic knowledge base of an expert system. The proposal is rationalized from the realization that most complex real-world problems are not solvable using just symbolic or just adaptive processing alone. However, not all problems are suitable for neural expert system integration [3]. The most suitable ones are those that seek to classify inputs into a small number of groups. Figure. 1 illustrates the top-level architecture that integrates neural network and expert system.

Fig. 1. Neural Expert system architecture

3

Illustrative Application

To illustrate the details of the various tasks in the proposed method, a simple and illustrative case study will be use. The system has been trained with animals data set illustrated in Table 1. Each pattern consists of 11 attributes and the whole animal data set with various attributes represented 11 kinds of animals, as shown in Table 1. The trained data is composed of 1023 examples.

136

Nazar Elfadil and Dino Isa Table 1. Portion of animal data set input data

Attributes Is Big

2 legs

4 legs

Hair

Feathers

Hunt

Run

Fly

Swim

Likes to

Medium

Dove Duck Goose Hawk Cat Eagle Fox Dog Wolf Lion Horse

Has

Small

Patterns

1 1 1 1 1 0 0 0 0 0 0

0 0 0 0 0 1 1 1 1 0 0

0 0 0 0 0 0 0 0 0 1 1

1 1 1 1 0 1 0 0 0 0 0

0 0 0 0 1 0 1 1 1 1 1

0 0 0 0 1 0 1 1 1 1 1

1 1 1 1 0 1 0 0 0 0 0

0 0 0 1 1 1 1 0 1 1 0

0 0 0 0 0 0 0 1 1 1 1

1 0 1 1 0 1 0 0 0 0 0

0 1 1 0 0 0 0 0 0 0 1

Step 1: initialize weight to small random values and set the initial neighborhood to be large. Step 2: stimulate the net with a given input vector. Step 3: calculate the Euclidean distance between the input and each output node and select the output with the minimum distance. D ( j) = ( w − x ) 2 D(j) is a minimum

∑

ij

i

i

Step 4: update weights for the selected node and the nodes within its neighborhood. wij (new) = wij (old ) + α ( xi − wij (old )) Step 5: repeat from step 2 unless stopping condition. Fig. 2. KSOM learning algorithm

The data-preprocessing phase consist of normalization and initialization of input data. In normalization, the objective is to ensure that there is no data that dominates over the rest of input data vectors [4]. The initialization task involves three tasks namely: weight initialization, topology initialization, and neighborhood initialization. The hexagonal lattice type is chosen as the map topology in animal data set case study. The choice of number of output nodes is done through comprehensive trials [5]. The weights of the neural network is initialized either by linear or random initialization. The random initialization technique is chosen here. For the neighborhood function, Gaussian or Bubble are the suitable choices [3]. The bubble function is considered the simpler but adequate one, and it is applied here.

Automated Knowledge Acquisition Based on Unsupervised Neural Network

137

Table 2. Portion of KSOM output

Output Nodes (Row, Column) (1,1) (1,2) (1,3) (1,4) (1,5)

Input nodes Q1

Q2

0 0 0 0 0

1.0 1.0 1.0 1.0 1.0

Q3 Q4 0 0 0 0 0

0 0 0 0 0

Q5

Q6

Q7

Q8

Q9

Q10

Q11

1.0 1.0 1.0 1.0 1.0

1.0 1.0 1.0 1.0 1.0

0 0 0 0 0

1.0 0.9 0.9 0.5 0.1

0 0 0 0 0

1.0 0.9 0.9 0.5 0.1

1.0 1.0 1.0 1.0 0.9

The machine learning and clustering phase composed of KSOM learning and Kmeans clustering algorithms. The proposed method employs unsupervised learning, and this is the key contribution of this research. Figure 2 outlines the KSOM learning algorithm. Table 2 illustrates a portion of the result of the KSOM training session. While Figure 3 and Figure 4 demonstrates the visual representation of KSOM output and Kmeans output, respectively. The selection of neighborhood range and learning rate techniques well defined in [5]. The next step is classifying the updated weights explicitly, and the process is referred as clustering. The terminology comes from the appearance of an incoming sequence of feature vectors which arrange themselves into clusters, groups of points that are closer to each other and their own centers rather than to other groups. When a feature vector is input to the system, its distance to the existing cluster representatives is determined, and it is either assigned to the cluster, with minimal distance or taken to be a representative if a new cluster. The required clustering process carried out by the modified K-means algorithm. The K-means algorithm self-organizes its input data to create clusters [5]. Figure 4 shows a visual representation for clustering session output. Each gray shade represents a certain cluster. The step now is to find the codebook vector and the indices for each cluster. This data contains the weights that distinguish and characterize each cluster. In this stage of the knowledge acquisition process, the extraction of a set of symbolic rules that map the input nodes into output nodes (with respect to each cluster) is performed. The antecedents of the rules that define these concepts consist of contributory and inhibitory input weights. In a KSOM network, each output node is connected to every input node, with the strength of interconnection reflected in the associated weight vector. The larger the weight vector associated with a link, the greater is the contribution of the corresponding input node to the output node. The input with the largest weight link makes the largest contribution to the output node [4]. To distinguish the contributory inputs from inhibitory inputs, then binarize the weights. If contributive, the real-valued weight is converted to 1, and is converted to 0 if inhibitory. There are two approaches to do this, namely: threshold and breakpoint technique [4]. The threshold technique has chosen. The threshold is set at 50% (i.e. below 0.5 is considered as 0 and above 0.5

138

Nazar Elfadil and Dino Isa

considered as 1). Such threshold selection made the probability of false positive and true negative are equal. More information concerning this selection found in [5].

Fig. 3. Output of Kohonen NN learning

Fig. 4. Output of modified Kmeans clustering

The final set of antecedents in each cluster usually contains some duplicated patterns. This redundancy is now removed. Now can map symbolically the antecedents to each cluster and obtain the rules for each cluster. The symbolic rule extraction algorithm is an inductive learning procedure. The algorithm as is provided in Figure 5 is self-explanatory.

Automated Knowledge Acquisition Based on Unsupervised Neural Network

139

Fig. 5. Symbolic rule extraction algorithm

The system provides rules that recognized the various types of animal's types. Given an input pattern of the animal data, the clusters can now recognized using this rule base. For making the rule antecedents more comprehensible we removed the inhibitory parts, as shown in Table 3. For the sake of testing and evaluating the final produced symbolic rules an expert system has developed using C language, and evaluated the rules using the expert system inference engine. Table 3. Elaborated extracted symbolic rule base

Rule No. 1 2 3 4 5 6 7 8 9 10 11

4

Antecedents parts

Conclusion

(Big)&(4_legs)& (hair)& (hunt)&(run) (Big)&(4_legs)& (hair)&(run)&(swim) (Small)&(2_legs)&(feathers)&(swim) (Small)& (2_legs)& (feathers)& (fly)& (swim) (Small)&(4_legs)& (hair)& (hunt) (Small)&(2_legs)& (feathers)& (fly) (Medium)& (4_legs)& (hair)& (run) (Small)&(2_legs)& (feathers)& (hunt)& (fly) (Medium)& (4_legs)& (hair)& (hunt) (Medium)&(4_legs)& (hair)& (hunt)& (run) (Medium)&(2_legs)&(feathers)& (hunt)& (fly)

Lion Horse Duck Goose Cat Dove Dog Hawk Fox Wolf Eagle

Conclusion

The animal data set case study shows that the proposed automated knowledge acquisition method can successfully extract knowledge in the form of production rules from numerical data set representing the salient features of the problem domain. This study has demonstrated that symbolic knowledge extraction can be performed using unsupervised learning KSOM neural networks, where no target output vectors are available during training. The system is able to learn from examples via the neural network section. The extracted knowledge can form the knowledge base of an expert system, from which explanations may be provided. Large, noisy and incomplete data

140

Nazar Elfadil and Dino Isa

set can be handled. The system proves the case of the viability of integrating neural network and expert system to solve real-world problems.

References [1]

[2] [3] [4]

[5]

T. S. Dillon, S. Sestito, M. Witten, M. Suing. “ Automated knowledge acquisition Using Unsupervised Learning”. Proceeding of the second IEEE workshop on Emerging Technology and Factory Automation (EFTA'93). Cairns, Sept. 1993, pp. 119-128. M. S. Kurzyn, “ Expert Systems and Neural Networks: A comparison”, IEEE Expert, pp. 222-223, 1993. Sabrina Sestito, Automated knowledge acquisition, Prentice Hall, 1994, Australia. N. Elfadil, M. Khalil, S. M. Nor, and S. Hussein. “Kohonen self-organizing maps & expert system for disk network performance prediction”. Journal of Systems Analysis Modeling & Simulation (SAMS), 2002, Vol. 42, pp. 10251043. England. N. Elfadil, M. Khalil, S. M. Nor, and S. Hussein. “Machine Learning: The Automation of Knowledge Acquisition Using Kohonen Self-Organizing Maps Neural Networks”. Proceeding of Malaysian Journal of Computer Science, Malaysia, June 2001: Vol. 14, No. 1, pp. 68-82.

Selective-Learning-Rate Approach for Stock Market Prediction by Simple Recurrent Neural Networks Kazuhiro Kohara Chiba Institute of Technology 2-17-1, Tsudanuma, Narashino, Chiba 275-0016, Japan [email protected]

Abstract. We have investigated selective learning techniques for improving the ability of back-propagation neural networks to predict large changes. The prediction of daily stock prices was taken as an example of a noisy real-world problem. We previously proposed the selective-presentation and selective-learning-rate approaches and applied them into feed-forward neural networks. This paper applies the selective-learning-rate approach into three types of simple recurrent neural networks. We evaluated their performances through experimental stock-price prediction. Using selective-learning-rate approach, the network can learn the large changes well and profit per trade was improved in all of simple recurrent neural networks.

1

Introduction

Prediction using back-propagation neural networks [1] has been extensively investigated (e.g., [2-5]), and various attempts have been made to apply neural networks to financial market prediction (e.g., [6-18]), electricity load forecasting (e.g., [19, 20]) and other areas (e.g., flour price prediction [21]). In the usual approach, all training data are equally presented to a neural network (i.e., presented in each cycle) and the learning rates are equal for all the training data independently of the size of the changes in the prediction-target time series. Generally, the ability to predict large changes is more important than the ability to predict small changes, as we mentioned in the previous paper [16]. When all training data are presented equally with an equal learning rate, the neural network will learn the small and large changes equally well, so it cannot learn the large changes more effectively. We have investigated selective learning techniques for improving the ability of a neural network to predict large changes. We previously proposed the selectivepresentation and selective-learning-rate approaches and applied them into feedforward neural networks [16, 17]. In the selective-presentation approach, the training data corresponding to large changes in the prediction-target time series are presented more often. In the selective-learning-rate approach, the learning rate for training data corresponding to small changes is reduced. This paper applies the selective-learningV. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 141-147, 2003.  Springer-Verlag Berlin Heidelberg 2003

142

Kazuhiro Kohara

rate approach into three types of simple recurrent neural networks. We evaluate their performances, using the prediction of daily stock prices as a noisy real-world problem.

2

Selective-Learning-Rate Approach

To allow neural networks to learn about large changes in prediction-target time series more effectively, we separate the training data into large-change data (L-data) and small-change data (S-data). L-data (S-data) have next-day changes that are larger (smaller) than a preset value. In the selective-learning-rate approach [17], all training data are presented in every cycle; however, the learning rate of the back-propagation training algorithm for S-data is reduced compared with that for L-data. The outline of the approach is as follows. Selective-Learning-Rate Approach: 1. 2. 3.

3

Separate the training data into L-data and S-data. Train back-propagation networks with a lower learning rate for the S-data than for the L-data. Stop network learning at the point satisfying a certain stopping criterion.

Simple Recurrent Neural Prediction Model

We considered the following types of knowledge for predicting Tokyo stock prices. These types of knowledge involve numerical economic indicators. 1. 2. 3.

If interest rates decrease, stock prices tend to increase, and vice versa. If the dollar-to-yen exchange rate decreases, stock prices tend to decrease, and vice versa. If the price of crude oil increases, stock prices tend to decrease, and vice versa.

We used the following five indicators as inputs to the neural network in the same way as in our previous work [16, 17]. • • • • •

TOPIX: the chief Tokyo stock exchange price index EXCHANGE: the dollar-to-yen exchange rate (yen/dollar) INTEREST: an interest rate (3-month CD, new issue, offered rates) (%) OIL: the price of crude oil (dollars/barrel) NY: New York Dow-Jones average of the closing prices of 30 industrial stocks (dollars)

TOPIX was the prediction target. EXCHANGE, INTEREST and OIL were chosen based on the knowledge of numerical economic indicators. The Dow-Jones average was used because Tokyo stock market prices are often influenced by New York exchange prices. The information for the five indicators was obtained from the Nihon Keizai Shinbun (a Japanese financial newspaper).

Selective-Learning-Rate Approach for Stock Market Prediction

143

The daily changes in these five indicators (e.g. ∆ TOPIX(t) = TOPIX(t) TOPIX(t-1)) were input into neural networks, and the next-day's change in TOPIX was presented to the neural network as the desired output. The back-propagation algorithm [1] was used to train the network. All the data of the daily changes were scaled to the interval [0.1, 0.9]. We considered three types of simple recurrent neural networks (SRN). The structures of three SRN are shown in Figures 1, 2 and 3. The SRN-1 is similar to the recurrent network proposed by Elman [22], where inputs C(t) to the context layer at time t are the outputs of the hidden layer at time t-1, H(t-1): C(t) = H(t-1). In the SRN-1 training, C (t) was initialized to be 0 when t = 0. The structure of the SRN-1 was 25-20-1 (the 25 includes 5 in the input layer and 20 in the context layer).

Fig.1. SRN-1

The SRN-2 is similar to the recurrent network proposed by Jordan [23], where input C(t) to the context layer at time t is the output of the output layer at time t-1, O(t-1): C(t) = O(t-1). In the SRN-2 training, C (t) was initialized to be 0 when t = 0. The structure of the SRN-2 was 6-6-1 (the 6 includes 5 in the input layer and 1 in the context layer). The SRN-3 is our original network. In the SRN-3, input C(t) to the context layer at time t is the prediction error at time t-1: C(t) = ∆ TOPIX(t-1) - O(t-1). In the SRN-3 training, C (t) was initialized to be 0 when t = 0. The structure of the SRN-3 was also 6-6-1 (the 6 includes 5 in the input layer and 1 in the context layer).

144

Kazuhiro Kohara

Fig. 2. SRN-2

Fig. 3. SRN-3

4

Evaluation through Experimental Stock-Price Prediction

4.1

Experiments

We used data from a total of 409 days (from August 1, 1989 to March 31, 1991): 300 days for training, 109 days for making predictions.

Selective-Learning-Rate Approach for Stock Market Prediction

145

In Experiments 1, 3 and 5, all training data were presented to the SRN-1, SRN-2, and SRN-3 in each cycle with an equal learning rate ( ε = 0.7), respectively. In Experiments 2, 4 and 6, the learning rate for the S-data was reduced up to 20% (i.e., ε = 0.7 for the L-data and ε = 0.14 for the S-data) in the SRN-1, SRN-2, and SRN-3 training, respectively. Here, the large-change threshold was 14.78 points (about US$ 1.40), which was the median (or the 50% point) of absolute value of TOPIX daily changes in the training data. In each experiment, network learning was stopped at 3000 learning cycles. The momentum parameter α was 0.7. All the weights and biases in the neural network were initialized randomly between -0.3 and 0.3. When a large change in TOPIX was predicted, we tried to calculate “Profit” as follows: when the predicted direction was the same as the actual direction, the daily change in TOPIX was earned, and when it was different, the daily change in TOPIX was lost. This calculation of profit corresponds to the following experimental TOPIX trading system. A buy (sell) order is issued when the predicted next-day's up (down) in TOPIX is larger than a preset value which corresponds to a large change. When a buy (sell) order is issued, the system buys (sells) TOPIX shares at the current price and subsequently sells (buys) them back at the next-day price. Transaction costs on the trades were ignored in calculating the profit. The more accurately a large change is predicted, the larger the profit is. 4.2

Results

In each experiment the neural network was run ten times for the same training data with different initial weights and the average was taken. The first experimental results are shown in Table 1, where a change larger than 14.78 points (the 50% point) in TOPIX was predicted (“prediction threshold” equals to 14.78), we tried to calculate “Profit” according to the above trading method. When the prediction threshold was comparatively low, number of trades was too large. In the actual stock trading, the larger number of trades became, the higher transaction costs became. Table 1. Experimental results 1 (prediction threshold = 14.78)

Ex. 1 Profit Number of trades Profit per trade

Ex. 2 SRN-1 equal selective 466 768 24.2 45.0 19.2 17.0

Ex. 3

Ex. 4 SRN-2 equal selective 486 781 24.4 44.4 19.9 17.5

Ex. 5

Ex. 6 SRN-3 equal selective 367 760 17.4 36.7 21.1 20.7

The second experimental results are shown in Table 2, where a change larger than 31.04 points (the 75% point) in TOPIX was predicted, we tried to calculate “Profit.” When the prediction threshold became high, number of trades became small. In equallearning-rate approach (Experiments 1, 3 and 5), the networks cannot learn the large changes well and the output value of the networks cannot reach high level. So, number of trades was very small. In our selective-learning-rate approach

146

Kazuhiro Kohara

(Experiments 2, 4 and 6), the networks can learn the large changes well and number of trades became larger than that of equal-learning-rate approach. Using selectivelearning-rate approach, number of trades became large and profit per trade was improved in all of three types of simple recurrent neural networks. The SRN-3 achieved the best profit per trade in these experiments. Table 2. Experimental results 2 (prediction threshold = 31.04)

Ex. 1

Profit Number of trades Profit per trade

5

Ex. 2 SRN-1 equal selective 27 256 1.4 10.5 19.2 24.3

Ex. 3

Ex. 4 SRN-2 equal selective 8 180 0.5 6.9 16.0 26.0

Ex. 5 Ex. 6 SRN-3 equal selective 20 161 0.8 5.2 25.0 31.0

Conclusion

We investigated selective learning techniques for stock market forecasting by neural networks. We applied our selective-learning-rate approach into three types of simple recurrent neural networks. The learning rate for training data corresponding to small changes is reduced. The results of several experiments on stock-price prediction showed that using selective-learning-rate approach, the network can learn the large changes well and profit per trade was improved in all simple recurrent neural networks. Next, we will apply these techniques to other real-world forecasting problems. We also plan to develop a forecasting method that integrates statistical analysis with neural networks.

References [1] [2] [3] [4] [5] [6] [7]

Rumelhart D, Hinton G, Williams, R. Learning internal representations by error propagation. In: Rumelhart D, McClelland, J and the PDP Research Group (eds.). Parallel Distributed Processing 1986; 1. MIT Press, Cambridge, MA Weigend A, Gershenfeld N. (eds). Time Series Prediction: Forecasting the Future and Understanding the Past. Addison-Wesley, Reading, MA, 1993 Vemuri V, Rogers R (eds). Artificial Neural Networks: Forecasting Time Series. IEEE Press, Los Alamitos, CA, 1994 Pham D, Liu X. Neural Networks for Identification, Prediction and Control. Springer, 1995 Kil D, Shin F. Pattern Recognition and Prediction with Applications to Signal Characterization. American Institute of Physics Press, 1996 Azoff E. Neural Network Time Series Forecasting of Financial Markets. John Wiley and Sons, West Sussex, 1994 Refenes A, Azema-Barac M. Neural network applications in financial asset management. Neural Comput & Applic 1994; 2(1): 13-39

Selective-Learning-Rate Approach for Stock Market Prediction

[8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23]

147

Goonatilake S, Treleaven P. (eds). Intelligent Systems for Finance and Business. John Wiley and Sons, 1995 White H. Economic prediction using neural networks: the case of IBM daily stock return. In: Proc of Int Conf Neural Networks 1988; II-451-II-458. San Diego, CA Baba N, Kozaki M. An intelligent forecasting system of stock price using neural networks. In: Proc of Int Conf Neural Networks 1992; I-371-I-377. Singapore Dutta S, Shekhar S. Bond rating: a non-conservative application of neural networks. In: Proc of Int Conf Neural Networks 1988; II-443-II-450. San Diego, CA Freisleben B. Stock market prediction with backpropagation networks. In: F Belli and J Rademacher (eds). Lecture Notes in Computer Science 604, pp. 451-460, Springer -Verlag, Heidelberg, 1992 Kamijo K, Tanigawa T. Stock price pattern recognition – a recurrent neural network approach. In: Proc of Int Conf Neural Networks 1990; I-215-I-221. San Diego, CA Kimoto T, Asakawa K, Yoda M, Takeoka M. Stock market prediction with modular neural networks. In: Proc of Int Conf Neural Networks 1990; I-1-I-6. San Diego, CA Tang Z, Almeida C, Fishwick P. Time series forecasting using neural networks vs. Box-Jenkins methodology. Simulation 1991; 57(5): 303-310 Kohara K, Fukuhara Y, Nakamura Y. Selective presentation learning for neural network forecasting of stock markets. Neural Comput & Applic 1996; 4(3): 143-148 Kohara K, Fukuhara Y, Nakamura Y. Selectively intensive learning to improve large-change prediction by neural networks. In: Proc of Int Conf Engineering Applications of Neural Networks 1996; 463-466. London Kohara K. Neural networks for economic forecasting problems. In: Cornelius T. Leondes (ed). Expert Systems -The Technology of Knowledge Management and Decision Making for the 21st Century- 2002; Academic Press Park D, El-Sharkawi M, Marks II R, Atlas L, Damborg M. Electric load forecasting using an artificial neural network. IEEE Trans Power Syst 1991; 6(2): 442-449 Caire P, Hatabian G, Muller C. Progress in forecasting by neural networks. In: Proc of Int Conf Neural Networks 1992; II-540-II-545. Baltimore, MD Chakraborty K, Mehrotra K, Mohan C, Ranka S. Forecasting the behavior of multivariate time series using neural networks. Neural Networks 1992; 5: 961970 Elman J. L. Finding structure in time, CRL Tech. Rep. 8801, Center for Research in Language, Univ. of California, San Diego, 1988 Jordan M. I. Attracter dynamics and parallelism in a connectionist sequential machine. In: Proc 8th Annual Conf Cognitive Science Society 1986; 531-546. Erlbaum

A Neural-Network Technique for Recognition of Filaments in Solar Images V.V. Zharkova1 and V. Schetinin2 1

Department of Cybernetics, University of Bradford, BD7 1DP, UK [email protected] 2 Department of Computer Science, University of Exeter, EX4 4QF,UK [email protected]

Abstract. We describe a new neural-network technique developed for an automated recognition of solar filaments visible in the hydrogen Halpha line full disk spectroheliograms. This technique deploys the artificial neural network (ANN) with one input and one output neurons and the two hidden neurons associated either with the filament or with background pixels in this fragment. The ANN learns to recognize the filament depicted on a local background from a single image fragment labelled manually. The trained neural network has properly recognized filaments in the testing image fragments depicted on backgrounds with various brightness caused by the atmospherics distortions. Using a parabolic activation function this technique was extended for recognition of multiple solar filaments occasionally appearing in selected fragments.

1

Introduction

Solar images observed from the ground and space-based observatories in various wavelengths were digitised and stored in different catalogues, which are to be unified under the grid technology. The robust techniques including limb fitting, removal of geometrical distortion, centre position and size standardisation and intensity normalisation were developed to put the Hα and Ca K lines full disk images taken at the Meudon Observatory (France) into a standardised form [1]. There is a growing interest to widespread ground-based daily observations of a full solar disk in the Hydrogen Hα-line, which can provide important information on the long-term solar activity variations during months or years. The project European Grid of Solar Observations [2] was designed to deal with the automated detection of various features associated with solar activity, such as: sunspots, active regions and filaments, or solar prominences. Filaments are the projections on a solar disk of prominences seen as very bright and large-scale features on the solar limb [1]. Their location and shape does not change very much for a long time and, hence, their lifetime is likely to be much longer then one solar rotation. However, there are visible changes seen in the filament V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 148-154, 2003.  Springer-Verlag Berlin Heidelberg 2003

A Neural-Network Technique for Recognition of Filaments in Solar Images

149

elongation, position with respect to an active region and magnetic field configuration. For this reason the automated detection of solar filaments is a very important task to tackle in sense of understanding the physics of prominence formation, support and disruptions. Quite a few techniques were explored for a different level of the feature detection such as: the rough detection with a mosaic threshold technique [3], the image segmentation and region growing techniques [4] – [7]. Artificial Neural Networks (ANNs) [8], [9] applied to the filament recognition problem commonly require a representative set of image data available for training. The training data have to represent image fragments, depicting filaments on different conditions, under which the ANN has to solve the recognition problem. For this reason the number of the training examples must be large. However the image fragments are still taken from the solar catalogues manually. In this paper we describe a new neural-network technique which is able to learn recognizing filaments from a few image fragments labelled manually. This technique deploys the artificial neural network (ANN) with one input and one output neurons and the two hidden neurons associated either with the filament or background pixels in this fragment. Despite the difference in backgrounds in the selected fragments of solar images containing filaments the trained network has properly recognized all the filaments in these fragments. Using a parabolic activation function, this technique was extended to recognize multiple solar filaments occasionally appearing in some fragments.

2

A Recognition Problem

First, let us introduce the image data as a n×m matrix X = {xij}, i = 1, …, n, j = 1, …, m, consisting of pixels whose brightness ranges between 1 and 255. This matrix depicts filament which is no more than a dark elongated feature observable on the solar surface with a high background brightness. Then, a given pixel xij ∈ X may belong to a filament region, class Ω0, or to a non-filament region, class Ω1. Note that the brightness of non-filament region varies hardly over the solar surface. Dealing with images we can make a realistic assumption that the influence of neighbouring pixels on the central pixel xij is restricted to k elements. Using this assumption we can easily define a triangular window, k×k matrix P(i,j), with central pixel xij and k nearest neighbours. The background of the filament elements is assumed to be additive to xij that allows us to evaluate and subtract it from the brightness values of all the elements of matrix P. Now we can define a background function u = ϕ(X; i, j), which reflects a total contribution of background elements to the pixel xij. Parameters of this function can be estimated from image data X. In order to learn background function ϕ and then decide whether a pixel x is a filament element or not, we can use image fragments whose pixels are manually labelled and assigned either to class Ω0 or class Ω1. A natural way to do this is to use a neural network which is able to learn recognising filaments from a few number of the labelled image fragments.

150

V.V. Zharkova and V. Schetinin

3

The Neural-Network Technique for Filament Recognition

As we stated in the Introduction, this technique deploys the artificial neural network with one input and one output neurons and the two hidden neurons associated either with the filament or with background pixels in this fragment. In general, neural networks perform a threshold technique of pattern recognition using one hidden neuron. However, in the case of solar images it turned out that background, on which darker filaments occur, vary from image to image. In order to make filament detection independent on the seeing conditions this technique has to take into account a variability of background elements in images. Hence, the idea behind the proposed method is to use the additional information on a contribution of the variable background elements which is represented by the function u = ϕ(X; i, j). This function, as we assume, can learn from image data. One of the possible ways to estimate the function ϕ is to approximate its values for each pixel xij of a given image X. Regarding to the filament recognition; we can use either a parabolic or linear approximation of this function. Whilst the first type is suitable for small size image fragments, the second one is used for relative large fragments of the solar surface. Below we describe our neural-network technique exploiting the last type of approximation. For image processing, our algorithm exploits a standard sliding technique for which a given image matrix X is transformed into a column matrix Z consisting of q = (n – k + 1)(m – k + 1) columns z(1), …, z(q)). Each column z presents the r pixels taken from the matrix P, where r = k2. The pixels x11, x12, …, x1k, …, xk1, xk2, …, xkk of this matrix are placed in the columns of the matrix Z so that the central element of P is located in the (r + 1)/2 position of the column z. Let us now introduce a feed-forward ANN consisting of the two hidden and one output neurons as depicted in Fig 1. The first hidden neuron is fed by r elements of column vector z(j). The second hidden neuron evaluates the value uj of a background for the jth vector z(j). The output neuron makes a decision, yi = {0,1}, on the central pixel in the column vector z(j). (j)

f2

uj

(j)

z1 z2(j)

w0(2)

f3

… zr(j)

f1

sj

yj = (0, 1)

w0(3)

w0(1)

Fig. 1. The feed-forward neural network consisting of the two hidden and one output neurons

Assuming that the first hidden neuron is fed by r elements of the column vector z, its output s is calculated as follows: sj = f1(w0(1), w(1); z(j)), j = 1, …, q,

(1)

A Neural-Network Technique for Recognition of Filaments in Solar Images

151

where w0(1), w(1), and f1 are the bias term, weight vector and activation function of the neuron, respectively. The activity of the second hidden neuron is proportional to the brightness of a background and can be described by the formula: uj = f2(w0(2), w(2); j), j = 1, …, q.

(2)

(2)

(2)

The bias term w0 and weight vector w of this neuron are updated so that the output u becomes an estimation of a background component contributed to the pixels of the jth column z(j). Parameters w0(2) and w (2) may learn from the image data Z. Taking into account the outputs of the hidden neurons, the output neuron makes a final decision, yj ∈ (0, 1), for each column vector z(j) as follows: yj = f3(w0(3), w(3); sj, uj), j = 1, …, q.

(3)

Depending on activities of the hidden neurons, the output neuron assigns a central pixel of the column z(j) either to the class Ω0 or Ω1.

4

A Training Algorithm

In order to train our feed-forward ANN plotted in Fig 1, we can use the backpropagation algorithms, which provide a global solution. These algorithms require recalculating the output sj for all q columns of matrix Z and for each the training epochs. However, there are some local solutions in which the hidden and output neurons are trained separately. Providing an acceptable accuracy, the local solutions can be found much more easer than the global ones. First, we need to fit the weight vector of the second hidden, or a “background” neuron that evaluates a contribution of the background elements. Fig 2 depict an example where the top left plot shows the image matrix X presenting a filament on the unknown background and the top right plot reveals the corresponding filament elements depicted in black. The two bottom plots in Fig 2 show the outputs s (the left plot) and the weighted sum of s and u (the right one) plotted versus the columns of matrix Z. From the left top plot we see that the brightness of a background varies from the lowest level at the left bottom corner to the highest at the right top corner. Such variations of the background increase the output value u calculated over the q columns of matrix Z, see the increasing curve depicted at the bottom left plot. This plot shows us that the background component is changed over j = 1, …, q and can be fitted by a parabola.. Based on this finding we can define a parabolic activation function of the “background” neuron as: uj = f2 (w0(2), w(2); j) = w0(2) + w1(2)j + w2(2)j2.

(4)

152

V.V. Zharkova and V. Schetinin

1(a)

1(b)

2000

4

1500

2

1000

0

500

-2

0

0

5000 Y

10000

-4

0

5000 F

10000

Fig. 2. The example of image matrix X depicting the filament on the unknown background

The weight coefficients w0(2), w1(2), and w2(2) of this neuron can be fitted to the image data Z so that a squared error e between the outputs ui and si became minimal: e = Σj(uj – sj)2 = Σi(w0(2) + w1(2)j + w2(2)j2 – sj)2 → min, j = 1,… , q.

(5)

The desirable weight coefficients can be found with the least square method. Using a recursive learning algorithm [10], we can improve the evaluations of these coefficients due to robustness to non-Gauss noise in image data. So the “background” neuron can be trained to evaluate the background component u. In the right bottom plot of Fig 2 we present the normalized values of si, which are no longer affected by the background component. The recognized filament elements are shown at the right top plot in the Fig 2. By comparing the left and right top plots in Fig 2 we can reveal that the second hidden neuron has successfully learnt to evaluate a background component from the given image data Z. Before training the output neuron, the weights of the first hidden neuron are to be found. For this neuron a local solution is achieved for a set of the coefficients to be equal to 1. In case, if it would be necessarily to improve the recognition accuracy, one can update these weights by using the back-propagation algorithm. After defining the weights for both hidden neurons, it is possible to train the output neuron which makes the decisions between 0 or 1. Let us re-write the output yi of this neuron as follows: yj = 0, if w1sj + w2uj < w0, and yj = 1, if w1sj + w2uj ≥ w0.

(6)

Then the weight coefficients w0, w1, and w2 can be fit in such way that the recognition error e is minimal: e = Σi| yi – ti | → min, i = 1,… , h,

(7)

A Neural-Network Technique for Recognition of Filaments in Solar Images

153

where | ⋅ | means a modulus operator, tj ∈ (0, 1) is the ith element of a target vector t and h is the number of its components, namely, the training examples. In order to minimize the error e one can apply any supervised learning methods, for example, the perceptron learning rule [8], [9].

5

Results and Discussion

The neural network technique with the two hidden neurons and one input and one output neurons described above was applied for recognition of dark filaments in solar images. The full disk solar images obtained on the Meudon Observatory (France) during the period of March-April 2002 were considered for the identification [1]. The fragments with filaments were picked from the Meudon full disk images taken for various dates and regions on the solar disk with different brightness and inhomogeneity in the solar atmosphere. There were 55 fragments selected depicting the filaments on a various background, one of them was used for training the ANN, the remaining 54 ones were used for testing the trained ANN. Visually comparing the resultant and origin images we can conclude that our algorithm recognised these testing filaments well. Using a parabolic approximation we can now recognize the large image fragments which may contain multiple filaments as depicted at the left plot in Fig 3.

1(a)

1(b)

Fig. 3. The recognition of the multiple filaments

The recognized filament elements depicted here in black are shown at the right plot. A visual comparison of the resulting and original images confirms that the proposed algorithm has recognized all four filaments being close to their location and shape.

154

V.V. Zharkova and V. Schetinin

6

Conclusions

The automated recognition of filaments on the solar disk images is still a difficult problem because of a variable background and inhomogeneities in the solar atmosphere. The proposed neural network technique with the two hidden neurons responsible for filaments and background pixel values can learn the recognition rules from a single image depicting the solar filament fragmented and labelled visually. The recognition rule has been successfully tested on the 54 other image fragments depicting the filaments on various backgrounds. Despite the background differences, the trained neural network has properly recognized as single as multiple filaments presented in the testing image fragments Therefore, the proposed neural network technique can be effectively used for an automated recognition of filaments in solar images.

References [1]

Zharkova, V.V., Ipson, S.S., Zharkov, S.I., Benkhalil, A., Aboudarham, J., Bentley, R.D.: A full disk image standardisation of the synoptic solar observations at the Meudon Observatory. Solar Physics (2002) accepted [2] Bentley, R.D. et al: The European grid of solar observations. Proceedings of the 2nd Solar Cycle and Space Weather Euro-Conference, Vico Equense, Italy (2001) 603 [3] Qahwaji, R., Green, R.: Detection of closed regions in digital images. The International Journal of Computers and Their Applications 8 (4) (2001) 202207 [4] Bader, D.A., Jaja, J., Harwood, D., Davis L.S.: Parallel algorithms for image enhancement and segmentation by region growing with experimental study. The IEEE Proceedings of IPPS'96 (1996) 414 [5] Turmon, M. Pap, J., Mukhtar, S.: Automatically finding solar active regions using SOHO/MDI photograms and magnetograms (2001) [6] Turmon, M., Mukhtar, S., Pap, J.: Bayesian inference for identifying solar active regions (2001) [7] Gao, J., Zhou, M., Wang, H.: A threshold and region growing method for filament disappearance area detection in solar images. The Conference on Information Science and Systems, The Johns Hopkins University (2001) [8] Bishop, C.M.: Neural networks for pattern recognition. Oxford University Press (1995) [9] Nabney, I.T.: NETLAB: Algorithms for pattern recognition. Springer-Verlag (1995) [10] Schetinin, V.: A Learning Algorithm for Evolving Cascade Neural Networks. Neural Processing Letters, Kluwer, 1 (2003 )

Learning Multi-class Neural-Network Models from Electroencephalograms Vitaly Schetinin1, Joachim Schult2, Burkhart Scheidt2, and Valery Kuriakin3 1

Department of Computer Science, Harrison Building, University of Exeter, EX4 4QF, UK [email protected] 2 Friedrich-Schiller-University of Jena, Ernst-Abbe-Platz 4, 07740 Jena, Germany [email protected] 3 Intel Russian Research Center, N. Novgorod, Russia

Abstract. We describe a new algorithm for learning multi-class neuralnetwork models from large-scale clinical electroencephalograms (EEGs). This algorithm trains hidden neurons separately to classify all the pairs of classes. To find best pairwise classifiers, our algorithm searches for input variables which are relevant to the classification problem. Despite patient variability and heavily overlapping classes, a 16-class model learnt from EEGs of 65 sleeping newborns correctly classified 80.8% of the training and 80.1% of the testing examples. Additionally, the neural-network model provides a probabilistic interpretation of decisions.

1

Introduction

Learning classification models from electroencephalograms (EEGs) is still a complex problem [1] - [7] because of the following problems: first, the EEGs are strongly nonstationary signals which depend on an individual Background Brain Activity (BBA) of patients; second, the EEGs are corrupted by noise and muscular artifacts; third, a given set of EEG features may contain features which are irrelevant to the classification problem and may seriously hurt the classification results and fourth, the clinical EEGs are the large-scale data which are recorded during several hours and for this reason the learning time becomes to be crucial. In general, multi-class problems can be solved by using one-against-all binary classification techniques [8]. However, a natural way to induce multi-class concepts from real data is to use Decision Tree (DT) techniques [9] - [12] which exploit a greedy heuristic or hill-climbing strategy to find out input variables which efficiently split the training data into classes. To induce linear concepts, multivariate or oblique DTs have been suggested which exploit the threshold logical units or so-called perceptions [13] - [16]. Such multivariate DTs known also as Linear Machines (LM) are able to classify linearly separable V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 155-162, 2003.  Springer-Verlag Berlin Heidelberg 2003

156

Vitaly Schetinin et al.

examples. Using the algorithms [8], [13] - [15], the LMs can also learn to classify non-linearly separable examples. However, such DT methods applied for inducing multi-scale problems from real large-scale data become impractical due to large computations [15], [16]. Another approach to multiple classification is based on pairwise classification [17]. A basic ides behind this method is to transform a q-class problem into q(q - 1)/2 binary problems, one for each pair of classes. In this case the binary decision problems are presented by fewer training examples and the decision boundaries may be considerably simpler than in the case of one-against-all binary classification. In this paper we describe a new algorithm for learning multi-class neural-network models from large-scale clinical EEGs. This algorithm trains hidden neurons separately to classify all the pairs of classes. To find best pairwise classifiers, our algorithm searches for input variables which are relevant to the classification problem. Additionally, the neural-network model provides a probabilistic interpretation of decisions. In the next section we define the classification problem and describe the EEG data. In section 3 we describe the neural-network model and algorithm for learning pairwise classification. In section 4 we compare our technique with the standard data mining techniques on the clinical EEGs, and finally we discuss the results.

2

A Classification Problem

In order to recognize some brain pathologies of newborns whose prenatal age range between 35 and 51 weeks, clinicians analyze their EEGs recorded during sleeping and then evaluate a EEG-based index of brain maturation [4] - [7]. So, in the pathological cases, the EEG index does not match to the prenatal maturation. Following [6], [7] we can use the EEGs recorded from the healthy newborns and define this problem as a multi-class one, i.e., a 16-class concept. Then all the EEGs of healthy newborns should be classified properly to their prenatal ages, but the pathological cases should not. To build up such a multi-class concept, we used the 65 EEGs of healthy newborns recorded via the standard electrodes C3 and C4. Then, following [4], [5], [6], these records were segmented and represented by 72 spectral and statistical features calculated on a 10-sec segment into 6 frequency bands such as sub-delta (0-1.5 Hz), delta (1.5-3.5 Hz), theta (3.5-7.5 Hz), alpha (7.5-13.5 Hz), beta 1 (13.5-19.5 Hz), and beta 2 (19.5-25 Hz). Additional features were calculated as the spectral variances. EEGviewer has manually deleted the artifacts from these EEGs and then assigned normal segments to the 16 classes correspondingly with age of newborns. Total sum of the labeled EEG segments was 59069. Analyzing the EEGs, we can see how heavily these depend on the BBA of newborns. Formally, we can define the BBA as a sum of spectral powers over all the frequency bands. As an example, Fig. 1 depicts the BBA calculated for two newborns aged 49 weeks. We can see that the BBA depicted as a dark line chaotically varies during sleeping and causes the variations of the EEG which significantly alter the class boundaries.

Learning Multi-class Neural-Network Models from Electroencephalograms

157

Clearly, we can beforehand calculate and then subtract the BBA from all the EEG features. Using this pre-processing technique we can remove the chaotic oscillations from the EEGs and expect improving the classification accuracy. 4

5

x 10

4 3 2 1 0

0

200

400

600

800

1000

1200

1400

4

5

x 10

4 3 2 1 0

0

500

1000

1500

Fig. 1. EEG segments of two sleeping newborns aged 49 weeks

Below we describe a neural-network technique we developed to learn multi-class concept from the EEGs.

3

The Neural-Network Model and Learning Algorithm

The idea behind our method of multiple classification is to separately train the hidden neurons of neural network and combine them to approximate the dividing surfaces. These hidden neurons learn to divide the examples from each pair of classes. For q classes, therefore, we need to learn q(q - 1)/2 binary classifiers. The hidden neurons that deal with one class are combined into one group, so that the number of the groups corresponds to the number of the classes. The hidden neurons combined into one group approximate the dividing surfaces for the corresponding classes. Let fi/j be a threshold activation function of hidden neuron which learns to divide the examples x of ith and jth classes Ωi and Ωj respectively. The output y of the hidden neuron is y = fi/j(x) = 1, ∀ x ∈ Ωi,, and y = fi/j(x) = – 1, ∀ x ∈ Ωj.

(1)

Assume q = 3 classification problem with overlapping classes Ω1, Ω2 and Ω3 centered into C1, C2, and C3, as Fig. 2(a) depicts. The number of hidden neurons for this example is equal to 3. In the Fig. 2(a) lines f1/2, f1/3 and f2/3 depict the hyperplanes of the hidden neurons trained to divide the examples of three pair of the classes which are (1) Ω1 and Ω2, (2) Ω1 and Ω3, and (3) Ω2 and Ω3.

158

Vitaly Schetinin et al. 1

w

Ω1 f1/3

g1 = f1/2 + f1/3

f1/2

f1/2

x1

+1

x

g3 = - f1/3 - f2/3

g1

2

w

Ω3

+1

-1

x2

f1/3

+1

g2

3

g2 = f2/3 - f1/2 f2/3

Ω2

xm

w

-1 f2/3

g3

-1

Fig. 2. The dividing surfaces (a) g1, g2, and g3, and the neural network (b) for q = 3 classes

Combining these hidden neurons into q = 3 groups we built up the new hyperplanes g1, g2, and g3. The first one, g1, is a superposition of the hidden neurons f1/2 and f1/3, i.e., g1 = f1/2 + f1/3. The second and third hyperplanes are g2 = f2/3 – f1/2 and g3 = – f1/3 – f2/3 correspondently. For hyperplane g1, the outputs f1/2 and f1/3 are summed with the weights 1 because both give the positive outputs on the examples of the class Ω1. Correspondingly, for hyperplane g3, the outputs f1/3 and f2/3 are summed with weights –1 because they give the negative outputs on the examples of the class Ω3. Fig. 2(b) depicts for this example a neural network structure consisting of three hidden neurons f1/2, f1/3, and f2/3 and three output neurons g1, g2, and g3. The weight vectors of hidden neurons here are w1, w2, and w3. These hidden neurons are connected to the output neurons with weights equal to (+1, +1), (–1, +1) and (–1, –1), respectively. We can see that in general case for q > 2 classes, the neural network consists of q output neurons g1, …, gq and q(q – 1)/2 hidden neurons f1/2, …, fi/j, …, fq - 1/q, where i < j = 2, …, q. Each output neuron gi is connected to the (q – 1) hidden neurons which are partitioned into two groups. The first group consists of the hidden neurons fi/k for which k > i. The second group consists of the hidden neurons fk/i for which k < i. So, the weights of output neuron gi connected to the hidden neurons fi/k and fk/i are equal to + 1 and –1, respectively. As the EEG features may be irrelevant to the binary classification problems, for learning the hidden neurons we use a bottom up search strategy which selects features providing the best classification accuracy [11]. Below we discuss the experimental results of our neural-network technique applied to the real multi-class problem.

4

Experiments and Results

To learn the 16-class concept from the 65 EEG records, we used the neural network model described above. For training and testing we used 39399 and 19670 EEG segments, respectively. Correspondingly, for q = 16 class problem, the neural network includes q (q – 1)/2 = 120 hidden neurons or binary classifiers with a threshold activation function (1).

Learning Multi-class Neural-Network Models from Electroencephalograms

159

The testing errors of binary classifiers varied from 0 to 15% as depicted in Fig. 3(a). The learnt classifiers exploit different sets of EEG features. The number of these features varies between 7 and 58 as depicted in Fig 3(b). Our method has correctly classified 80.8% of the training and 80.1% of the testing examples taken from 65 EEG records. Summing all the segments belonging to one EEG record, we can improve the classification accuracy up to 89.2% and 87.7% of the 65 EEG records for training and testing, respectively. 20

Error,%

15 10 5 0

0

20

40

60 Classifiers

80

100

120

0

20

40

60 Classifiers

80

100

120

Number of Features

60

40 20

0

Fig. 3. The testing errors (a) and the number of features (b) for each of 120 binary classifiers

In Fig. 4 we depict the outputs of our model summed over all the testing EEG segments of two patients which belong to the second and third age groups, respectively. In both cases, most parts of the segments were correctly classified. In addition, summing the outputs over all the testing EEG segments, we may provide a probabilistic interpretation of decisions. For example, we assign the patients to 2 and 3 classes with probabilities 0.92 and 0.58, respectively, calculated as parts of the correctly classified segments. These probabilities give us the additional information about the confidence of decisions. We compared the performance of our neural network technique and the standard data mining techniques on the same EEG data. First, we tried to train a standard feedforward neural network consisted of q = 16 output neurons and a predefined number of hidden neurons and input nodes. The number of hidden neurons was defined between 5 and 20, and the number of the input nodes between 72 and 12 by using the standard principal component analysis. Note that in our experiments we could not use more than 20 hidden neurons because even a fast Levenberg-Marquardt learning algorithm provided by MATLAB has required an enormous time-consuming computational effort. Because of a large number of the training examples and classes, we could not also use more than and 25% of the training data and, as a result, the trained neural network has correctly classified less than 60% of the testing examples.

Vitaly Schetinin et al.

Disribution of Segments

Disribution of Segments

160

1 0.8 0.6 0.4 0.2 0

0

2

4

6

8 Classes

10

12

14

16

0

2

4

6

8 Classes

10

12

14

16

0.8 0.6 0.4 0.2 0

Fig. 4. Probabilistic interpretation of decisions for two patients

Second, we trained q = 16 binary classifiers to distinguish one class against others. We defined the same activation function for these classifiers and trained them on the whole data. However the classification accuracy was 72% of the testing examples. Third, we induced a decision tree consisting of (q – 1) = 15 binary decision trees trained to split (q – 1) subsets of the EEG data. That is, the first classifier learnt to divide the examples taken from classes Ω1, …, Ω8 and Ω9, …, Ω16. The second classifier learnt to divide the examples taken from classes Ω1, …, Ω4 and Ω5, …, Ω8, and so on. However the classification accuracy on the testing data was 65%.

5

Conclusion

For learning multi-class concepts from large-scale heavily overlapping EEG data, we developed a neural network technique and learning algorithm. Our neural network consists of hidden neurons which perform the binary classification for each pairs of classes. The hidden neurons are trained separately and then their outputs are combined in order to perform the multiple classification. This technique has been used to learn a 16-class concept from 65 EEG records represented by 72 features some of which were irrelevant. Having compared our technique with the other classification methods on the same EEG data, we found out that it gives the better classification accuracy for an acceptable learning time. Thus, we conclude that the new technique we developed for learning multi-class neural network models performs on the clinical EEGs well. We believe that this technique may be also used to solve other large-scale multi-class problems presented many irrelevant features.

Learning Multi-class Neural-Network Models from Electroencephalograms

161

Acknowledgments The research has been supported by the University of Jena (Germany). The authors are grateful to Frank Pasemann for enlightening discussions, Joachim Frenzel from the University of Jena for the clinical EEG records, and to Jonathan Fieldsend from the University of Exeter (UK) for useful comments.

References [1] [2] [3]

[4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

Riddington, E., Ifeachor, E., Allen, N., Hudson, N., Mapps, D.: A Fuzzy Expert System for EEG Interpretation. Neural Networks and Expert Systems in Medicine and Healthcare, University of Plymouth (1994) 291-302 Anderson, C., Devulapalli, S., Stolz, E.: Determining Mental State from EEG Signals Using Neural Networks. Scientific Programming 4 (1995) 71-183 Galicki, M., Witte, H., Dörschel, J., Do ering, A., Eiselt, M., Grießbach, G.: Common Optimization of Adaptive Preprocessing Units and a Neural Network During the Learning: Application in EEG Pattern Recognition. J. Neural Networks 10 (1997) 1153-1163 Breidbach, O., Holthausen, K., Scheidt, B., Frenzel, J.: Analysis of EEG Data Room in Sudden Infant Death Risk Patients. J. Theory Bioscience 117 (1998) 377-392 Holthausen, K., Breidbach, O., Schiedt, B., Frenzel, J.: Clinical Relevance of Age Dependent EEG Signatures in Detection of Neonates at High Risk of Apnea. J. Neuroscience Letter 268 (1999) 123-126 Holthausen, K., Breidbach, O., Scheidt, B., Schult, J., Queregasser, J.: Brain Dismaturity as Index for Detection of Infants Considered to be at Risk for Apnea. J. Theory Bioscience 118 (1999) 189-198 Wackermann, J., Matousek, M.: From the EEG Age to Rational Scale of Brain Electric maturation. J. Electroencephalogram Clinical Neurophysiology 107 (1998) 415-421 Duda, R.O., Hart, P.E.: Pattern Classification and Scene Analysis. Wiley and Sons, New York (1973) Quinlan, J.: Induction of Decision Trees. J. Machine Learning 1 (1986) 81-106 Cios, K., Liu, N.: A Machine Learning Method for Generation of Neural Network Architecture. J. IEEE Transaction of Neural Networks 3 (1992) 280-291 Galant, S.: Neural Network Learning and Expert Systems. MIT Press, Cambridge, MA (1993) Kononenko, I., Šimec, E.: Induction of Decision Trees using RELIEFF. In: Kruse, R., Viertl, R., Riccia, G. Della (eds.). CISM Lecture Notes, Springer Verlag (1994) Utgoff, P., Brodley, C.: Linear Machine Decision Trees. COINS Technical Report 91-10, University of Massachusetts, Amhert, MA (1991) Brodley, C., Utgoff, P.: Multivariate Decision Trees. COINS Technical Report 92-82, University of Massachusetts, Amhert, MA (1992)

162

Vitaly Schetinin et al.

[15] Frean, M.: A Thermal Perceptron Learning Rule. J. Neural Computational 4 (1992) [16] Murthy, S., Kasif, S., Salzberg, S.: A System for Induction of Oblique Decision Trees. J. Artificial Intelligence Research 2 (1994) 1-33 [17] Hastie, T., Tibshirani, R.: Classification by Pairwise Coupling. Advances in Neural Information Processing Systems 10 (NIPS-97) (1998) 507-513

Establishing Safety Criteria for Artificial Neural Networks Zeshan Kurd and Tim Kelly Department of Computer Science University of York, York, YO10 5DD, UK. {zeshan.kurd,tim.kelly}@cs.york.ac.uk

Abstract. Artificial neural networks are employed in many areas of industry such as medicine and defence. There are many techniques that aim to improve the performance of neural networks for safety-critical systems. However, there is a complete absence of analytical certification methods for neural network paradigms. Consequently, their role in safety-critical applications, if any, is typically restricted to advisory systems. It is therefore desirable to enable neural networks for highlydependable roles. This paper defines the safety criteria which if enforced, would contribute to justifying the safety of neural networks. The criteria are a set of safety requirements for the behaviour of neural networks. The paper also highlights the challenge of maintaining performance in terms of adaptability and generalisation whilst providing acceptable safety arguments.

1

Introduction

Typical uses of ANNs (Artificial Neural Networks) in safety-critical systems include areas such as medical systems, industrial process & control and defence. An extensive review of many areas of ANN use in safety-related applications has been provided by a recent U.K. HSE (Health & Safety Executive) report [1]. However, the roles for these systems have been restricted to advisory, since no convincing safety arguments has yet been produced. They are typical examples of achieving improved performance without providing sufficient safety assurance. There are many techniques for traditional multi-layered perceptrons [2, 3] which aim to improve generalisation performance. However, these performance-related techniques do not provide acceptable forms of safety arguments. Moreover, any proposed ANN model that attempts to satisfy safety arguments must carefully ensure that measures incorporated do not diminish the advantages of the ANN. Many of the existing approaches which claim to contribute to safety-critical applications focus more upon improving generalisation performance and not generating suitable safety arguments. One particular type of ANN model employs diversity [2] which is an ensemble of ANNs where each is devised by different methods to encapsulate as much of the target function as possible. Results demonstrate improvement in V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 163-169, 2003.  Springer-Verlag Berlin Heidelberg 2003

164

Zeshan Kurd and Tim Kelly

generalisation performance over some test set but lack in the ability to analyse and determine the function performed by each member of the ensemble. Other techniques such as the validation of ANNs and error bars [3] may help deal with uncertainty in network outputs. However, the ANN is still viewed as a black-box and lacks satisfactory analytical processes to determine the overall behaviour. Finally, many development lifecycles [4] for ANNs lack provision for analytical processes such as hazard analysis and focuses more upon black-box approaches. However, work performed on fuzzy-ANNs [5] attempts to approach development based upon refining partial specifications.

2

The Problem

The criticality of safety-critical software can be defined as directly or indirectly contributing to the occurrence of a hazardous system state [6]. A hazard is a situation, which is potentially dangerous to man, society or the environment [7]. When developing safety-critical software, there are a set of requirements which must be enforced and are formally defined in many safety standards [8]. One of the main tools for determining requirements is the use of a safety case to encapsulate all safety arguments for the software. A safety case is defined in Defence Standard 00-55 [8] as: “The software safety case shall present a well-organised and reasoned justification based on objective evidence, that the software does or will satisfy the safety aspects of the Statement of Technical Requirements and the Software Requirements specification.” Some of the main components in formulating safety requirements are hazard analysis and mitigation. Function Failure Analysis (FFA) [9] is a predictive technique to distinguish and refine safety requirements. It revolves around system functions and deals with identifying failures to provide a function, incorrect function outputs, analysing effects of failures and determining actions to improve design. Another technique is Software Hazard Operability Study (SHAZOP) [10] which uses ‘guide words' for qualitative analysis to identify hazards not previously considered. It attempts to analyse all variations of the system based on these guide words and can uncover causes, consequences, indications and recommendations for particular identified hazards. Arguing the satisfaction of the safety requirements is divided into analytical arguments and arguments generated from testing. This is defined in a widely accepted Defence Standard 00-55 [8]. Arguments from testing (such as white-box or blackbox) can be generated more easily than analytical arguments. Unlike neural networks, the behaviour of the software is fully described and available for analysis. However, generating analytical arguments is more problematic. Some of these problems are mainly associated with providing sufficient evidence that all hazards have been identified and mitigated. There are several safety processes performed during the life-cycle of safety-critical systems [11]. Preliminary Hazard Identification (PHI) is the first step in the lifecycle,

Establishing Safety Criteria for Artificial Neural Networks

165

it forms the backbone of all following processes. It aims to identify, manage and control all potential hazards in the proposed system. Risk Analysis and Functional Hazard Analysis (FHA) analyses the severity and probability of potential accidents for each identified hazard. Preliminary System Safety Assessment (PSSA) [9] ensures that the proposed design will refine and adhere to safety requirements and help guide the design process. System Safety Analysis (SSA) demonstrates through evidence that safety requirements have been met. It uses inductive and deductive techniques to examine the completed design and implementation. Finally, the Safety Case [11] generated throughout development delivers a comprehensible and defensible argument that the system is acceptably safe to use in a given context. Any potential safety case must overcome problems associated with typical neural networks. Some typical problems may be concerned with ANN structure and topology. These are factors that may influence the generalisation performance of the ANN. Another problem lies in determining the training and test set where they must represent the desired function using a limited number of samples. Dealing with noise during training is also problematic to ensure that the network does not deviate from the required target function. Other issues related to the learning or training process may involve forgetting of data particularly when using the back-propagation algorithm. This could lead to poor generalisation and long training times. Furthermore, problems during training may result in learning settling in local minima instead of global and deciding upon appropriate stopping points for the training process. This could be aided by cross-validation [12] but relies heavily on test sets. One of the key problems is the inability to analyse the network such that a whitebox view of the behaviour can be presented. This contributes to the need for using test sets to determine generalisation performance as an overall error and the lack of explanation mechanisms for network outputs. The inability to analyse also makes it difficult to identify and control potential hazards in the system and provide assurance that some set of requirements are met.

3

Safety Criteria for Neural Networks

We have attempted to establish safety criteria for ANNs (Artificial Neural Networks) that defines minimum behavioural properties which must be enforced for safetycritical contexts. By defining requirements from a high-level perspective, the criteria are intended to apply to most types of neural networks. Figure 1.0 illustrates the safety criteria in the form of Goal Structuring Notation (GSN) [11] which is commonly used for composing safety case patterns. Each symbol in figure 1.0 has a distinct meaning and is a subset of the notation used in GSN. The boxes illustrate goals or sub-goals which need to be fulfilled. Rounded boxes denote the context in which the corresponding goal is stated. The rhomboid represents strategies to achieve goals. Diamonds underneath goals symbolise that the goal requires further development leading to supporting arguments and evidence.

166

Zeshan Kurd and Tim Kelly G1

C2 Use of network in safetycritical context must ensure specific requirements are met

Neural network is acceptably safe to perform a specified function within the safety-critical context

C1 Neural Network model definition

S1 Argument over key safety criteria

C6 A fault is classified as an input that lies outside the specified input set

C3 ‘Acceptably safe’ will be determined by the satisfaction of safety criteria

G2 Pattern matching functions for neural network have been correctly mapped

C4 Function may partially or completely satisfy target function

G3

G4

G5

Observable behavior of the neural network must be predictable and repeatable

The neural network tolerates faults in its inputs

The neural network does not create hazardous outputs

C5 Known and unknown inputs

C7 Hazardous output is defined as an output outside a specified set or target function

Fig. 1. Preliminary Safety Criteria for Artificial Neural Networks

The upper part of the safety criteria consists of a top-goal, strategy and other contextual information. The goal G1 if achieved, allows the neural network to be used in safety-critical applications. C1 requires that a specific ANN model is defined such as multi-layered perceptron or other models. C2 intends for the ANN to be used when conventional software or other systems cannot provide the desired advantages. C3 attempts to highlight that ‘acceptably safe' is related to product and process based arguments and will rely heavily on sub-goals. The strategy S1 will attempt to generate safety arguments from the sub-goals (which form the criteria) to fulfil G1. The goals G2 to G5 presents the following criteria: Criterion G2 ensures that the function performed by the network represents the target or desired function. The function represented by the network may be considered as input-output mappings and the term ‘correct' refers to the target function. As expressed in C4, the network may also represent a subset of the target function. This is a more realistic condition, if all hazards can be identified and mitigated for the subset. This may also avoid concerns for attempting to solve a problem where analysis may not be able to determine whether totality of the target function is achieved. Previous work on dealing with and refining partial specification for neural networks [5] may apply. However, additional methods to analyse specifications in terms of performance and safety (existence of potential hazards) may be necessary. Forms of sub-goals or strategies for arguing G2 may involve using analytical methods such as decompositional approaches [13]. This attempts to extract behaviour

Establishing Safety Criteria for Artificial Neural Networks

167

by analysing the intrinsic structure of the ANN such as each neuron or weight. This will help analyse the function performed by the ANN and help present a white-box view of the network. Techniques to achieve this may involve determining suitable ANN architectures whereby a meaningful mapping exists from each network parameter to some functional description. On the other hand, pedagogical approaches involve determining the function by analysing outputs for input patterns such as sensitivity analysis [14]. This methodology however, maintains a black-box perspective and will not be enough to provide satisfactory arguments for G2. Overall, approaches must attempt to overcome problems associated with the ability of the ANN to explain outputs and generalisation behaviour. Criterion G3 will provide assurance that safety is maintained during ANN learning. The ‘observable behaviour' of the neuron means the input and output mappings that take place regardless of the weights stored on each connection. The ANN must be predictable given examples learnt during training. ‘Repeatable' ensures that any previous valid mapping or output does not become flawed during learning. The forgetting of previously learnt samples must not occur given adaptation to a changing environment. Possible forms of arguments may be associated with functional properties. This may involve providing assurance that learning maintains safety by abiding to some set of behavioural constraints identified through processes such as hazard analysis. Criterion G4 ensures the ANN is robust and safe under all input conditions. An assumption is made that the ANN might be exposed to training samples that do not represent the desired function. Achievement of this goal is optional as it could be considered specific to the application context. The network must either detect these inputs and suppress them, or ensure a safe state with respect to the output and weight configuration. Other possible forms of argument and solutions may involve ‘gating networks' [15] which receives data within a specific validation area of the input space. Criterion G5 is based upon arguments similar to G2. However, this goal focuses solely on the network output. This will result in a robust ANN, which through training and utilisation, ensures that the output is not hazardous regardless of the integrity of the input. For example, output monitors or bounds might be used as possible solutions. Other possible forms of arguments may include derivatives of ‘guarded networks' [15] that receives all data but is monitored to ensure that behaviour does not deviate too much from expectations. The above criteria focuses on product-based arguments in contrast to processbased arguments commonly used for conventional software [16]. This follows an approach which attempts to provide arguments based upon the product rather than a set of processes that have been routinely carried out [16]. However, process and product-based arguments may be closely coupled for justifying the safety ANNs. For example, methods such as developing ANNs using formal languages [17] might reduce faults incorporated during implementation. However the role of analytical tools is highly important and will involve hazard analysis for identifying potential hazards. Some factors which prevent this type of analysis can be demonstrated in typical monolithic neural networks. These scatter their functional behaviour around its

168

Zeshan Kurd and Tim Kelly

weights in incomprehensible fashion resulting in black-box views and pedagogical approaches. Refining various properties of the network such as structural and topological factors may help overcome this problem such as modular ANNs [18]. It may help improve the relationship between intrinsic ANN parameters and meaningful descriptions of functional behaviour. This can enable analysis in terms of domainspecific base functions associated with the application domain [19]. Other safety processes often used for determining existence of potential hazards may require adaptation for compatibility with neural network paradigms.

4

Performance vs. Safety Trade-off

One of the main challenges when attempting to satisfy the criteria is constructing suitable ANN models. The aim is to maintain feasibility by providing acceptable performance (such as generalisation) whilst generating acceptable safety assurance. Certain attributes of the model may need to be limited or constrained so that particular arguments are possible. However, this could lead to over-constraining the performance of ANNs. Typical examples may include permissible learning during development but not whilst deployed. This may provide safety arguments about the behaviour of the network without invalidating them with future changes to implementation. The aim is to generate acceptable safety arguments providing comparable assurance to that achieved with conventional software.

5

Conclusion

To justify the use of ANNs within safety critical applications will require the development of a safety case. For high-criticality applications, this safety case will require arguments of correct behaviour based both upon analysis and test. Previous approaches to justifying ANNs have focussed predominantly on (test-based) arguments of high performance. In this paper we have presented the key criteria for establishing an acceptable safety case for ANNs. Detailed analysis of potential supporting arguments is beyond the scope of this paper. However, we have discussed how it will be necessary to constrain learning and other factors, in order to provide analytical arguments and evidence. The criteria also highlight the need for adapting current safety processes (hazard analysis) and devising suitable ANN models that can accommodate them. The criteria can be considered as a benchmark for assessing the safety of artificial neural networks for highly-dependable roles.

References [1] [2]

Lisboa, P., Industrial use of safety-related artificial neural networks. Health & Safety Executive 327, (2001). Sharkey, A.J.C. and N.E. Sharkey, Combining Diverse Neural Nets, in Computer Science. (1997), University of Sheffield: Sheffield, UK.

Establishing Safety Criteria for Artificial Neural Networks

[3] [4] [5]

[6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19]

169

Nabney, I., et al., Practical Assessment of Neural Network Applications. (2000), Aston University & Lloyd's Register: UK. Rodvold, D.M. A Software Development Process Model for Artificial Neural Networks in Critical Applications. in Proceedings of the 1999 International Conference on Neural Networks (IJCNN'99). (1999). Washington D.C. Wen, W., J. Callahan, and M. Napolitano, Towards Developing Verifiable Neural Network Controller, in Department of Aerospace Engineering, NASA/WVU Software Research Laboratory. (1996), West Virginia University: Morgantown, WV. Leveson, N., Safeware: system safety and computers. (1995): Addison-Wesley. Villemeur, A., Reliability, Availability, Maintainability and Safety Assessment. Vol. 1. (1992): John Wiley & Sons. MoD, Defence Standard 00-55: Requirements for Safety Related Software in Defence Equipment. (1996), UK Ministry of Defence. SAE, ARP 4761 - Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment. (1996), The Society for Automotive Engineers,. MoD, Interim Defence Standard 00-58 Issue 1: HAZOP Studies on Systems Containing Programmable Electronics. (1996), UK Ministry of Defence. Kelly, T.P., Arguing Safety – A Systematic Approach to Managing Safety Cases, in Department of Computer Science. (1998), University of York: York, UK. Kearns, M., A Bound on the Error of Cross Validation Using the Approximation and Estimation Rates, with Consequences for the Training-Test Split, AT&T Bell Laboratories. Andrews, R., J. Diederich, and A. Tickle, A survey and critique of techniques for extracting rules from trained artificial neural networks. (1995), Neurocomputing Research Centre, Queensland University of Technology. Kilimasaukas, C.C., Neural nets tell why. Dr Dobbs's, (April 1991): p. 16-24. Venema, R.S., Aspects of an Integrated Neural Prediction System. (1999), Rijksuniversiteit, Groningen: Groningen, Netherlands. Weaver, R.A., J.A. McDermid, and T.P. Kelly. Software Safety Arguments: Towards a Systematic Categorisation of Evidence. in International System Safety Conference. (2002). Denver, CO. Dorffner, G., H. Wiklicky, and E. Prem, Formal neural network specification and its implications on standardisation. Computer Standards and Interfaces, (1994). 16 (205-219). Osherson, D.N., S. Weinstein, and M. Stoli, Modular Learning. Computational Neuroscience, Cambridge - MA, (1990): p. 369-377. Zwaag, B.J. and K. Slump. Process Identification Through Modular Neural Networks and Rule Extraction. in 5th International FLINS Conference. (2002). Gent, Belgium: World Scientific.

Neural Chaos Scheme of Perceptual Conflicts Haruhiko Nishimura1 , Natsuki Nagao2 , and Nobuyuki Matsui3 1

Hyogo University of Education 942-1 Yashiro-cho, Hyogo 673-1494, Japan [email protected] 2 Kobe College of Liberal Arts 1-50 Meinan-cho, Akashi-shi, Hyogo 673-0001, Japan [email protected] 3 Himeji Institute of Technology 2167 Shosya, Himeji-shi, Hyogo 671-2201, Japan [email protected]

Abstract. Multistable perception is perception in which two (or more) interpretations of the same ambiguous image alternate while an obserber looks at them. Perception undergoes involuntary and random-like change. The question arises whether the apparent randomness of alternation is real (that is, due to a stochastic process) or whether any underlying deterministic structure to it exists. Upon this motivation, we have examined the spatially coherent temporal behaviors of multistable perception model based on the chaotic neural network from the viewpoint of bottom-up high dimensional approach. In this paper, we focus on dynamic processes in a simple (minimal) system which consists of three neurons, aiming at further understanding of the deterministic mechanism.

1

Introduction

Multistable perception is perception in which two (or more) interpretations of the same ambiguous image alternate spontaneously while an obserber looks at them. Figure-ground, perspective (depth) and semantic ambiguities are well known (As an overview, for example, see [1] and [2]). Actually, when we view the Necker cube which is a classic example of perspective alternation, a part of the figure is perceived either as front or back of a cube and our perception switches between the two different interpretations. In this circumstance the external stimulus is kept constant, but perception undergoes involuntary and random-like change. The measurements have been quantified in psychophysical experiments and it has become evident that the frequency of the time intervals spent on each percept T (n) in Fig.1 is approximately Gamma distributed [3, 4]. The Gestalt psychologist Wolfgang K¨ ohler [5] claimed that perceptual alternation occurs owing to different sets of neurons getting “tired” of firing after they have done so for a long time. The underlying theoretical assumptions are that: 1. different interpretations are represented by different patterns of neural activity, 2. perception corresponds to whichever pattern is most active in the V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 170–176, 2003. c Springer-Verlag Berlin Heidelberg 2003


Neural Chaos Scheme of Perceptual Conflicts

171

t

T(2)

T(1)

T(3)

Fig. 1. A schematic example of switching behavior brain at the time, 3. neural fatigue causes different patterns of activation to dominate at different times. Upon this neural fatigue hypothesis, synergetic Ditzinger-Haken model [6, 7] was proposed and is known to be able to reproduce the Gamma distribution of T well by subjecting the time-dependent attention parameters to stochastic forces (white noises). However, the question arises whether the apparent randomness of alternation is real (that is, due to a stochastic process) or whether any underlying deterministic structure to it exists. Until now diverse types of chaos have been confirmed at several hierarchical levels in the real neural systems from single cells to cortical networks (e.g. ionic channels, spike trains from cells, EEG) [8]. This suggests that artificial neural networks based on the McCulloch-Pitts neuron model should be re-examined and re-developed. Chaos may play an essential role in the extended frame of the Hopfield neural network beyond the only equilibrium point attractors. Following this idea, we have examined the spatially coherent temporal behaviors of multistable perception model based on the chaotic neural network [9, 10] from the viewpoint of bottom-up high dimensional approach. In this paper, we focus on dynamic processes in a simple (minimal) system which consists of three neurons, aiming at further understanding of the deterministic mechanism.

2

Model and Method

The chaotic neural network (CNN) composed of N chaotic neurons is described as [11, 12] Xi (t + 1) = f (ηi (t + 1) + ζi (t + 1)) ηi (t + 1) =

N


wij

j=1

ζi (t + 1) = −α

t


(1)

kfd Xj (t − d)

(2)

krd Xi (t − d) − θi

(3)

d=0 t
d=0

where Xi : output of neuron i(−1 ≤ Xi ≤ 1), wij : synaptic weight from neuron j to neuron i, θi : threshold of neuron i, kf (kr ) : decay factor for the feed-

172

Haruhiko Nishimura et al.

back(refractoriness) (0 ≤ kf , kr < 1), α : refractory scaling parameter, f : output function defined by f (y) = tanh(y/2ε) with the steepness parameter ε. Owing to the exponentially decaying form of the past influence, Eqs.(2) and (3) can be reduced to ηi (t + 1) = kf ηi (t) +

N


wij Xj (t)

(4)

j=1

ζi (t + 1) = kr ζi (t) − αXi (t) + a

(5)

where a is temporally constant a ≡ −θi (1 − kr ). All neurons are updated in parallel, that is, synchronously. The network corresponds to the conventional discrete-time Hopfield network when α = kf = kr = 0 (Hopfield network point (HNP)). Under an external stimulus, Eq.(1) is influenced as   (6) Xi (t + 1) = f ηi (t + 1) + ζi (t + 1) + σi where {σi } is the effective term by external stimulus. The two competitive interpretations are stored in the network as minima of the energy map : E=−

1
wij Xi Xj 2 ij

(7)

at HNP. The conceptual picture of our model is shown in Fig.2. Under the external stimulus {σi }, chaotic activities arise on the neural network and cause the transitions between stable states of HNP. This situation corresponds to the dynamic multistable perception.

3

Simulations and Results

To carry out computational experiments, we consider a minimal network which consists of three neurons (N = 3). Of 23 possible states, Two interpretation patterns {ξi11 } = (1, −1, 1) and {ξi12 } = (−1, 1, −1) are chosen and then the corresponding bistable landscape is made on the network dynamics by   0 −2 2 1 {wij } =  −2 0 −2  . 3 2 −2 0 In the case of HNP, the stored patterns {ξi11 } and {ξi12 } are stable (with E = −2) and the remaining 6 patterns are all unstable (with E = 23 ). (−1, −1, 1), (1, 1, 1) and (1, −1, −1) converge to {ξi11 }, and (1, 1, −1), (−1, −1, −1) and (−1, 1, 1) converge to {ξi12 } as shown in Fig.3.

Neural Chaos Scheme of Perceptual Conflicts

173

1 {σi } = s { ξ i }

stimulus

E{X}

(−1, 1, 1)

( 1, 1, 1)

ambiguous 1 { ξi }(1,1,0) 12

stable { ξi } (−1, 1,−1)

( 1, 1,−1) 11

11

{ξ i }

1

{ξ i }

12

{ξ i }

stable { ξi } (1, −1, 1)

(−1,−1, 1)

{X} (−1,−1,−1)

Fig. 2. Conceptual picture illustrating state transitions induced by chaotic activity

( 1,−1,−1)

Fig. 3. Pattern states for N=3 neurons corresponding to the ambiguous figure {ξi1 } and its interpretations {ξi11 } and {ξi12 }, and flow of the network at HNP

Figure 4 shows a time series evolution of CNN (kf = 0.5, kr = 0.8, α = 0.46, a = 0, ε = 0.015) under the stimulus {σi } = 0.59{ξi1}. Here, m11 (t) =

N 1
11 ξ Xi (t) N i=1 i

(8)

and is called the overlap of the network state {Xi } and the interpretation pattern {ξi11 }. A switching phenomenon between {ξi11 } (m11 = 1.0) and {ξi12 } (m11 = −1.0) can be observed. Bursts of switching are interspersed with prolonged periods during which {Xi } trembles near {ξi11 } or {ξi12 }. Evaluating the maximum Lyapunov exponent to be positive (λ1 = 0.30), we find that the network is dynamically in chaos. Figure 5 shows the return map of the active potential hi (t) = ηi (t) + ζi (t) + σi of a neuron(#3). In the cases λ1 < 0, such switching phenomena do not arise. From the 2×105 iteration data (until t = 2×105 ) of Fig.4, we get 1545 events T (1) ∼ T (1545) staying near one of the two interpretations, {ξi12 }. They have various persistent durations which seem to have a random time course. From the evaluation of the autocorrelation function for T (n), C(k) =< T (n + k)T (n) > − < T (n + k) >< T (n) > (here, <> means an average over time), we find the lack of even short term correlations (−0.06 < C(k)/C(0) < 0.06 for all k). This indicates that successive durations T (n) are independent. The frequency of

174

Haruhiko Nishimura et al.

neuron#:3 2



hi(t+1)





ξ











ξ









ξ







0








−2





 





 




−2

0

2

hi(t)




Fig. 4. Time series of the overlap with Fig. 5. Return map of the active {ξi11 } and the energy map under the stim- potential hi of #3 neuron for the ulus {ξi1 } data up to t=5000. Solid line traces its behavior for a typical T term t=3019 to 3137 0.3

Frequency

0.25 0.2 0.15 0.1 0.05 0 30

60

90

120 150 180 210 240 270 300 T

Fig. 6. Frequency distribution of the persistent durations T (n) and the corresponding Gamma distribution

occurrence of T is plotted for 1545 events in Fig.6. The distribution is well fitted by Gamma distribution ˜

G(T˜ ) =

bn T˜ n−1 e−bT Γ (n)

(9)

with b = 2.03, n = 8.73(χ2 = 0.0078, r = 0.97), where Γ (n) is the Euler-Gamma function. T˜ is the normalized duration T /15 and here 15 step interval is applied to determine the relative frequencies.

Neural Chaos Scheme of Perceptual Conflicts

175

Frequency

0.3

TI=50

0.2 0.1 0

Lower cortical area

Higher cortical area

Frequency

0.3

TI=75

0.2 0.1 0

Neuron assembly

Neuron assembly

Frequency

0.3

TI=100

0.2 0.1 0

Ping-pong

Fig. 7. Conceptual diagram of cortico-cortical interaction within the cortex

30

60

90

120

150

180

210

240

270

300

T

Fig. 8. Frequency distribution of the persistent durations T (n) and the corresponding Gamma distribution, under the background cortical rhythm with bs=0.005

The results are in good agreement with the characteristics of psychophysical experiments [3, 4]. It is found that aperiodic spontaneous switching does not necessitate some stochastic description as in the synergetic D-H model [6, 7]. In our model, the neural fatigue effect proposed by K¨ ohler [5] is considered to be supported by the neuronal refractriness α, and its fluctuation originates in the intrinsic chaotic dynamics of the network. The perceptual switching in binocular rivalry shares many features with multistable perception, and might be the outcome of a general (common) neural mechanism. Ping-pong style matching process based on the interaction between the lower and higher cortical area [13] in Fig.7 may serve as a candidate of this mechanism. According to the input signal from the lower level, the higher level feeds back a suitable candidate among the stored templates to the lower level. If the lower area cannot get a good match, the process starts over again and lasts until a suitable interpretation is found. The principal circuitry among the areas by the cortico-cortical fibers is known to be uniform over the cortex. As a background rhythm from the ping-pong style matching process, we introduce the following sinusoidal stimulus changing between bistable patterns {ξi11 } and {ξi12 }: S1 (t) = bs · cos(2πt/T0 ), S2 (t) = −bs · cos(2πt/T0 ), and S3 (t) = bs · cos(2πt/T0 ), where T0 = 2TI is the period and bs is the background strength. As we can see from the three cases TI = 50, 75, and 100 in Fig.8, the mean value of T is well controlled by the cortical matching period.

176

4

Haruhiko Nishimura et al.

Conclusion

We have shown that the deterministic neural chaos leads to perceptual alternations as responses to ambiguous stimuli in the chaotic neural network. Its emergence is based on the simple process in a realistic bottom-up framework. Our demonstration suggests functional usefulness of the chaotic activity in perceptual systems even at higher cognitive levels. The perceptual alternation appears to be an inherent feature built in the chaotic neuron assembly.It will be interesting to study the brain with the experimental technique (e.g., fMRI) under the circumstance where the perceptual alternation is running.

References [1] Attneave, F.: Multistability in perception. Scientific American 225 (1971) 62–71 170 [2] Kruse, P., Stadler, M., eds.: Ambiguity in Mind and Nature. Springer-Verlag (1995) 170 [3] Borsellino, A., Marco, A. D., Allazatta, A., Rinsei, S., Bartolini, B.: Reversal time distribution in the perception of visual ambiguous stimuli. Kybernetik 10 (1972) 139–144 170, 175 [4] Borsellino, A., Carlini, F., Riani, M., Tuccio, M. T., Marco, A. D., Penengo, P., Trabucco, A.: Effects of visual angle on perspective reversal for ambiguous patterns. Perception 11 (1982) 263–273 170, 175 [5] K¨ ohler, W.: Dynamics in psychology. Liveright, New York (1940) 170, 175 [6] Ditzinger, T., Haken, H.: Oscillations in the perception of ambiguous patterns: A model based on synergetics. Biological Cybernetics 61 (1989) 279–287 171, 175 [7] Ditzinger, T., Haken, H.: The impact of fluctuations on the recognition of ambiguous patterns. Biological Cybernetics 63 (1990) 453–456 171, 175 [8] Arbib, M. A.: The Handbook of Brain Theory and Neural Networks. MIT Press (1995) 171 [9] Nishimura, H., Nagao, N., Matsui, N.: A perception model of ambiguous figures based on the neural chaos. In Kasabov, N., et al., eds.: Progress in ConnectionistBased Information Systems. Volume 1. Springer-Verlag (1997) 89–92 171 [10] Nagao, N., Nishimura, H., Matsui, N.: A neural chaos model of multistable perception. Neural Processing Letters 12 (2000) 267–276 171 [11] Aihara, K., Takabe, T., Toyoda, M.: Chaotic neural networks. Phys.Lett. A 144 (1990) 333–340 171 [12] Nishimura, H., Katada, N., Fujita, Y.: Dynamic learning and retrieving scheme based on chaotic neuron model. In Nakamura, R., et al., eds.: Complexity and Diversity. Springer-Verlag (1997) 64–66 171 [13] Mumford, D.: On the computational architecture of the neocortex ii. the role of cortico-cortical loops. Biol. Cybern. 66 (1992) 241–251 175

Learning of SAINNs from Covariance Function: Historical Learning Paolo Crippa and Claudio Turchetti Dipartimento di Elettronica ed Automatica, Universit` a Politecnica delle Marche Via Brecce Bianche, I-60131 Ancona, Italy {pcrippa, turchetti}@ea.univpm.it

Abstract. In this paper the learning capabilities of a class of neural networks named Stochastic Approximate Identity Neural Networks (SAINNs) have been analyzed. In particular these networks are able to approximate a large class of stochastic processes from the knowledge of their covariance function.

1

Introduction

One attracting property of neural networks is their capability in approximating stochastic processes or, more in general, input-output transformations of stochastic processes. Here we consider a wide class of nonstationary processes for which a canonical representation can be defined from the Karhunen-Lo`eve theorem [1]. Such a representation constitutes a model that can be used for the definition of stochastic neural networks based on the Approximate Identity (AI) functions whose approximating properties are widely studied in [3, 2]. The aim of this work is to show that these stochastic neural networks are able to approximate a given stochastic process belonging to this class, provided only its covariance function is known. This property has been called historical learning.

2

Approximation of Stochastic Processes by SAINNs

Let us consider a stochastic process (SP) admitting the canonical representation
ϕ (t) = Φ (t, λ) η (dλ) , (1) Λ

where η (dλ) is a stochastic measure and Φ (t, λ) is a family of complete realvalued functions of λ ∈ Λ depending on the parameter t ∈ T such that
2 Φ (t, λ) Φ (s, λ)F (dλ) with F (∆λ) = E{|η (∆λ)| } (2) B (s, t) = Λ

In this Section we want to show that a SP ϕ(t) of this kind can be approximated in mean square by neural networks of the kind  ψm um (t) (3) ψ (t) = m

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 177–184, 2003. c Springer-Verlag Berlin Heidelberg 2003


178

Paolo Crippa and Claudio Turchetti

where ψm are random variables and um (t) are AI functions [2]. These networks are named Stochastic Approximate Identity Neural Networks (SAINNs). For this purpose, let be given the SP ϕ (t) admitting the canonical representation  2 (1) where Φ (t, λ) is L2 -summable, i.e. T Λ |Φ (t, λ)| F (dλ) dt < +∞. Let us consider the function  Ψ (t, λ) = am um (t) um (λ) (4) m

where um (·) are AI functions. As it has been shown in [2], the set of functions Ψ (t, λ) is dense in L 2 (Λ × T ). Thus, ∀ ε > 0 it results d (Ψ (t, λ) , Φ (t, λ)) < ε, ∀ t ∈ T , λ ∈ Λ, where d( , ) is the usual distance. As the stochastic measure η (∆) establishes a correspondence U between L 2 (Λ) and L 2 (ϕ), therefore a process ψ (t) corresponds through U to the function Ψ (t, λ) ∈ L 2 (Λ) with canonical representation given by
Ψ (t, λ) η (dλ) . (5) ψ (t) = Λ

Due to the isometry property of U the following equality holds E{|ϕ (t) − ψ (t)|2 }  2 = Λ |Φ (t, λ) − Ψ (t, λ)| F (dλ), ∀ t ∈ T . By taking advantage of the dense prop  2 erty of the functions Ψ (t, λ) we have T Λ |Φ (t, λ) − Ψ (t, λ)| F (dλ)dt → 0 that  2 implies E 2 (t) = Λ |Φ (t, λ) − Ψ (t, λ)| F (dλ) → 0 in all Λ but a set belonging 2 to Λ of null measure, and thus also E{|ϕ (t) − ψ (t)| } → 0. From what showed above we conclude that the SP ψ (t) defined by (5) is able to approximate in mean square the given process ϕ (t) [4]. We can also write

 ψ (t) = am um (t) um (λ) η (dλ) = am um (t) um (λ) η (dλ) . (6) Λ

m

m

Λ

By defining the random variables
ψm = am

Λ

um (λ) η (dλ) ,

(7)

we finally demonstrate that (3) holds. Eq. (3) can be viewed as a stochastic neural network expressed in terms of linear combination of AI functions, through the coefficients um , being um random variables whose statistics depends on the process ϕ (t) to be approximated. It is clear the similarity of (3) to Karhunen-Lo`eve representation, although in K.L. expansion the functions of time are eigenfunctions dependent on the covariance function, while in (3) they are AI functions.

3

Learning of SAINNs from Covariance Function

The process ψ (t) in (3) defines a stochastic neural network that, on the basis of what previously proven, is able to approximate any given process ϕ (t) belonging to the class of SP’s (nonstationary in general) admitting a canonical form. As

Learning of SAINNs from Covariance Function: Historical Learning

179

 the process ψ (t) depends on function Ψ (t, λ) = m am um (t) um (λ), which approximates the function Φ (t, λ), it is essential to know Φ (t, λ) in order to perform the learning of the neural network. To this end it is convenient to treat the case of finite T separately to the case of infinite T . Finite T In this case the set Λ is countable, as it follows from Kahrunen-Lo`eve theory [1, 4], thus the approximating process ψ(t) reduces to:   ψ (t) = ψm um (t) with ψm = am um (λj ) η (λj ) . (8) m

j

Moreover we can write (8) as    ψ (t) = am um (t) um (λj ) η (λj ) . j

m

(9)

Karhunen-Lo`eve theory establishes that ϕ (t) can be expanded as  ϕ (t) = Φ (t, λj ) η (λj ) .

(10)

Comparing (9) and (10), and assuming the approximation  Φ (t, λj ) ≈ am um (t) um (λj )

(11)

j

m

then we have ϕ (t) ≈ ψ (t) .

(12)

More rigorously



2    

 

   

 E |ϕ (t) − ψ (t)|2 = E

am um (t) um (λj ) η (λj )

Φ (t, λj ) − 

  j  m

2

  



= am um (t) um (λj ) E |η (λj )| 2

Φ (t, λj ) −

m j

2





= am um (t) um (λj ) λ2j → 0 , (13)

Φ (t, λj ) −

m

j

where the orthogonality of the stochastic measure η (λj ) and the approximating property of the AINN’s have been adopted. In the case of finite T the functions Φ (t, λj ) are known, since they are the eigenfunctions of the integral operator defined by the covariance function B (s, t). Therefore (13) can be viewed as a learning relationship of the stochastic neural network. The other learning relationship is given by
η (λj ) = ϕ (t) Φ (t, λ j ) dt . (14) T

180

Paolo Crippa and Claudio Turchetti

Thus, in conclusion, in the case of finite T from the knowledge of covariance function B (s, t) (or an estimation of it) we can define the stochastic neural network   ψ (t) = ψm um (t) , ψm = am um (λj ) η (λj ) (15) m

m

approximating the process ϕ (t), through the relationships  am um (t) um (λj )| 2 → 0 i) | Φ (t, λj ) − m
ii ) η (λj ) = ϕ (t) Φ (t, λ j ) dt .

(16)

T

It is worth to note that when ϕ (t) is Gaussian, η (λj ) are independent Gaussian random variables. This case is of particular interest for applications because the statistical behavior of η (λj ) is completely specified by the variance E{|η (λj )|2 }, and thus the eigenvalues λj . As an example of the complexity of the approximating problem let us consider a stationary process ϕ (t) with covariance function B (s, t) = a exp (−a |t − s|)

(17)

on the interval [0, T ]. The eigenvalue equation in this case is


T

a exp (−a |t − s|) Φ (s) ds = λ Φ (t) .

(18)

0

In order to solve this integral equation it is useful to rewrite the integral as the convolution over the entire real axis (−∞, +∞) h (t) ∗ Φ (t) = λ Φ (t)

with h (t) = a exp (−a |t|)

(19)

constraining (19) to the boundary conditions as derived from (18) at the boundaries of the time interval [0,T], so that (19) is identical of (18). By Fourier transform (19) becomes H (jω) · U (jω) = λU (jω)

(20)

where H (jω), U (jω) are the Fourier transforms of h (t), Φ (t) respectively. Since it results   H (jω) = 2a2 a2 + ω 2 , (21) eq. (20) reduces to   2  2 2a a + ω 2 U (jω) = λU (jω)

(22)

and the eigenvalues λ are given by λ = 2a2

 2  a + ω2 .

(23)

Learning of SAINNs from Covariance Function: Historical Learning

181

Applying the inverse Fourier transform to (22), a second-order differential equation results Φ (t) + [(2 − λ) /λ] a2 Φ (t) = 0 (24) Solving this equation requires two boundary conditions on Φ (t) and its derivative Φ (t) that can be easily obtained from (18): Φ (T ) /Φ (T ) = −1/a and Φ (0) /Φ (0) = 1/a. The general solution of (24) is given by Φ (t) = A sin (ωt + α)

(25)

and the above boundary conditions take the form tan (ωT + α) = −ω/a and tan α = ω/a, respectively. After some manipulations these boundary conditions reduce to a unique equation   tan (ωk T ) = −2aωk a2 + ωk (26) where the index k means that a countable set of values of ωk satisfying (26) exists. Eq. (23) establishes that a corresponding set of values λk exists, given by ∆λk = 2a2

 2  a + ωk2

, k = 1, 2, . . .

(27)

thus all the eigenfunctions are expressed as Φk (t) = Ak sin (ωk t + αk )

(28)

where the constants Ak , αk are determined from orthonormalization conditions
T Φk (t) Φl (t) dt = δkl . (29) 0

As for as the stochastic neural network is concerned, (16) apply in this case where Φ is given by (28). Infinite T In this case the set Λ is not countable, and the problem of determining a complete set of functions Φ (t, λ) falls in the theory of operators whose eigenvalues form a noncountable set. Since this subject is out of the scope of this work, we restrict the treatment of this case to a specific class of random processes. Let us assume Φ (t, λ) = a (t) exp (iλt)

(30)

where a (t) is an arbitrary function (such that ∀ λ ∈ Λ, Φ (t, λ) is L2 -integrable), then the set of functions {Φ (t, λ)}, yielded by varying the parameter t ∈ T , is a complete family in L2 (F ). Any linear combination of Φ (t, λ) can be written as    bk Φ (tk , λ) = bk a (tk ) exp (iλtk ) = ck exp (iλtk ) . (31) k

k

k

Yet from the theory of Fourier integral it is known that the set {exp (iλt)} is complete in L2 (F ), meaning that linear combinations of such kind generate the

182

Paolo Crippa and Claudio Turchetti

 entire space L2 (F ). As a consequence also the set { k bk Φ (tk , λ)} has the same property of generating the space L2 (F ), which is equivalent to say that, given  any function f (λ) ∈ L2 (F ), it results f (λ) = limN →∞ N k=1 bk Φ (tk , λ). The covariance function of the process ϕ (t) can be written as

B (s, t) = Φ (t, λ) Φ (s, λ)F (dλ) = a (t) a (s) exp [iλ (t − s)] F (dλ) (32) Λ

Λ

and the canonical representation (1) reduces to ϕ (t) = a (t) ζ (t)

(33)




where ζ (t) =

exp (iλt) η (dλ)

(34)

Λ

is a stationary SP since the canonical representation holds for it. Thus we conclude that the canonical representation in (1), with Φ (t, λ) given by (30), is valid for the class of nonstationary processes which can be expressed in the form ϕ (t) = a (t) ζ (t), being a (t) an arbitrary function and ζ (t) a stationary process. From (1) it can be shown that the stochastic measure η (∆λ), with ∆λ = [λ1 , λ2 ) can be written as
η (∆λ) = ϕ (t) ξ∆λ (t) dt (35) T

where ξ∆λ (t) is defined through the integral equation 
1 ξ∆λ (t) Φ (t, λ) dt = χ∆λ (λ) with χ∆λ (λ) = 0 T

if λ ∈ ∆λ . if λ ∈ / ∆λ

(36)

As far as the inversion formula (35) is concerned, by inserting Φ (t, λ), given by (30), in (36) and performing the Fourier inverse transform, it results ξ∆λ (t) a (t) =

exp(−iλ2 t)−exp(−iλ1 t) . −2πit

Finally from (35) and (37) we have

exp(−iλ2 t)−exp(−iλ1 t) ζ (t) dt . η (∆λ) = ζ (t) a (t) ξ∆λ (t) dt = −2πit T T

(37)

(38)

Eq. (38) establishes that the stochastic measure η (∆λ) is only dependent on the stationary component ζ (t) of the process ϕ (t) = a (t) ζ (t). The measure F (∆λ) can be derived from general equation (35)

F (∆λ) = E{|η (∆λ)|2 } = ξ∆λ (t) ξ∆λ (s) B (s, t) dt ds T T


= ξ∆λ (t) ξ∆λ (s)a (t) a (s) exp [iλ (t − s)] F (dλ)dtds (39) T

T

Λ

Learning of SAINNs from Covariance Function: Historical Learning

183

which, by taking into account of (37), reduces to F (∆λ) =  

 exp(−iλ2 t)−exp(−iλ1 t) exp(−iλ2 s)−exp(−iλ1 s) Bζ (t−s)dtds(40) −2πit −2πis T T where Bζ (t − s) is the covariance function of ζ (t) and is defined by B (s, t) = a (t) a (s)Bζ (t − s) .

(41)

Coming back to the problem of approximating the SP ϕ (t) = a (t) ζ (t), it is equivalent to approximating the resulting Φ (t, λ) = a (t) eiλt by the AINN, i.e.  am um (t) um (λ) . (42) a (t) eiλt ≈ m

Indeed the mean square error between the SP ϕ (t) and the approximating process
 ψ (t) = Ψ (t, λ) η (dλ) with Ψ (t, λ) = am um (t) um (λ) , (43) Λ

m

is given by    

2   E |ϕ (t) − ψ (t)|2 = E Λ Φ (t, λ) η (dλ) − Λ Ψ (t, λ) η (dλ)  2 = Λ |Φ (t, λ) − g (t, λ)| F (dλ)   = Λ |a (t) eiλt − m am um (t) um (λ)|2 F (dλ) → 0

(44)

The stochastic neural network is defined by (44), with η (∆λ) given by (38). If the process ϕ (t) is Gaussian the random variables η (∆λ) are independent and 2 completely specified by the variance E{|η (∆λ)| } = F (∆λ) so that Bζ (t − s) completely characterizes the process as it follows from (40), (41).

4

Conclusions

In this work the historical learning, i.e. the SAINNs capability of approximating a wide class of nonstationary stochastic processes from their covariance function has been demonstrated. In particular the two cases where the ‘time’ t belongs to a finite or infinite set T have been analyzed separately.

References [1] Karhunen, K.: Uber lineare Methoden in der Wahrscheinlicherechnung, Ann. Acad. Sci. Fennicae, Ser. A. Math Phys., Vol. 37, (1947) 3–79 177, 179 [2] Turchetti, C., Conti, M., Crippa, P., Orcioni, S.: On the Approximation of Stochastic Processes by Approximate Identity Neural Networks. IEEE Trans. Neural Networks, Vol. 9, 6 (1998) 1069–1085 177, 178

184

Paolo Crippa and Claudio Turchetti

[3] Belli, M. R., Conti, M., Crippa, P., Turchetti, C.: Artificial Neural Networks as Approximators of Stochastic Processes. Neural Networks, vol. 12, 4-5 (1999) 647– 658 177 [4] Doob, J. L.: Stochastic Processes. J. Wiley & Sons, New York, USA, (1990) 178, 179

Use of the Kolmogorov's Superposition Theorem and Cubic Splines for Efficient Neural-Network Modeling Boris Igelnik Pegasus Technologies, Inc. 5970 Heisley Road, Suite 300, Mentor OH 44060, USA [email protected]

Abstract. In this article an innovative neural-network architecture, called the Kolmogorov's Spline Network (KSN) and based on the Kolmogorov's Superposition Theorem and cubic splines, is proposed and elucidated. The main result is the Theorem giving the bound on the approximation error and the number of adjustable parameters, which favorably compares KSN with other one-hidden layer feed-forward neural-network architectures. The sketch of the proof is presented. The implementation of the KSN is discussed.

1

Introduction

The Kolmogorov's Superposition Theorem (KST) gives the general and very parsimonious representation of a multivariate continuous function through superposition and addition of univariate functions [1]. The KST states that any function, f ∈ C ( I d ) , has the following exact representation on I d f ( x) =

  ∑ g ∑ λψ ( x )

2 d +1

d

n =1

i =1

i

ni

i

(1)

with some continuous univariate function g depending on f, while univariate functions, ψ n , and constants, λi , are independent of f. Hecht-Nielsen [2] was first who recognized that the KST could be utilized in neural network computing. Using early Sprecher's enhancement of the KST [3] he proved that the Kolmogorov's superpositions could be interpreted as a four-layer feedforward neural network. Girosi and Poggio [4] pointed out that the KSN is irrelevant to neural-network computing, because of high complexity of computation of the functions g and ψ n from the finite set of data. However Kurkova [5] noticed that in the Kolmogorov's proof of the KST the fixed number of basis function, 2d + 1, can be replaced by a variable N, and the task of function representation can be replaced by the task of function approximation. She also demonstrated [6] how to approximate Hecht-Nielsen's network by the traditional neural network.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 184-190, 2003.  Springer-Verlag Berlin Heidelberg 2003

Use of the Kolmogorov’s Superposition Theorem and Cubic Splines

185

Numerical implementation of the Kolmogorov's superpositions was analyzed in [7]-[10]. In particular Sprecher [9], [10] derived the elegant numerical algorithms for implementation of both internal and external univariate functions in [1]. The common starting point of the works [2], [3], [5]-[10] is the structure of the Kolmogorov's superpositions. Then the main effort is made: to represent the Kolmogorov's superpositions as a neural network [2], [3]; to approximate them by a neural network [5], [6]; to directly approximate the external and internal univariate functions in the Kolmogorov's representation by a numerical algorithm [7]-[10]. In our view it is an attempt to preserve the efficiency of the Kolmogorov's theorem in representation of a multivariate continuous function in its practical implementation. Especially attractive is the idea of preserving the universal character of internal functions. If implemented with reasonable complexity of the suggested algorithms this feature can make a breakthrough in building efficient approximations. However, since the estimates of the complexity of these algorithms are not available so far, the arguments of Girosi and Poggio [4] are not yet refuted. The approach adopted in this paper is different. The starting point is the function approximation from the finite set of data, by a neural net of the following type N d  f N ,W ( x ) = ∑ an g n  ∑ψ i ( xi , wni )  , W = {an , wni } n =1  i =1 

(2)

The function, f, to be approximated belongs to the class, Φ ( I d ) of continuously differentiable functions with bounded gradient, which is wide enough for applications. We are looking for the qualitative improvement of the approximation f ≈ f N ,W , f ∈ Φ ( I d )

(3)

using some ideas of the KST proof. In particular we see from the Kolmogorov's proof that it is important to vary, dependent on data, the shape of external univariate function, g, in contrast to traditional neural networks with fixed-shape basis functions. Use of variable-shape basis functions in neural networks is described in [11]-[15]. In this paper innovative neural-network architecture, KSN, is introduced. The distinctive features of this architecture are: it is obtained from (1) by replacing fixed number of basis functions, 2d + 1, by the variable N, and by replacing both the external function, and the internal functions, ψ n , by the cubic spline functions, s (., γ n ) and s (., γ ni ) respectively. Use of cubic splines allows to vary the shape of basis functions in the KSN by adjusting the spline parameters γ n and γ ni . Thus the KSN, f Ns ,W , is defined as follows N  d  f Ns ,W ( x ) = ∑ s  ∑ λi s ( xi , γ ni ) , γ n  n =1  i =1 

(4)

where λ1 ,...λd > 0 are rationally independent numbers [16] satisfying the condition

∑

d i =1

λi ≤ 1 . These numbers are not adjustable on the data and are independent of an

application. In Section 2 we give a sketch of the proof for the main result of this

186

Boris Igelnik

paper. It states that the rate of convergence of approximation error to zero with N → ∞ is significantly higher for the KSN than the corresponding nets for traditional neural networks described by equation (2). Simultaneously we proved that the complexity of the approximation of KSN with N → ∞ tends to infinity significantly slower than the complexity of the traditional neural networks. The complexity is defined as the number of adjustable parameters needed to achieve the same approximation error for both types of neural networks. This main result is the justification for introducing the KSN. Thus, utilizing some of the ideas of the KST proof, a significant gain, both in accelerating rate of convergence of approximation error to zero and reducing the complexity of the approximation algorithm can be achieved by the KSN. Section 3 describes some future work in this direction.

2

Main Result

Main result is contained in the following theorem: Theorem. For any function f ∈ Φ ( I d ) and any natural N there exists a KSN defined by equation (4), with the cubic spline univariate functions g ns ,ψ nis 1, defined on [0,1] and rationally independent numbers λ1 > 0,...λd > 0, ∑ i =1 λi ≤ 1 , such that i=d

f − f Ns ,W = Ο (1/ N ) ,

(5)

f − f Ns ,W = max f ( x ) − f Ns ,W ( x ) . d

(6)

where x∈ I

The number of network parameters, P, satisfies the equation P = Ο( N 3/ 2 ) .

(7)

This statement favorably compares the KSN with the one-hidden layer feedforward networks currently in use. Most of the existing such networks provides the estimate of approximation [17]-[22] error by the following equation

(

)

f − f N ,W = Ο 1/ N .

(8)

Denote N∗ and P∗ the number of basis functions and the number of adjustable parameters for a network currently in use with the same approximation error as the KSN with N basis functions. Comparison of (5) and (8) shows that N∗ = Ο ( N 2 ) and P∗ = Ο ( N 2 ) . Therefore, P∗ >> P for large values of N, which confirms the significant

advantage of the KSN in complexity. 1

These notations are used in section 2 instead of notations s (., γ n ) , s (., γ ni ) in formula (4)

Use of the Kolmogorov’s Superposition Theorem and Cubic Splines

187

The detailed proof of the theorem is contained in [23]. The proof is rather technical and lengthy. In particular large efforts were spent on proving that the conditions imposed on the function f are sufficient for justification of equations (5) and (7) In this paper we concentrate on the description of the major steps of the proof. The first step of the proof follows the arguments of the first step of the KST proof, except two things: it considers variable number of basis functions, N, and differently defines the internal functions, ψ i in formula (2). In the latter case piecewise linear functions of the Kolmogorov's construction are replaced by almost everywhere piecewise linear functions with connections between adjacent linear parts made from nine degree spline interpolates on the arbitrarily small intervals. As a result of this first step a network, f N ,W , described by equation (2) is obtained, satisfying the equation f − f N ,W = Ο (1/ N ) .

(9)

The univariate functions, g n and ψ i , in the first step of the proof have uncontrolled complexity. That is why they are replaced in the second step by their cubic spline interpolates g ns and ψ nis . The number of spline knots, M, depends on N, and was chosen so that the spline network, f Ns ,W , defined as N d  s f Ns ,W ( x ) = ∑ g n  ∑ λψ i ni ( xi , γ ni ) , γ n  n =1  i =1 

(10)

approximates f with approximation error and complexity satisfying equations (5) and (7). The functions, g n and ψ i , defined in the first step of proof, should be four times continuously differentiable in order to be replaced by their cubic spline interpolates, g ns and ψ nis respectively, with estimated approximation error. That is why in the construction of g n and ψ i nine degree spline interpolates should be used.

3

Implementation of the KSN

In the implementation of the KSN we use the Ensemble Approach (EA) [24], [25]. EA contains two optimization procedures, the Recursive Linear Regression (RLR) [26] and the Adaptive Stochastic Optimization (ASO), and one module of the specific neural net architecture (NNA). Both RLR and ASO operate on the values of the T × N design matrices defined for N = 1,...N max

(

 Γ1 x (1) , w1N   ...   Γ1 x (T ) , w1N 

(

) )

(

)

1 ... Γ N x ( ) , wNN    , ... ...  T ... Γ N x ( ) , wNN  

(

)

(11)

188

Boris Igelnik

and

the

matrix

of

the

target

output

values

 y (1) ,... y (T )  ,  

where

d  t (t ) s Γ n x ( ) , wnN = g ns  ∑ λψ i ni xi , γ ni , γ n  , n = 1,...N , t = 1,...T , T is the number of  i =1  points in the training set. Since these matrices are numerical, operation of ASO and RLR is independent of the specifics of NNA. The NNA for spline functions is defined in detail in [23]. The use of the EA can be supplemented by the use of the Ensemble Multi-Net (EMN) [27]-[29] in order to increase the generalization capability of the KSN (it can be used as well for training of other predictors with the same purpose). The main features of the EMN are as follows. First the ensemble of the nets

(

{f

)

n ,Wn , En

(

)

}

, n = 1,...N , having N 0 basis functions and trained on different subsets,

En , n = 1,...N , of the training set E, is created. Then these nets are combined in one

net, FN ,W , E , trained (if needed) on the set E. We prefer the method of linear combining FN ,W , E ( x ) = ∑ an f n,W∗ N , En ( x )

(12)

where f n,W* N , En , n = 1,...N are the trained nets. The parameters an , n = 1,...N are the only adjustable parameters on the right side of equation (12). The training sets are chosen so that to minimize the possibility of linear dependences among basis functions f n,W* N , En , n = 1,...N . One of the methods used for creation of training sets is called bagging [29]. For each ( x, y ) ∈ E the pair ( x, y + N ( 0, σ% ) ) is included in En with the probability 1/ E , where N ( 0, σ% ) is a Gaussian random variable with the zero mean and the standard deviation σ% equal to the estimate of the standard deviation of the net output noise. Inclusions in En are independent of inclusions in other sets, Em , m ≠ n . The number of basis functions, N 0 , in each net f n,W* N , En , n = 1,...N is small [27], [28] compared to N. It was proven [29], both theoretically and experimentally, that this method might lead to significant decrease of generalization error. Given relatively small size of the nets f n,W* N , En , n = 1,...N the EMN can be called the “parsimonious cross-validation”.

4

Conclusion and Future Work

In this article we have laid down the justification for a new, highly adaptive modeling architecture, the KSN, and developed the methods of its implementation and learning. Several theoretical issues remain for future work. In particular we are interested in deriving a bound on the estimation error [2] of the KSN and finding the methods for automatic choosing the intervals for the shape parameters of the KSN. Not less

Use of the Kolmogorov’s Superposition Theorem and Cubic Splines

189

important is the experimental work where not only the practical advantages and disadvantages of the KSN as a modeling tool will be checked but the KSN coupling with the preprocessing and post-processing tools.

References [1]

[2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]

[16]

Kolmogorov, A.N.: On the Representation of Continuous Functions of Many Variables by Superposition of Continuous Functions of One Variable and Addition. Dokl. Akad. Nauk SSSR 114 (1957) 953-956, Trans. Amer. Math. Soc. 2 (28) (1963) 55-59 Hecht-Nielsen, R.: Counter Propagation Networks. Proc. IEEE Int. Conf. Neural Networks. 2 (1987) 19-32 Sprecher, D.A.: On the Structure of Continuous Functions of Several Variables. Trans. Amer. Math. Soc. 115 (1965) 340-355 Girosi F., Poggio, T.: Representation Properties of Networks: Kolmogorov's Theorem Is Irrelevant. Neur.Comp. 1 (1989) 465-469 Kurkova, V.: Kolmogorov's Theorem Is Relevant. Neur.Comp. 3 (1991) 617622 Kurkova, V.: Kolmogorov's Theorem and Multilayer Neural Networks. Neural Networks. 5 (1992) 501-506 Nakamura, M., Mines, R., and Kreinovich, V.: Guaranteed Intervals for Kolmogorov's Theorem (and their Possible Relations to Neural Networks) Interval Comp. 3 (1993) 183-199 Nees, M.: Approximative Versions of Kolmogorov's Superposition Theorem, Proved Constructively. J. Comp. Appl. Math. 54 (1994) 239-250 Sprecher, D.A.: A Numerical Implementation of Kolmogorov's Superpositions. Neural Networks. 9 (1996) 765-772 Sprecher, D.A.: A Numerical Implementation of Kolmogorov's Superpositions. Neural Networks II. 10 (1997) 447-457 Guarnieri, S., Piazza, F., and Uncini, A.: Multilayer Neural Networks with Adaptive Spline-based Activation Function. Proc. Int. Neural Network Soc. Annu. Meet. (1995) 1695-1699 Vecci, L., Piazza, F., and Uncini, A.: Learning and Generalization Capabilities of Adaptive Spline Activation Function Neural Networks. Neural Networks. 11 (1998) 259-270 Uncini, A., Vecci, L., Campolucci, P., and Piazza, F.: Complex-valued Neural Networks with Adaptive Spline Activation Function for Digital Radio Links Nonlinear Equalization. IEEE Trans. Signal Proc. 47 (1999) 505-514 Guarnieri, S., Piazza, F., and Uncini, A.: Multilayer Feedforward Networks with Adaptive Spline Activation Function. IEEE Trans. Neural Networks. 10 (1999) 672-683 Igelnik, B.: Some New Adaptive Architectures for Learning, Generalization, and Visualization of Multivariate Data. In: Sincak, P., Vascak, J. (eds.): Quo Vadis Computational Intelligence? New Trends and Approaches in Computational Intelligence. Physica-Verlag, Heidelberg (2000) 63-78 Shidlovskii, A.V.: Transcendental Numbers. Walter de Gruyter, Berlin (1989)

190

Boris Igelnik

[17] Barron, A.R., Universal Approximation Bounds for Superpositions of a Sigmoidal Function. IEEE Trans. Inform. Theory. 39 (1993) 930-945 [18] Breiman, L.: Hinging Hyperplanes for Regression, Classification, and Function Approximation. IEEE Trans. Inform. Theory. 39 (1993) 999-1013 [19] Jones, L.K.: Good Weights and Hyperbolic Kernels for Neural Networks, Projection Pursuit, and Pattern Classification: Fourier Strategies for Extracting Information from High-dimensional Data. IEEE Trans. Inform. Theory. 40 (1994) 439-454 [20] Makovoz, Y.: Random Approximants and Neural Networks. Jour. Approx. Theory. 85 (1996) 98-109 [21] Scarcelli, F and Tsoi, A.C.: Universal Approximation Using Feedforward Neural Networks: a Survey of Some Existing Methods and Some New Results. Neural Networks. 11 (1998) 15-37 [22] Townsend, N. W. and Tarassenko, L.: Estimating of Error Bounds for NeuralNetwork Function Approximators. IEEE Trans. Neural Networks. 10 (1999) 217-230 [23] Igelnik, B., Parikh, N.: Kolmogorov's Spline Network. IEEE Trans. Neural Networks. (2003) Accepted for publication [24] Igelnik, B., Pao, Y.-H., LeClair, S.R., and Chen, C. Y.: The Ensemble Approach to Neural Net Training and Generalization. IEEE Trans. Neural Networks. 10 (1999) 19-30 [25] Igelnik, B., Tabib-Azar, M., and LeClair, S.R.: A Net with Complex Coefficients. IEEE Trans. Neural Networks. 12 (2001) 236-249 [26] Albert, A.: Regression and the Moore-Penrose Pseudoinverse. Academic Press, New York (1972) [27] Shapire, R. E.: The Strength of Weak Learnability. Machine Learning. 5 (1990) 197-227 [28] Ji, S and Ma, S.: Combinations of Weak Classifiers. IEEE Trans. Neural Networks. 8 (1997) 32-42 [29] Breiman, L.: Combining Predictors. In: Sharkey, A.J.C. (ed.): Combining Artificial Neural Nets. Ensemble and Modular Nets. Springer, London (1999) 31-48

The Influence of Prior Knowledge and Related Experience on Generalisation Performance in Connectionist Networks F.M. Richardson 1,2, N. Davey1, L. Peters1, D.J. Done2, and S.H. Anthony2 1

Department of Computer Science, University of Hertfordshire, College Lane, Hatfield, Hertfordshire, AL10 9AB, UK {F.1.Richardson, N.Davey, L.Peters}@herts.ac.uk 2 Department of Psychology, University of Hertfordshire, College Lane, Hatfield, Hertfordshire, AL10 9AB, UK {D.J.Done, S.H.1.Anthony}@herts.ac.uk

Abstract. The work outlined in this paper explores the influence of prior knowledge and related experience (held in the form of weights) on the generalisation performance of connectionist models. Networks were trained on simple classification and associated tasks. Results regarding the transfer of related experience between networks trained using backpropagation and recurrent networks performing sequence production, are reported. In terms of prior knowledge, results demonstrate that experienced networks produced their most pronounced generalisation performance advantage over navï e networks when a specific point of difficulty during learning was identified and an incremental training strategy applied at this point. Interestingly, the second set of results showed that knowledge learnt about in one task could be used to facilitate learning of a different but related task. However, in the third experiment, when the network architecture was changed, prior knowledge did not provide any advantage and indeed when learning was expanded, even found to deteriorated.

1

Introduction

Some complex tasks are difficult for neural networks to learn. In such circumstances an incremental learning approach, which places initial restrictions on the network in terms of memory or complexity of training data has been shown to improve learning [1], [2], [3]. However, the purpose of the majority of networks is not simply to learn the training data but to generalise to unseen data. Therefore, it can be expected, but not assumed, that using incremental learning may also improve generalisation performance. The work reported in this paper extends upon the original work of Elman [3], in which networks trained incrementally showed a dramatic improvement in learning. In this paper three different ways of breaking down the complexity of the task are V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 191-198, 2003.  Springer-Verlag Berlin Heidelberg 2003

192

F.M. Richardson et al.

investigated, with specific reference to generalisation performance. In the first experiment an incremental training regime, in which the training set is gradually increased in size, is evaluated against the standard method of presenting the complete training set. In the second experiment the hypothesis that knowledge held in the weights of a network from one task might be useful to a network learning a related task, is explored. The third experiment completes the investigation by determining whether knowledge transfer between networks such as those in the second experiment, may prove beneficial to learning across different types of networks, performing related tasks. The use of well-known learning rules (back-propagation and recurrent learning) allows this work to complement previously mentioned earlier work, allowing a more complete picture of learning and generalisation performance.

2

Experiment One: Investigating Prior Knowledge

The aim of this experiment was to compare the generalisation performance of networks trained using incremental learning in which the input to the network is staged (experienced networks), with equivalent networks initialised with random weights (navï e networks). Networks were trained with the task of classifying static depictions of four simple line-types as seen in Figure 1. Classification of the line-type was to be made irrespective of its size and location on the input array. In order to accomplish this task the network, must learn the spatial relationship of units in the input and then give the appropriate output in the form of the activation of a single classification unit on the output layer.

Horizontal

Vertical

Diagonal acute

Diagonal obtuse

Fig. 1. Shows the four basic line-types which the network was required to classify

2.1

Network Architecture

A simple feed-forward network consisting of an input layer of 49 units arranged as a 7x7 grid was fully connected to a hidden layer of 8 units, which was also connected to an output layer of 4 units, each unit representing a single line type was used (see Figure 2). 2.2

Training and Testing of Naïve Networks

A full set of patterns for all simple line-types of lengths ranging from 3 to 7 units, for all locations upon the input grid, were randomly allocated to one of two equal-sized training and testing sets (160 patterns per set). Three batches of networks (each consisting of 10 runs) were initialised with random weights (navï e networks). The first batch of networks was trained to classify two different line-types (horizontal and vertical lines), the second three and the third, all four different line-types. All

The Influence of Prior Knowledge and Related Experience

193

networks were trained using back-propagation with the same learning rate (0.25) and momentum (0.9) to the point where minimal generalisation error was reached. At this stage the number of epochs that each network had taken to reach the stop-criterion of minimal generalisation error was noted. These networks formed the basis of comparison with experienced networks. 2.3

Training and Testing of Experienced Networks

These networks were trained using the same parameters as those used for navï e networks. The training and testing of the four line-types was divided into three increments, the first increment consisting of two line-types (horizontal and vertical lines) with an additional line-type being added at each subsequent increment. The network progressed from one increment to the next upon attaining minimal generalisation error for the current increment. At this point the weights of the network were saved and then used as the starting point for learning in the following increment, patterns for the additional line-type were added along with an additional output unit (the weights for the additional unit were randomly initialised). O u tp u t lay er

H id den la y er

Inp u t lay er

Fig. 2. Shows the network architecture of the 7x7-classification network. The network consisted of an input layer with units arranged as a grid, a hidden layer, and an output layer consisting of a number of classification units. Given a static visual depiction of a line of any length as input, the network was required to classify the input accordingly by activating the corresponding unit on the output layer. In the example shown the network is given the input of a vertical line with a length of 3 units, which is classified as a vertical line-type

2.4

Results

Generalisation performance was assessed in terms of the number of output errors and poor activations produced by each type of network. Outputs were considered errors if activation of the target unit was less than 0.50, or if a non-target unit had an activation of 0.50 or higher. Poor activations were marginally correct classifications (activation of between 0.50-0.60 on the target unit, and/or activation of 0.40-0.49 on non-target units) and were used to give a more detailed indicator of the level of classification success.

194

F.M. Richardson et al.

The generalisation performance of navï e networks trained on two line-types was good, with networks classifying on average, 89.84% of previously unseen patterns correctly. However, performance decreased as the number of different line-types in the training set increased, with classification performance for all four line-types dropping to an average of 79.45%. In comparison, experienced networks proved marginally better, with an average of 81.76%. Further comparisons between navï e and expe rienced networks revealed that a navï e network trained upon three line-types produced a better generalisation performance than an experienced network at the same stage. It seemed that the level of task difficulty increased between the learning of three and four line-types. So the weights from navï e networks trained with three lin e-types were used as a starting point for training further networks upon four line-types, resulting in a two-stage incremental strategy. This training regime resulted in a further improvement in generalisation performance, with an average of 84.48%. A comparison of the results for the three different training strategies implemented can be seen in Figure 3.

Fig. 3. Shows a comparison of generalisation performance between networks trained using the three different strategies. It can be seen that naïve networks produced th e lowest generalisation success. Of the experienced networks, those trained using the two-stage strategy produced the best generalisation performance

3

Experiment Two: Investigating Related Experience I

The aim of this experiment was to attempt to determine whether the knowledge acquired by networks trained to classify line-types in Experiment One would aid generalisation performance of networks trained upon different but related task. In the new task, networks were given the same type of input patterns as those used for the classification network, but were required to produce a displacement vector as output. This displacement vector contained information as to the length of the line and the direction in which the line would be drawn if produced (see Figure 4). It was hypothesised that the knowledge held in the weights from the previous network would aid learning in the related task because the task of determining how to

The Influence of Prior Knowledge and Related Experience

195

interpret the input layer had already been solved by the previous set of weights. The divisions between line-types created upon output in the classification task were also relevant to the related task, in that same line-types shared activation properties upon the output layer, for example all vertical lines are the result of activation upon the yaxis. 3.1

Network Architecture

All networks consisted of the same input layer as used in Experiment One, a hidden layer of 12 units and an output layer of 26 units. -

+ y x

Fig. 4. Shows the static input to the network of a vertical line of a length of five units and the desired output of the network. Output is encoded in terms of movement from the starting point along the x and y-axis. This form of thermometer encoding preserves issometry and is position invariant [3]

3.2

Training and Testing

All networks were trained and tested in the same manner and using the same parameters as those used in Experiment One. Navï e networks were initialised with a random set of weights. For experienced networks, weights from networks producing the best generalisation performance in Experiment One were loaded. Additional connections required for these networks (due to an increase in the number of hidden units) were initialised randomly. Generalisation performance was assessed. 3.3

Results

Navï e networks reached an average generalisation performance of 59.59%. Experienced networks were substantially better, with an average generalisation performance of 70.42%. This result as seen in Figure 5, clearly demonstrates the advantage of related knowledge about lines and the spatial relationship of units on the input grid in the production of displacement vectors.

4

Experiment Three: Investigating Related Experience II

The aim of this experiment was to examine whether weights from a classification task could be used to aid learning in a recurrent network required to carry out an extension of the static task. Initially, a standard feed-forward network (as shown in Figure 2) was trained to classify line-types of simple two-line shapes. Following this a recurrent

196

F.M. Richardson et al.

network [5] was trained to carry out this task in addition to generating the sequence in which the line-segments for each shape would be produced if drawn (as shown in Figure 6).

Fig. 5. Shows a comparison of generalisation performance between naïve and experienced networks. It is clear that the experienced networks produced the best generalisation performance

static

t1

sequential static

t2

sequential Input

Output

Fig. 6. Shows an example of the input and output for the sequential shape classification task. The input was a simple shape composed of two lines of different types. The output consisted of two components. The static, identifying the line-types, and the sequential generating the order of production. Networks used in Experiment One were trained to give the static output of the task. Weights from these networks were then used by recurrent networks to produce both the static and sequential outputs as shown above

4.1

Training and Testing

Both navï e and experienced networks were trained on an initial sequence production task, using simple shapes composed of diagonal line-types only. Following this, the initial task was extended for both networks to include shapes composed of horizontal and vertical line-types. 4.2

Results

The generalisation performance of navï e and experienced networks for the initial and extended task was assessed. Initially, comparison showed no notable difference

The Influence of Prior Knowledge and Related Experience

197

between navï e and experienced networks. However, performance deteriorated for experienced networks with the addition of the extended task. This drop in performance was attributed to a reduction in performance upon the initial task.

Fig. 7. Shows a comparison of generalisation performance between naïve and experienced networks, performing the initial sequence production task followed by the extended task. It can be seen that performance for the two types of networks for the initial task is relatively equal. However, for the extended task, performance of the experienced networks is poor in comparison to naïve networks trained with random weights

5

Discussion

The experiments conducted have provided useful insights into how prior knowledge and related experience may be used to improve the generalisation performance of connectionist networks. Firstly, it has been demonstrated that incremental learning was of a notable benefit. Secondly, by selecting the point at which learning becomes difficult as the time to increment the training set produces a further advantage. Thirdly, an interesting result is that knowledge learnt about in one task can be used to facilitate learning in a different but related task. Finally, the exploration into whether knowledge can transfer and aid learning between networks of different architectures has a less clear outcome; prior knowledge was successfully transferred, but was found to deteriorate as the network attempted to expand its learning. Further work involves exploring methods by which knowledge transfer between static and recurrent networks may prove beneficial in both learning and generalisation performance.

References [1] [2]

Altmann, G.T.M.: Learning and Development in Connectionist Learning. Cognition (2002) 85, B43-B50 Clarke, A.: Representational Trajectories in Connectionist Learning. Minds and Machines (1994) 4, 317-322

198

F.M. Richardson et al.

[3]

Elman, J.L.: Learning and Development in Neural Networks: the Importance of Starting Small. Cognition. (1993) 48, 71-99 Richardson, F.M., Davey, N., Peters, L., Done, D.J., Anthony, S.H.: Connectionist Models Investigating Representations Formed in the Sequential Generation of Characters. Proceedings of the 10th European Symposium on Artificial Neural Networks. D-side publications, Belgium (2002) 83-88 Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Internal Representations by Error Propagation. In: Parallel Distributed Processing. Vol. 1. Chapter 8. MIT Press, Cambridge (1986)

[4]

[5]

Urinary Bladder Tumor Grade Diagnosis Using On-line Trained Neural Networks D.K. Tasoulis1,2 , P. Spyridonos3 , N.G. Pavlidis1,2 , D. Cavouras4, P. Ravazoula5 , G. Nikiforidis3 , and M.N. Vrahatis1,2 1

Department of Mathematics, University of Patras GR–26110 Patras, Greece [email protected] 2 University of Patras Artificial Intelligence Research Center (UPAIRC) 3 Computer Laboratory, School of Medicine University of Patras GR–26110 Patras, Greece 4 Department of Medical Instrumentation Technology, TEI of Athens Ag. Spyridonos Street Aigaleo, GR–12210 Athens, Greece 5 Department of Pathology, University Hospital GR–26110 Patras, Greece

Abstract. This paper extends the line of research that considers the application of Artificial Neural Networks (ANNs) as an automated system, for the assignment of tumors grade. One hundred twenty nine cases were classified according to the WHO grading system by experienced pathologists in three classes: Grade I, Grade II and Grade III. 36 morphological and textural, cell nuclei features represented each case. These features were used as an input to the ANN classifier, which was trained using a novel stochastic training algorithm, namely, the Adaptive Stochastic On-Line method. The resulting automated classification system achieved classification accuracy of 90%, 94.9% and 97.3% for tumors of Grade I, II and III respectively.

1

Introduction

Bladder cancer is the fifth most common type of cancer. Superficial Transitional cell carcinoma (TCC) is the most frequent histological type of bladder cancer [13]. Currently, these tumors are assessed using a grading system based on a variety of histopathological characteristics. Tumor grade, which is determined by the pathologist from tissue biopsy, is associated with tumor aggressiveness. The most widely accepted grading system is the WHO (World Health Organization) system, which stratifies TCCs into three categories: tumors of Grade I, Grade II and Grade III. Grade I tumors are not associated with invasion or metastasis but present a risk for the development of recurrent lesions. Grade II carcinomas are associated with low risk of further progression, yet they frequent recur. Grade III tumors are characterized by a much higher risk of progression


Corresponding author.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 199–206, 2003. c Springer-Verlag Berlin Heidelberg 2003


200

D.K. Tasoulis et al.

and also high risk of association with disease invasion [7]. Although histological grade has been recognized as one of the most powerful predictors of the biological behavior of tumors and significantly affects patients’ management, it suffers from low inter and intra observer reproducibility due to the subjectivity inherent to visual observation [12]. Digital image analysis techniques and classification systems constitute alternative means to perform tumor grading in a less subjective manner. Numerous research groups have proposed quantitative assessments to address this problem. H-K Choi et al [3] have developed an automatic grading system using texture features on a large region of interest, covering a typical area in the histological section. The textural based system produced an overall accuracy of 84.3% in assessing tumors grade. In a different study [6], researchers have employed tissue architectural features and classified tumors with an accuracy of 73%. More recent studies have focused on the analysis of cell nuclei characteristics to perform tumor grade classification with success rates that do not significantly exceed 80% [2, 17]. In this study, we present a methodology which improves considerably the level of diagnostic accuracy in assigning tumor grade. The method is based on the application of an ANN as a classifier system. The input data for the ANN, describe a number of nuclear morphological and textural features, that were obtained through an automatic image processing analysis technique. It is worth noting that the prognostic and diagnostic value of these features, has been confirmed [4].

2

Materials and Methods

129 tissue sections (slides) from 129 patients (cases) with superficial TCC were retrieved from the archives of the Department of Pathology of Patras University Hospital in Greece. Tissue sections were routinely stained with HaematoxylinEosin. All cases were reviewed independently by the experts to safeguard reproducibility. Cases were classified following the WHO grading system as follows: thirty-tree cases as Grade I, fifty-nine as Grade II and thirty-seven as Grade III. Images from tissue specimens were captured using a light microscopy imaging system. The method of digitalization and cell nuclei segmentation for analysis has been described in previous work [17]. Finally, from each case 36 features were estimated: 18 features were used to describe information concerning nuclear size and shape distribution. The rest were textural features that encoded chromatin distribution of the cell nucleus [17]. These features were used as an input to ANN classifier.

3

Artificial Neural Networks

Back Propagation Neural Networks (BPNNs) are the most popular artificial neural network models. The efficient supervised training of BPNNs is a subject of considerable ongoing research and numerous algorithms have been proposed to this end. Supervised training amounts to the global minimization of the network learning error.

Urinary Bladder Tumor Grade Diagnosis

201

Applications of supervised learning can be divided into two categories: stochastic (also called on line) and batch (also called off line) learning. Batch training can be viewed as the minimization of the error function E. This minimization corresponds to updating the weights by epoch and to be successful it requires a sequence of weight iterates {wk }∞ k=0 where k indicates epochs, which converges to a minimizer w∗ . In on line training, network weights are updated after the presentation of each training pattern. This corresponds to the minimization of the instantaneous error of the network E(p) for each pattern p individually. On line training may be chosen for a learning task either because of the very large (or even redundant) training set or because we want to model a gradually time varying system. Moreover, it helps escaping local minima. Given the inherent efficiency of stochastic gradient descent, various schemes have been proposed recently [1, 18, 19]. Unfortunately, on line training suffers from several drawbacks such as sensitivity to learning parameters [16]. Another disadvantage is that most advanced optimization methods, such as conjugate gradient, variable metric, simulated annealing etc., rely on a fixed error surface, and thus it is difficult to apply them for on line training [16]. Regarding the topology of the network it has been proven [5, 20] that standard feedforward networks with a single hidden layer can approximate any continuous function uniformly on any compact set and any measurable function to any desired degree of accuracy. This implies that any lack of success in applications must arise from inadequate learning, insufficient number of hidden units or the lack of a deterministic relationship between inputs and targets. Keeping these theoretical results we restrict the network topology of ANNs used in this study to one hidden layer.

4

Training Method

For the purpose of training neural networks an on-line stochastic method was employed. For recent proposed on–line training methods as well as application in medical applications see [8, 9, 10, 11, 14] This method uses a learning rate adaptation scheme that exploits gradient-related information from the previous patterns. This algorithm is described in [9], and is based on the stochastic gradient descent proposed in [1]. The basic algorithmic scheme is exhibited in Table 1. As pointed out in Step 4, the algorithm adapts the learning rate using the dot product of the gradient from the previous two patterns. This algorithm produced both fast and stable learning in all the experiments we performed, and very good generalization results.

5

Results and Discussion

To measure the ANN efficiency the dataset was randomly permutated five times. Each time it was split into a train set and a test set. The training set contained about 2/3 of the original dataset from each class. For each permutation the

202

D.K. Tasoulis et al.

Table 1. Stochastic On-Line Training with adaptive stepsize 0: 1: 2: 3: 4: 5: 6:

Initialize weights w0 ,η 0 , and meta-stepsize K. Repeat for each input pattern p Calculate E(wp ) and then ∂E(wp ) Update the weights: wp+1 = wp − η p ∂E(wp ) Calculate the stepsize to be used with the next pattern p + 1: η p+1 = η p + K∂E(wp−1 ), ∂E(wp ) Until the termination condition is met. Return the final weights w∗ .

Table 2. Accuracy of Grade I,II and III for various topologies Topology 36-1-3 36-2-3 36-5-3 36-16-3

Grade I 18.18% 81.81% 90.90% 81.81%

Grade II 100% 100% 100% 100%

Grade III 100% 100% 100% 100%

network was trained with the Stochastic On Line method with adaptive step size discussed previously. Two terminating conditions were used: the maximum number of cycles over the entire training set was set to 100, and the correct classification of all the training patterns. Alternatively, the Leave-One-Out (LOO) method [15] was employed to validate ANN classification accuracy. According to this method, the ANN is initialized with the training set including all patterns except one. The excluded pattern is used to assess the classification ability of the network. This process is repeated for all the patterns available and results are recorded in the form of a truth table. The software used for this task was developed under the Linux Operating System using the C++ programming language, and the gcc ver2.96 compiler. A great number of different ANN topologies (number of nodes in the hidden layer) were tested for the grade classification task. Some of these tests are exhibited in Table 2. Best results were obtained using the topology: 36-5-3. Table 3 illustrates analytically the ANN performance for each Crossover permutation. The ANN exhibited high classification accuracy for each grade category. It is worth noting that Grade I tumors were differentiated successfully from Grade III tumors. In four out of five Crossovers neither Grade I to III nor Grade III to I errors occurred. As can be seen from Table 3 in one permutation only 1 case of Grade I was misclassified as Grade III.rom a clinical point of view, it is important to distinguish low grade tumors, which can generally be treated

Urinary Bladder Tumor Grade Diagnosis

203

Table 3. Crossover Results For the ANNs

Histological finding GRADE I GRADE II GRADE III Overall Accuracy Histological finding GRADE I GRADE II GRADE III Overall Accuracy Histological finding GRADE I GRADE II GRADE III Overall Accuracy Histological finding GRADE I GRADE II GRADE III Overall Accuracy Histological finding GRADE I GRADE II GRADE III Overall Accuracy

Crossover I ANN classification Grade I Grade II Grade III 10 1 0 0 19 1 0 1 12 Crossover II ANN classification Grade I Grade II Grade III 10 1 0 0 20 0 0 2 11 Crossover III ANN classification Grade I Grade II Grade III 10 1 0 0 19 1 0 1 12 Crossover IV ANN classification Grade I Grade II Grade III 10 0 1 0 19 1 0 0 13 Crossover V ANN classification Grade I Grade II Grade III 10 1 0 0 18 2 0 0 13

Accuracy(%) 90.9 95 92.3 93.2

Accuracy(%) 90.9 100 84.62 93.2

Accuracy(%) 90.9 95 92.3 93.2

Accuracy(%) 90.9 95 100 95.4

Accuracy(%) 90.9 90 100 93.2

conservatively in contrast to high-grade tumors. The latter often require a more aggressive therapy because of a high-risk cancer progression. Results could also be interpreted in terms of specificity and sensitivity. That is specificity is the percentage of Grade I tumors correctly classified and sensitivity is the percentage of Grade III tumors correctly classified. ANN grade classification safeguarded high sensitivity which is of vital importance for patients treatment course, retaining

204

D.K. Tasoulis et al.

Table 4. Leave One out Results For the ANNs Histological finding GRADE I GRADE II GRADE III Overall Accuracy

ANN classification Grade I Grade II Grade III 30 1 2 0 56 3 0 1 36

Accuracy(%) 90 94.9 97.3 94.06

at the same time high specificity. Another important outcome is that the intermediate Grade II tumors were recognized with high confidence from Grade I and III. This would be particular helpful for pathologist who encounter difficulties in assessing Grade II tumors since some of them fall into the gray zone bordering on either Grade I or Grade III, and the decision is subject to the judgment of the pathologist. The simplicity and efficiency of the training method enabled us to verify the ANN classification accuracy by employing the LOO method (the whole procedure required 46 seconds to complete in Athlon CPU running at 1500 MHz). It is well known that this method is optimal to test the performance of a classifier when small data sets are available, but this testing procedure is computationally expensive when used with conventional training methods. Classification results employing the LOO method are shown in Table 4. The consistency of the system in terms of high sensitivity (no Grade III to Grade I error occurred) was verified. In [3], a textural based system produced an overall accuracy of 84.3% in assessing tumors grade. In a different study [6], researchers have employed tissue architectural features and classified tumors with an accuracy of 73%. More recent studies have focused on the analysis of cell nuclei characteristics to perform tumor grade classification with success rates that do not significantly exceed 80% [2, 17]. The ANN methodology proposed in this paper, improved significantly the tumor grade assessment with success rates of 90%, 94.9%, and 97.3%, for Grade I, II and III respectively.

6

Conclusions

In this study an ANN was designed to improve the automatic characterization of TCCs employing nuclear features. The ANN exhibited high performance in correctly classifying tumors into three categories utilizing all the available diagnostic information carried by nuclear size, shape and texture descriptors. The proposed ANN could be considered as an efficient and robust classification engine able to generalize in making decisions about complex input data improving significantly the diagnostic accuracy. The present study extends previous work in terms of the features used and enforces the belief that objective measurements on nuclear morphometry and texture offer a potential solution for the accurate

Urinary Bladder Tumor Grade Diagnosis

205

characterization of tumor aggressiveness. The novelty of this paper resides in the results obtained since they are the highest reported in the literature. Since most Grade I tumors are considered to be good prognosis, while Grade III is associated with bad prognosis, the 0% misclassification of Grade III tumors as Grade I, gives an advantage to the proposed methodology to be part of a fully automated computer aided diagnosis system.

References [1] L. B. Almeida, T. Langlois, L. D. Amaral, and A. Plankhov. Parameter adaption in stohastic optimization. On-Line Learning in Neural Networks, pages 111–134, 1998. 201 [2] N. Belacel and M. R. Boulassel. Multicriteria fuzzy assignment method: a useful tool to assist medical diagnosis. Artificial Intelligence in Medicine, 21:201–207, 2001. 200, 204 [3] H.-K. Choi, J. Vasko, E. Bengtsson, T. Jarkrans, U. Malmstrom, K. Wester, and C. Busch. Grading of transitional cell bladder carcinoma by texture analysis of histological sections. Analytical Cellular Pathology, 6:327–343, 1994. 200, 204 [4] C. De Prez, Y. De Launoit, R. Kiss, M. Petein, J.-L. Pasteels, and A. Verhest. Computerized morphonuclear cell image analysis of malignant disease in bladder tissues. Journal of Urology, 143:694–699, 1990. 200 [5] K. Hornik. Multilayer feedforward networks are universal approximators. Neural Networks, 2:359–366, 1989. 201 [6] T. Jarkrans, J. Vasko, E. Bengtsson, H.-K. Choi, U. Malmstrom, K. Wester, and C. Busch. Grading of transitional cell bladder carcinoma by image analysis of histological sections. Analytical Cellular Pathology, 18:135–158, 1995. 200, 204 [7] I. E. Jonathan, B. A. Mahul, R.R Victor, and F.K Mostofi. (transitional cell) neoplasms of the urinary bladder. The American Journal of Surgical Pathology, 22(12):1435–1448, 1998. 200 [8] G. D. Magoulas, V. P. Plagianakos, and M. N. Vrahatis. Global learning rate adaptation in on–line neural network training. In Proceedings of the Second International Symposium in Neural Computation May 23–26, Berlin, Germany, 2000. 201 [9] G. D. Magoulas, V. P. Plagianakos, and M. N. Vrahatis. Adaptive stepsize algorithms for on-line training of neural networks. Nonlinear Analysis, T. M. A., 47(5):3425–3430, 2001. 201 [10] G. D. Magoulas, V. P. Plagianakos, and M. N. Vrahatis. Hybrid methods using evolutionary algorithms for on–line training. In INNS–IEEE International Joint Conference on Neural Networks (IJCNN), July 14–19, Washington, D. C., U. S. A., volume 3, pages 2218–2223, 2001. 201 [11] G. D. Magoulas, V. P. Plagianakos, and M. N. Vrahatis. Improved neural networkbased interpretation of colonoscopy images through on-line learning and evolution. In D. Dounias and D. A. Linkens, editors, European Network of Excellence on Intelligent Technologies for Smart Adaptive Systems, pages 38–43. 2001. 201 [12] E. Ooms, W. Anderson, C. Alons, M. Boon, and R. Veldhuizen. Analysis of the performance of pathologists in grading of bladder tumors. Human Pathology, 14:140–143, 1983. 200 [13] S. L. Parker, T. Tony, S. Bolden, and P. A. Wingo. Cancer statistics. Cancer Statistics 1997. CA Cancer J Clin, 47(5):5–27, 1997. 199

206

D.K. Tasoulis et al.

[14] V. P. Plagianakos, G. D. Magoulas, and M. N. Vrahatis. Tumor detection in colonoscopic images using hybrid methods for on–line neural network training. In G. M. Papadourakis, editor, Neural Networks and Expert Systems in Medicine and Healthcare (NNESMED), pages 59–64. Technological Educational Institute of Crete, Heraklion, 2001. 201 [15] S. J. Raudys and A. K. Jain. Small sample size effects in statistical pattern recognition: Recommendations for practitioners. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 252–264, 1991. 202 [16] N. N. Schraudolf. Gain adaptation in stochastic gradient descend. 1999. 201 [17] P. Spyridonos, D. Cavouras, P. Ravazoula, and G. Nikiforidis. Neural network based segmentation and classification system for the automatic grading of histological sections of urinary bladder carcinoma. Analytical and Quantitative Cytology and Histology, 24:317–324, 2002. 200, 204 [18] R. S. Suton. Adapting bias by gradient descent: an incremental version of deltabar-delta. In Proc. 10th National Conference on Artificial Intelligence, pages 171–176. MIT Press, 1992. 201 [19] R. S. Suton. Online learning with random representations. In Proc. 10th International Conference on Machine Learning, pages 314–321. Morgan Kaufmann, 1993. 201 [20] H. White. Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings. Neural Networks, 3:535–549, 1990. 201

Invariants and Fuzzy Logic Germano Resconi and Chiara Ratti Dipartimento di Matematica, Universit` a cattolica via Trieste 17, 25128 Brescia, Italy [email protected]

Abstract. In this paper with the meta-theory based on modal logic, we study the possibility to build invariants in fuzzy logic. Every fuzzy expression will be compensated by other expressions so that to get a global expression which value is always true. The global expression takes the name of invariant because it always assumes true value also if the single components of the expression are fuzzy and they assume value between false and true. Key words: Fuzzy logic, meta-theory, modal logic, invariant, compensation, tautology.

1

Introduction

In the classical modal logic, given a proposition p and a world W, p can be true or false. In the meta-theory, we consider more worlds. P is valued in a set of worlds understood as only indivisible entity. If in the same set of worlds we have different evaluations, the uncertainty is born. In classical logic, if we consider a tautology, its evaluation is always true, but if the evaluation is done on a set of worlds according to the meta-theory, the same expression couldn’t be true. This fuzzy expression can be compensate by other expressions. We get a global expression which value is always true. It is an invariant.

2

Meta-theory Based on Modal Logic

In a series of papers Klir, et al.,(1994, 1995; Resconi, et al., 1992, 1993, 1994, 1996) a meta-theory was developed. The new approach is based on Kripke model of modal logic. A Kripke model is given by the structure: < W, R, V >. Resconi, et al. (1992-1996) suggested to adjoin a function Ψ : W → R where R is the set of real numbers assigned to worlds W in order to obtain the new model S1 =< W, R, V, Y > . (1) That is for every world, there is an associated real number that is assigned to it. With the model S1, we can build the hierarchical meta-theory where we can calculate the expression for the membership function of truthood in the fuzzy set theory to verify a given sentence via a computational method based V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 207–212, 2003. c Springer-Verlag Berlin Heidelberg 2003


208

Germano Resconi and Chiara Ratti

on {1, 0} values corresponding to the truth values {T, F } assigned to a given sentence as the response of a world, a person, or a sensor, etc. At this point, we should ask: “what are the linkages between the concepts of a population of observers, a population of possible worlds and the algebra of fuzzy subsets of well defined universe of discourse?” To restate this question, we need to point out that fuzzy set was introduced for the representation of imprecision in natural languages. However, imprecision means that a word representing an “entity” (temperature, velocity) cannot have a crisp logic evaluation. The meaning of a word in a proposition may usually be evaluated in different ways for different assessments of an entity by different agents, i.e. worlds. An important principle is: “we cannot separate the assessments of an entity without some loss of property in the representation of that entity itself ”. Different and in some cases conflicting evaluations for the same proposition may come up for the same entity. The presence of conflicting properties within the entity itself is the principal source of the imprecision in meaning representation of the entity. For example, suppose the entity is a particular temperature of a room, and we ask for the property cold, when we have no instrument to measure that property. The meaning of the entity, “temperature”, is composed of assessments that are the opinions of a population of observers that evaluate the predicate “cold”. Without the population of observers, and their assessments, we cannot have the entity “temperature” and the predicate “cold”. When we move from crisp set to the fuzzy set, we move from atomic elements, i.e., individual assessments to non-atomic entities, i.e., aggregate assessments. In an abstract way, the population of the assessments of the entity becomes the population of the worlds that we associate the set of propositions with crisp logic evaluation. Perception based data embedded in a context generate imprecise properties. Imprecise properties are generating by the individual generalization to the other worlds of the evaluation of the properties itself. For example when four persons give four different evaluations and any person think that its evaluation is the right evaluation and the others are wrong, we have in the four worlds a superposition of the individual evaluation and this generate uncertainty or fuzziness. With the perception-based data, as assessments of an entity, in context, we evaluate the properties of an entity. The evaluation can be conflicting. In such cases, the world population assessment composed of individual assessments give the context structure of the perception-based data. If we know only the name of a person (entity), we cannot know if s/he is “old”. Additional observationbased information that s/he is married, is a manager, and that s/he plays with toys are conflicting information of the proposition that associates her/him with the predicate “old”, etc. The aim of this paper is to show that with a model of a perception based imprecision generated by worlds, i.e., agents, we can on the one hand simplify the definitions of the operations in fuzzy logic and on the other hand expose and explain deeper issues embedded within the fuzzy theory. Consider a sentence such as: “John is tall” where x ∈ X is a person, “John”, in a population of people, X, and “tall” is a linguistic term of a linguistic

Invariants and Fuzzy Logic

209

variable, the height of people. Let the meta-linguistic expression that represents the proposition “John is tall” written in fuzzy set canonical form as follows: pA (x) ::= “x ∈ XisrA ,

(2)

where “isr” means “x ∈ X is in relation to a fuzzy information granule A”, and pA (x) is the proposition that relates x to A. Next consider that the imprecise sentence “John is tall” which can be modelled by different entities, i.e., different measures of the physical height h. At each height h, let us associate the opinions, based on their perceptions, of a set of observers or sensors that give an evaluation for the sentence “John is tall”. Any observer or sensor can be modelled in an abstract way by a world wk ∈ W , where W is the set of all possible worlds that compose the indivisible entity associated with a particular h and k is the index, 1, 2, 3 · · · n, that associates with any assessment of the entity given by the population of the observers. It should be noted that the world, i.e., the person or the sensor, does not say anything about the qualification, i.e., descriptive gradation of “John’s being tall”, but just verifies on the basis of a valuation schema. With these preliminaries, we can then write for short: pA (x) such that V (pA (x), wk ) evaluates to T, true. Next let us assign h(wk , x, A) = 1 if or h(wk , x, A) = 0 if With this background, we next define the membership expression of truthood of a given atomic sentence in a finite set of worlds wk ∈ W as follows:
η(wk , x, A) set of worlds wherepA (x)is true = k . (3) µPA (x) = |W (X)| |W (x)| In the equation 3 for any value of the variable x we associate the set of the worlds W (x) that represent the entity with the opinions, based on their perceptions, of the population of the observers, where pA represents the proposition that the atomic sentence “John is tall”, for x = “John”, and A = “tall” such that ; and |W | is the cardinality of the set of worlds in our domain of concern. Recall once again that these worlds, wk ∈ W , may be agents, sensors, persons, etc. Let us define the subset of worlds WA = wk ∈ W |V (pA(x), wk) = T . We can write, expression 3 as follows: |WA | . (4) µPA (x) = |W | With the understanding that WA represent the subset of the worlds W where the valuation of pA (x) is “true” in the Kripke sense. For the special case, where the relation R in the Kripke model is wk Rwk at any world wk only itself is accessible (any world wk is isolated from the others) the membership expression is computed as the value of Y in S1 stated in 1 above. It is computed by the 1 expression Ψ = |W | for any (single) world w in W. Thus we can write the expression 3 as follows:  µPA (x) = η(wk )Ψk . (5) k

210

3

Germano Resconi and Chiara Ratti

Transformation by External Agent

Given a set of worlds with a parallel evaluation of true value for every world, we can use an operator Γ to change the logic value of the propositions in the worlds. In the modal logic we have the possible ♦ and necessary , unary operators that change the logic value of the proposition p. One proposition p in one world w is necessary true or p is true, when p is true in all accessible worlds. One proposition p in one world w is possible true or ♦p is true, when is true in almost one world accessible from w. Near to these operators, we adjoin a new operator Γ . One proposition p in one world is true under the agent’s action, or Γ p is true, when there is an agent under whose action p is true. The action of the agent can be neutral so in the Γ p expression p does not change its logic value. The action of the agent can also change the logic value of p in false. With the agent operator G we can generate any type of t-norm or t-conorm and also non classical complement operator. (6) Exemple. For example given five worlds (w1 , w2 , w3 , w4 , w5 ) and two propositions p and q. When the logic evaluation of p in the four worlds is V (p) = (T rue, T rue, F alse, F alse, F alse)

(7)

V (q) = (F alse, T rue, T rue, T rue, F alse) .

(8)

and

For 7 and 8 we have µp = 25 and µq = 35 . The evaluation of V (pORq) = (T rue, T rue, T rue, T rue, F alse) where we use the classical logic operation OR for every world. So we have that µporq =

4 2 3 3 > max( , ) = 5 5 5 5

(9)

but when we introduce an external agent whose action changes the true value of p in this way V (Γ p) = (F alse, T rue, T rue, F alse, F alse) we have that µΓ porq =

2 3 3 3 = max( , ) = . 5 5 5 5

(10)

The agent has transformed the general fuzzy logic operation OR into the Zadeh max rule for the same operation OR. In conclusion the agent G can move from one fuzzy logic operation to another with the same membership function value for p and q. We know that, given the proposition p and the membership function µp , the Zadeh complement is µcp = 1 − µp . In this case we have that µp and cp = min(µp , 1 − µp ) that in general is different from zero or false value. We know that for Zadeh fuzzy logic the absurd condition is not always false. (11) Exemple. When we use the world image for the uncertainty we have evaluation 7 and the evaluation of its complement:

Invariants and Fuzzy Logic

211

V (cp) = (F alse, F alse, T rue, T rue, T rue); we have that V (p and cp) = (F alse, F alse, F alse, F alse, F alse); then V (p and cp) is always false. In this case the absurd condition is always false. But when an agent changes the true value position of the cp expression, in this way, if V (Γ cp) = (T rue, T rue, T rue, F alse, F alse), we have that V (p and Γ cp) = (T rue, T rue, F alse, F alse, F alse) and 2 2 3 µpandΓ cp = min( , ) = 5 5 5

(12)

that is different from false. We remark that V (Γ cp) = V (cΓ p). To have uniform language with classical logic we write cp = ¬p

4

Invariants in Fuzzy Logic

We know that in classical logic we have the simple tautology T = p ∨ ¬p

(13)

that is always true for every logic value of p When we evaluate the previous expression by the worlds and we introduce an external agent that use the operator G to change the logic value of p, we have the fuzzy expression of T or TF T F = p ∨ Γ ¬p = p ∨ ¬Γ p

(14)

Because in 14 we introduce a new variable or Γ p, the 14 is not formally equal to 13 so can assume also false value when the variable Γ p is different from the variable p. In fact the 14 can be written, for every world , in this way TF = Γp → p

(15)

that is false when Γ p is true and p is false. So a tautology in the classical logic is not a tautology in the fuzzy logic. In fact with the Zadeh operation we have µporcp = max(µp , 1 − µp ) ≤ 1

(16)

To find an invariant inside the fuzzy logic, we change the variables in the expression 13 in this way p → p ∨ ¬Γ p (17) because the expression 13 is a tautology , it is always true for every substitution of the variable. So we have that T = (p ∨ ¬Γ p) ∨ ¬(p ∨ ¬Γ p) is always true.

(18)

212

Germano Resconi and Chiara Ratti

So we have that T can separate in two parts one is the expression (4) and the other is the compensation part. In conclusion for the DeMorgan rule we have (p ∨ ¬Γ p) ∨ (¬p ∧ Γ p)

(19)

(20) Exemple. When we use the evaluation in the example 11, we have V (p) = (T rue, T rue, F alse, F alse, F alse), V (¬Γ p) = (T rue, T rue, T rue, F alse, F alse); V (Γ p) = (F alse, F alse, F alse, T rue, T rue), V (¬p) = (F alse, F alse, T rue, T rue, T rue). So we have V (p ∨ ¬Γ p) = (T rue, T rue, T rue, F alse, F alse) and V (¬p ∨ Γ p) = (F alse, F alse, F alse, T rue, T rue). When we come back to the original fuzzy logic with Zadeh operation we obtain max[µp , (1 − µp )] + min[(1 − µp ), µp ] = 1

(21)

where max[µp , (1 − µp )] is associate to the expression V (p ∨ ¬Γ p) and min[(1 − µp ), µp ] to V (¬p ∧ Γ p). The term min[(1 − µp ), µp ] is the compensation term. We can extend to other tautology the same process and in this way we can find new invariants in fuzzy logic.

5

Conclusion

In fuzzy logic, given a proposition p, we have fractionary evaluation of p. Then we can study t-norms, t-conorms and complement as general operations. In classical logic we have general properties for operations. They are always true for every evaluation(tautology). In fuzzy logic with invariants, we want to introduce greatness that don’t change if the single evaluation of p changes. In this way, invariants give a general formalization to fuzzy operations. The idea is to build a formal structure for fuzzy operations and to get a global expression always true even if every component is fuzzy.

References [1] G.Resconi,G.J Klir e U.St.Clair Hierarchical Uncertainity Metatheory Based Upon Modal Logic Int J. of General System, Vol. 21, pp. 23–50, 1992. [2] G.Resconi e I. B. Turksen. Canonical forms of fuzzy truthoods by meta theory based upon modal logic Information Sciences 131/2001 pp. 157–194. [3] G.Resconi, T.Murai. Field Theory and Modal Logic by Semantic Field to Make Uncertainity Emerge from Information IntJ. General System 2000. [4] G.Klir e Bo Yuan. Fuzzy sets and Fuzzy Logic Prentice Hall, 1995. [5] T.Muray, M.Nakata e M.Shimbo. Ambiguity, Inconsistency, and Possible-Worlds: A New Logical Approach Ninth Conference on Intelligence Technologies in HumanRelated Sciences, Leon, Spain, pp. 177, 184, 1996. [6] L. A.Zadeh. Fuzzy Sets Information and Control, Vol. 8, pp. 87–100, 1995.

Newsvendor Problems Based on Possibility Theory Peijun Guo Faculty of Economics, Kagawa University Takamatsu, Kagawa 760-8523, Japan [email protected]

Abstract. In this paper, the uncertainty of market of new products with short life cycle is characterized by possibility distribution. Two possibilistic models for newsvendor problem are proposed, one is based on optimistic criterion and the other is based on pessimistic criterion to reflect the different preference for risk in such one-shot decision problem, which are very different from the conventional newsvendor problem based on probability distribution in which maximizing expected utility is goal.

1

Introduction

The newsvendor problem, also known as newsboy or single-period problem is a wellknown inventory management problem. In general, the newsvendor problem has the following characteristics. Prior to the season, the buyer must decide how many units of goods to purchase. The procurement lead-time tends to be quite long relative to the selling season, so the buyer can not observe demand prior to placing the order. Due to the long lead-time, there is no opportunity to replenish inventory once the season has begun. Excess products can not be sold (or the price is trivial) over season. As well known that Newsvendor problem derives its name from a common problem faced by a person selling newspapers on the street, interest in such a problem has increased over the past 40 years partially because the increased dominance of service industrial for which newsboy problem is very applicable in both retailing and service organization. Also, the reduction in product life cycles makes newsboy problem more relevant. Many extensions have been made in last decade, such as different objects and utility function, different supplier pricing policies, different new-vendor pricing policies [11,17]. Almost all of extensions have been made in the probabilistic framework, that is, the uncertainty of demand and supply is characterized by the probability distribution, and the objective function is used to maximizing the expected profit or probability measure of achieving a target profit. However, some papers [3,8,9,10,13,14,15,18] have dealt inventory problems using fuzzy sets theory. There are few papers dealing with the uncertainty in newsvendor problems by fuzzy methods. Buckley [1] used possibility distribution to represents a decision-maker's linguistic express on demand, such as “good” and “not good” etc., and introduced some fuzzy goal to express decision-maker's satisfaction. The order quantity was obtained V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 213-219, 2003.  Springer-Verlag Berlin Heidelberg 2003

214

Peijun Guo

based on possibility and necessity measures, which made the possibility of not achieving fuzzy goal was sufficiently low and achieving fuzzy goal was sufficiently high. Ishi etc. al [6] investigated the fuzzy newsboy problem in which the shortage cost was vague and expressed as a fuzzy number. An optimal order quantity was obtained by fuzzy maximal order. Petrovic et al. [16] gave a fuzzy newsboy model where uncertain demand was represented by a group of fuzzy sets and inventory cost was given by a fuzzy number. The defuzzification method was used to obtain an optimal order quantity. Kao et al. [7] obtained the optimal order quantity to minimize the fuzzy cost by comparing the area of fuzzy numbers. Li et al. [12] proposed two models, in one the demand was probabilistic while the cost components were fuzzy and in the other the costs were deterministic but the demand was fuzzy. The profit was maximized through ordering fuzzy numbers with respect to their total integral values. Guo et al [4,5] proposed some new newsboy models with possibilistic information. In this paper, new newsboy models are proposed with emphasis that the proposed models are for new life-cycle short products, such as fashion goods, season presents where no data can be used for statistical analysis to predict the coming demand. The uncertainty of demand is characterized by the possibility distribution where possibility degrees are used to catch the potential of market determined by the matching degree between customer needs and goods features, which can obtained by detailed market investigation before selling season. In other words, the uncertain demand is not described by the linguistic language, such as “good”, “better” etc., to only reflect the subjective judgment of decision-maker. The other should be noted is that newsboy problem is a typical one-shot decision problem so that maximizing expected profit or probability measure seems less meaningful. In this paper, the optimal order is determined by maximizing optimistic and pessimistic values of order quantity instead of maximizing mean value in probabilistic models and ranking fuzzy numbers in fuzzy models. The third should be noted is that a general model is given where the possibility distributions and utility functions are not given as specified function so that the general analysis and conclusion are made.

2

Newsvendor Model Based on Possibility Theory

Consider a retailer who sells a short life cycle, or single-period new product. The retailer orders q units before the season at the unit wholesale price W. When demand x is observed, the retailer sell units (limited by the supply q and the demand x ) at unit revenue R with R > W . Any excess units can be salvaged at unit salvage price S o with W > S o . If shortage, the lost chance price is S u . The profit function of the retailer is as follows:  Rx + S o (q − x) − Wq ; if x < q r ( x, q ) =  ( R − W )q − S u ( x − q) ; if x ≥ q

(1)

Newsvendor Problems Based on Possibility Theory

215

Fig. 1. The possibility distribution of demand

The plausible information of demand x is represented by a possibility distribution. The possibility distribution of x , denoted as π D (x) is defined by the following continuous function

π D : [d l , d u ] → [0,1]

(2)

∃d c ∈ [d l , d u ] , π D (d c ) = 1 , π D (d l ) = 0 and π D (d u ) = 0 . π D increases within x ∈ [d l , d c ] and decreases within x ∈ [d c , d u ] . d l and d u are the lower and upper

bounds of demand, respectively, d c is the most possible amount of demand, which is shown in Fig. 1. Because demand is inside the interval [d l , d u ] , a reasonable supply also should lie inside this region. The highest profit of retailer is ru = ( R − W )d u . That is, the retailer orders the most d u and the demand is the largest d u . The lowest profit is rl = (d l R + (d u − d l ) S o − d u W ) ∧ (d l R − (d u − d l ) S u − d l W ) , which is determined by the minimum of two cases, one is that the retailer order the most but the demand is the lowest, the other is that the retailer order the lowest but the demand is the highest. Without loss of generality, the assumption W ≥ S o + S u is made, which leads to rl = d l R + (d u − d l ) S o − d u W . Definition 1. The utility function of retailer is defined by the following continuous strictly increasing function of profit r , U : [rl , ru ] → [0,1]

(3)

where U (rl ) = 0 , U (ru ) = 1 . (3) gives a general form of utility function of decisionmaker where the utility of the lowest profit is 0 and the utility of the highest profit is 1. Definition 2. The optimistic value of supply q, denoted as Vo (q) is defined as follows,

216

Peijun Guo

Fig. 2. The optimistic value of supply q

Vo (q ) = max min(π D ( x), U (r ( x, q ))) x

(4)

It can be seen that Vo (q) is similar to the concept of possibility measure of fuzzy event, illustrated by Fig. 2, where U (r ( x, q)) corresponds to fuzzy membership function. Definition 3. The pessimistic value of supply q, denoted as V p (q ) is defined as follows: V p (q) = min max(1 − π D ( x),U (r ( x, p))) x

(5)

It can be seen V p (q ) is similar to the concept of necessity measure of fuzzy event, illustrated by Fig. 3, where U (r ( x, q)) corresponds to fuzzy membership function. The retailer should decide an optimal order quantity, which maximizes Vo (q) or V p (q) , that is, q o* = arg max V o (q)

(6)

q *p = arg max V p (q)

(7)

where q o* and q *p are called optimistic order quantity and pessimistic order quantity, respectively. It is obvious that an order quantity q will be given a higher evaluation by optimistic criterion (4) if this order can lead to a higher utility with a high possibility. On the other hand, an order quantity q will be given a lower evaluation by pessimistic criterion (5) if this order can lead to a lower utility with a high possibility. It is clear the possibility theory based-approach makes some decision to balance plausibility and satisfaction. The optimistic criterion and pessimistic criterion are initially proposed by Yager [20] and Whalen [19], respectively. These criteria have been axiomatized in the style of Savage by Dubois Prade and Sabbadin [2].

Newsvendor Problems Based on Possibility Theory

217

Fig. 3. The pessimistic value of supply q

Lemma 1. In the case of π D (q ) ≥ U (r (q, q)) , arg max ( max min(π D ( x),U (r ( x, q )))) = qo* where q o* is the solution of the equaq∈[ d l , d u ] x∈[ d l , d u ]

tion π D (q ) = U (r (q, q )) with q > d c . Proof. Considering (1) max U (r ( x, q)) = U (r (q, q )) holds. Then min(π D ( x), U (r ( x, q))) ≤ U (r ( x, q)) ≤ U (r (q, q)) holds for any x ∈ [d l , d u ] . π D (q ) ≥ U (r (q, q)) makes min(π D (q ),U (r (q, q ))) = U (r (q, q )) hold. So that Vo (q ) = max min(π D ( x),U (r ( x, q))) = U (r (q, q)) . Because U (r (q, q)) is a strictly x∈[ d l , d u ]

increasing function of q , max Vo (q) leads to q increasing and q > d c holding. q

Within x ∈ [d c , d u ] increasing q makes U (r (q, q)) increase and π D (q ) decrease. With considering the condition π D (q ) ≥ U (r (q, q)) , when π D (q ) = U (r (q, q)) holds, Vo (q) reaches maximum where the optimal q is denoted as q o* .

Lemma 2. In Case of π D (q ) < U (r (q, q)) , max min(π D ( x), U (r ( x, q))) < U (r (q o* , q o* )) holds for any q < d c .

x∈[ d l , d u ]

Lemma 3. π D (q ) ≥ U (r (q, q)) holds for d c < q < q o* . Lemma 4. In the case of π D (q ) < U (r (q, q)) , max (min(π D ( x),U (r ( x, q ))) < U (r (qo* , qo* )) holds for any q > qo* .

x∈[ d l , d u ]

Theorem 1. q o* is the solution of the following equation.

π D (q) = U (( R − W )q) where q ∈ [d c , d u ] .

(8)

218

Peijun Guo

Proof. The relation between π (x) and U (r ( x, q)) can be divided into three cases, that is, Case 1: π D (q ) ≥ U (r (q, q)) ; Case 2: π D (q ) < U (r (q, q)) and q < d c , and Case 3: π D (q ) < U (r (q, q)) and q > d c . It should be noted that there is no such case of π D (q ) < U (r (q, q)) and q = d c because of 1 = π D (d c ) > U (r (d c , d c )) . It is straightforward from Lemma 1, 2, 3 and 4 that the optimal order quantity is the solution of the equation U (r (q, q)) = π D (q ) with q ∈ [d c , d u ] . Considering (1), π (q ) = U (( R − W )q ) is obtained. (8) can be easily solved by Newton method with the condition q ∈ [d c , d u ] . Theorem 1 shows that the optimal order quantity obtained from optimistic criterion is only based on the revenue and wholesale price, which means that the retailer has enough confidence that he can sell what he orders. Theorem 2. The pessimistic order q *p is the solution of the following equation. U (r (d *pl , q )) = U (r (d *pu , q ))

(9)

where d *pl and d *pu are the horizontal coordinates of the intersections of U (r ( x, q)) and 1 − π D ( x) within [dl , min[q, d c ]] and [max[q, d c ], d u ] , respectively.

3

Conclusions

In this paper, the uncertainty of market of new products with short life cycle is characterized by possibility degrees to catch the potential of market determined by the matching degree between customer needs and goods features. Two possibilistic models for newsvendor problem are proposed, one is based on optimistic criterion and the other is based on pessimistic criterion to reflect the different preference for risk in such one-shot decision problem, which are very different from the conventional newsboy problem maximizing expected utility. The optimal order quantity based on optimistic criterion is only depended on the revenue and wholesale price, which means that the retailer has enough confidence that he can sell what he has ordered. On the contrary, the optimal order quantity based on pessimistic criterion needs all information of market, which reflects a more conservative attitude. Because newsvendor problem is a typical one-shot decision problem and the uncertainty of the coming market is easily and reasonably characterized by possibility distribution, it can be believed that the proposed models can make some suitable decision for newsvendor problem.

References [1] [2]

Buckley, J.J., Possibility and necessity in optimization, Fuzzy Sets and Systems 25 (1988) 1-13 Dubois D., Prade H., and Sabbadin R. Decision-theoretic foundations of possibilty theory. European Journal of Operational Research, 128 (2001) 459-478.

Newsvendor Problems Based on Possibility Theory

[3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]

219

Gen, M., Tsujimura, Y. and Zheng, D., An application of fuzzy set theory to inventory control models, Computers Ind. Engng 33 (1997) 553-556. Guo, P., The newsboy problem based on possibility theory, Proceeding of the 2001 Fall National Conference of the Operations research Society of Japan (2001) 96-97. Guo, P. and Chen, Y., Possibility approach to newsboy problem, Proceedings of the First International Conference on Electronic Business (2001) 385-386. Ishii, H. and Konno, T., A stochastic inventory problem with fuzzy shortage cost, European Journal of Operational Research 106 (1998) 90-94. Kao, C. and Hsu., W. A single-period inventory model with fuzzy demand, Computers and Mathematics with Applications 43 (2002) 841-848. Katagiri, H. and Ishii, H., Some inventory problems with fuzzy shortage cost, Fuzzy Sets and Systems 111 (2000) 87-97. Katagiri, H. and Ishii, H., Fuzzy inventory problem for perishable commodities, European Journal of Operational Research 138 (2002) 545-553. Lee, H. and Yao, J. Economic order quantity in fuzzy sense for inventory without backorder model, Fuzzy Sets and Systems 105 (1999) 13-31. Lippman, S. A. and McCardle,K.F., The competitive newsboy, Operations Research, 45 (1997) 54-65. Li, L., Kabadi, S.N. and Nair, K.P.K., Fuzzy models for single-period inventory problem, Fuzzy sets and Systems (In Press) Lin, D. and Yao, J., Fuzzy economic production for inventory, Fuzzy Sets and Systems 111 (2000) 465-495. Kacpryzk, P. and Staniewski, P., Long-term inventory policy-making through fuzzy decision-making models, Fuzzy Sets and Systems 8 (1982) 117-132. Park, K.S., Fuzzy set theoretic interpretation of economic order quantity, IEEE Trans. Systems Man Cybernet. SMC 17(6) (1996) 1082-1084. Petrovic, D., Petrovic,R. and Vujosevic, M., Fuzzy model for newsboy problem, Internat. J. Prod. Econom. 45 (1996) 435-441. Porteus, E., L. Stochastic Inventory Theory, Handbooks in OR and MS, Vol. 2 Heyman D. P. and Sobel, M.J. eds. Elsevier Science Publisher, 605-652. Roy, T. K. and Maiti, M., A fuzzy EOQ model with demand-dependent unit cost under limited storage capacity, European Journal of Operational Research 99(1997) 425-432. Whalen, T. Decision making under uncertainty with various assumptions about available information, IEEE Transaction on Systems, Man and Cybernetics 14 (1984) 888-900. Yager, R. R., Possibilistic decision making, IEEE Transaction on Systems, Man and Cybernetics 9 (1979) 388-392.

Uncertainty Management in Rule Based Systems Application to Maneuvers Recognition T. Benouhiba, and J. M. Nigro Université de Technologie de Troyes 12 rue Marie Curie, BP 2060 10010 Troyes Cedex, France {toufik.benouhiba, nigro}@utt.fr

Abstract. In this paper we study the uncertainty management in expert systems. This task is very important especially when noised data are used such as in the CASSICE project, which will be presented later. Indeed, the use of a classical expert system in this case will generally produce mediocre results. We investigate the uncertainty management using the Dempster-Shafer theory of evidence and we discuss benefits and disadvantages of this approach by comparing the obtained results with those obtained by an expert system based on the fuzzy logic. Keywords: Expert system, evidence theory, uncertain reasoning.

1

Introduction

The expert systems technique is a very used approach in Artificial Intelligence; it can easily model expert knowledge in a well known field. Expert systems were used successfully in diagnostics, failure detection, recognition, etc. Nevertheless, they suffer from many problems. On one hand, they are closed systems, i.e., they could not react positively when the environment changes. On the other hand, they can neither make an uncertain reasoning nor handle noised or vague data. Since the first versions of MYCIN [16], many theoretical works have studied uncertainty in expert systems such as probabilities [7] and fuzzy logic [18]. However, there is another promising method that has not been well studied, which is the evidence theory. This one offers a mathematical framework for subjectivity and total ignorance but does not manage the uncertainty directly. In this paper we study the application of this theory in expert systems by combining it with some concepts of fuzzy logic. We see a real application of this approach to the recognition of driving maneuvers within the framework of the CASSICE project [12]. This paper is organized as follows. First, section 2 describes briefly the CASSICE project. Next, section 3 presents the Dempster-Shafer theory of evidence. Then, we see in section 4 the use of this theory in order to make an uncertain reasoning in expert systems. After that, section 5 shows an application example of the proposed approach in the CASSICE project; we discuss the obtained results by comparing them with those obtained by another expert system based on fuzzy logic. At the end, we give limitations and perspectives of this work. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 220-228, 2003.  Springer-Verlag Berlin Heidelberg 2003

Uncertainty Management in Rule Based Systems Application

2

221

The CASSICE Project

The CASSICE project aims to help psychologists to understand a driver behavior in order to model it. The final goal is to improve the comfort and the security of the driver. The concrete objective of CASSICE is to build a computer system, which indexes all real driving situations. This system allows psychologists to efficiently search driving situations in a given context via a multi-criteria research interface. This project uses a car equipped with a set of proprioceptive sensors and cameras. Until now, data which are used in the recognition come from a simulator. Table1 shows the used data types. Table 1. Simulator data types

Data Time Acc Phi Rd Rg Teta V X,Y

Signification Clock (second) Acceleration of EV compared to TV1 EV front wheels angle EV distance from the right border EV distance from the left border Angle with TV Speed of EV compared TV Position of TV compared to EV (x and y axis )

It should be noted that two classical expert systems have been developed: the first one, called IDRES [13], makes an exhaustive recognition of the states (we consider that a maneuver is composed of a succession of several states). A maneuver is recognized when all the states that compose it are detected in the right order. The second system, called DRSC [11], considers a maneuver as an automaton. A maneuver is recognized when the sensors data enable the recognition process to pass from the initial state to the last state of the automaton. In this paper we have limited the recognition to the overtaking maneuver; we consider that this one is composed of ten successive states: wait to overtake, overtaking intention, beginning of the lane changing to the left, crossing left discontinuous line, end of the lane changing to the left, passing, end of passing, beginning of the lane changing to the right, crossing right discontinuous line and end of the lane changing to the right.

3

Uncertainty Management

3.1

Evidence Theory Elements

This theory can be regarded as a general extension of the Bayesian theory. It can deal with total ignorance and subjectivity of experts by combining many points of view. The last point is the most important because it enables us to manage many data uncertainties at the same time. The evidence theory originates in the work of 1

TV means target vehicle whereas EV means experimental vehicle

222

T. Benouhiba and J. M. Nigro

Dempster on theory of probabilities with upper and lower bounds [2]. It was concretely formulated by his student Shafer in 1976 through his book “A Mathematical Theory of Evidence” [15]. Let Ω be a finite set of hypotheses, an evidence mass m is a function defined on 2 Ω such that ∑ m( A) = 1 where A ⊆ Ω . m( A) represents the exact belief in A. Any subset of Ω for which the evidence mass is non-zero is called a focal element. If Ω = {a, b} then m({a,b}) measures the fact that one is indifferent to a or b, in other words, it's an ignorance measure. This theory is a generalization of the probabilities theory: if all focal elements are singleton sets, then this measure is a probability [16]. Many other measures are defined in this theory but the most used are the belief (bel) and the plausibility (pl) measures: bel ( A) = ∑ m( B) B⊆ A

pl ( A) =

∑ m( B )

A∩ B ≠ ∅

(1)

The belief represents the confidence that one lies in A or any subset of A whereas the plausibility represents the extent to which we fail to disbelieve A. These two measures are, in reality, the lower and upper bounds of the probability of A. The Dempster combination operator is one of the most important elements of the evidence theory. It makes it possible to combine many evidence masses (many points of view) about the same evidence. Let m1 and m2 be two evidence masses, so the combination of two points of view give: 0  m1 ( A)m2 ( B ) m1 ⊕ m2 (C ) =  A∩∑ B =C 1 − ∑ m ( A)m ( B ) 2  A∩ B =∅1

3.2

if C = ∅ else

(2)

Adaptation of the Evidence Theory to Expert Systems

The evidence theory has been successfully used in many fields such as medical imagery [4], defaults classification by imagery [6], robot navigation systems [10] [17] and decision making [2] [3]. In General, this theory has shined in all fields relating to data fusion. However, too little works have been made to apply this theory to expert systems field; this is probably due to the combination operator. In fact, this operator can not directly combine many evidence masses associated to different variables. Some theoretical works have tried to resolve this problem, in this paper we use the method presented in [14]. In the following formulae, A and B are two distinct facts, m1 is the evidence mass associated to A, m2 is the evidence mass associated to B and m is the evidence mass associated to the fact (A and B). m( A ∩ B ) = m1 ( A).m 2 ( B )

(3)

Uncertainty Management in Rule Based Systems Application

223

m( A ∩ B ) = m1 ( A).m 2 ( B ) + m1 ( A).m 2 ( B ) + m1 ( A).m 2 ( B ) + m1 ( A).m 2 ( B ∪ B )

(4)

+ m1 ( A ∪ A).m 2 ( B ) m( A ∩ B ∪ A ∩ B) = m1 ( A).m 2 ( B ∪ B ) + m1 ( A ∪ A).m 2 ( B ) + m1 ( A ∪ A).m 2 ( B ∪ B )

4

Uncertain Reasoning

4.1

Evidence Masses Generating

(5)

In order to generate evidence masses, we consider each input data as an intuitionistic number (see Intuitionistic fuzzy sets [1]). The input data represents the center of this number while the choice of parameters depends on the scale and the variations of data. A condition is a conjunction of many simple conditions which have one of the following forms: or where belongs to the set {=, >, <, ≥, ≤, ≠} . The second form can be easily transformed into the first one by using operations on intuitionistic number. We associate to every pair () an intuitionistic set which has nearly the same form as an intuitionistic number, e.g., the belonging function of the operator (=n) is µ n,op = exp(−( x − n) 2 / α 2 ) whereas the non-belonging function is ν n,op = 1 − exp(−( x − n) 2 / β 2 ) with α ≤ β . To compute evidence masses, we use the Zadeh combination principle: m(True) = m(T ) = sup inf( µ , µ n,op )

m( False) = m( F ) = inf max(ν ,ν n,op )

(6)

µ and ν are, respectively, the belonging and the non-belonging functions of the intuitionistic number to be compared. The computed evidence mass is eventually combined with another one if the input data id given with an evidence mass m' such that m'(T) < 1 . 4.2

Inference System

When the evidence masses of all simple conditions are computed, we can calculate the evidence mass of the conjunction by using the Dempster operator: If S = A1 ∧ A2 ∧ ... ∧ An then m s = ⊕ m Ai i

The inference is made by applying the modus ponens rule and comparing the m s (T) with a threshold θ ; if m s (T ) ≥ θ then the rule is activated and all facts which are inserted by this rule will be associated to the evidence mass m s . In this paper we

224

T. Benouhiba and J. M. Nigro

use the CLIPS tool [5] which does not offer any direct tool to manage uncertainty. In particular it offers too little possibilities to make calculus in the left rule side. To solve this problem, each premise that contains a comparison between data will be transferred to the right side of the rule then we can decide whether the original rule could be fired. Concretely, if we consider the following classical rule in the left, it will be rewritten as in the right: Rule: Wait to overtake {sup, inf and val are parameters}

Rule: Wait to overtake {sup, inf, val and threshold are parameters}

If (y<sup) and (y>inf) and (x
If (true) Then E=M(y<sup) ⊕ M(y>inf) ⊕ M(xthreshold then State= («Wait to overtake», E)

However, another problem arises: useless rules could be fired. In reality, this problem can not be avoided because of the overhead introduced by the uncertainty management but we can solve it partially by using of a threshold. Even if we have a dedicated tool to write our expert system, this problem will remain but the code of the system would be more readable and easier to maintain.

5

Experience and Results

In order to make the driving maneuvers recognition, some considerations have to be mentioned. It should be noted that parameters are chosen according to the scale and the variations of data. The flexibility of the system depends on the choice of thresholds: if the threshold is important then the system will be close to a classical expert system, if it's, on the contrary, too small then the research tends to become random. 5.1

Results

The developed system enables us to trace the graphs of the variation of evidence masses for a given state through time (see paragraph 2). For example, figure 1 shows the graphs of the variations of the evidence masses associated to the “true” value (m(T)) for the two states: “overtaking intention” (continuous line) and “crossing the left discontinuous line” (dotted line):

Uncertainty Management in Rule Based Systems Application

225

Fig. 1. Variations of the evidence masses of two states

Figure 2 shows now the graph of the variations of the evidence mass associated to “true” for the whole maneuver:

Fig. 2. Evidence mass variations for the overtaking maneuver

Numbers on the graph refer to the rank of the recognized state within the maneuver. The evidence mass associated to the “true” value is a decreasing function which passes sometimes by picks to zero. This can be explained by the fact that we are in a total ignorance situation, i.e., we can not decide which state is occurring (unknown state). We remark that some states do not appear on the graph, this is due to the fact that these states are not indispensables to the maneuver recognition. 5.2

A Recognition System Based on Fuzzy Logic

We have developed a second expert system based on fuzzy logic in order to better evaluate the performance of the first system. We have used here a specific tool: FUZZYCLIPS [8] which is an extension to CLIPS tool in order to manage uncertainty by using fuzzy logic. In this system we transform input data into fuzzy numbers [9], this converting operation uses many parameters like the first system. The figure 3 shows the graph of the variations of the certitude factors (CF) of the two states “overtaking intention” (continuous line) and “crossing left discontinuous line” (dotted line) while figure 4 shows the graph of the variations of the CF of the whole maneuver:

226

T. Benouhiba and J. M. Nigro

Fig. 3. The CF variations of two states

Fig. 4. The CF variations of the maneuver

We can explain cuts in figure 4 by the fact that the system can not find the current state, i.e., these are total ignorance situations. The fuzzy logic theory can not manage these situations because it does not have any measure to quantify the ignorance. 5.3

Discussion

The obtained results by the both systems are nearly similar. They can handle imprecise data but each one uses its own method. The first system uses the intuitionistic fuzzy sets concept because they enable us to generate evidence masses; the second system uses the vague predicates concepts, the possibilities and necessities calculations. In addition, thanks to the evidence theory; the first system could make data fusion, e.g., if a right rule side inserts numeric data in the facts base and if it's about the same variable a first firing of the rule inserts the data in the facts base while the second one will merge the new data with the first one inserted. Both the systems enable us to see when the car gets ready to pass to a given state or to quit it. This makes easier the understanding of the maneuver progress through time. The rules of the second system are easier to write because of the use of vague predicates. In the first system, rules are neither easy to read nor to design because that CLIPS can not support directly the uncertainty management. In addition, transferring

Uncertainty Management in Rule Based Systems Application

227

premises from the left side to right side involves that lot of useless rules would be fired. We can solve the first problem by designing a specific tool by analogy with FUZZYCLIPS in order to hide all computing details. The second problem could not be solved; in reality this problem is due to the overhead introduced by the uncertainty management. However, because FUZZYCLIPS has the control of uncertainty, the second system is not so flexible. The first system does not suffer from this limitation; we can easily modify the system flexibility by changing the parameters α i and β i of the intuitionistic numbers. It offers several types of measures that can help us to better understand the maneuver progress.

6

Conclusion and Perspectives

In this paper we have studied two ways to manage uncertainty in expert system by applying them in a driving maneuver recognition application (the CASSICE project). The both systems consist in recognizing driving maneuvers by using data coming from several sensors, only the overtaking maneuver has been considered here. The first system is based on the Dempster-Shafer theory of evidence; however, we have used some fuzzy logic concepts in order to manage uncertainty in the rule-based system. The second system relies on the fuzzy sets theory but does not support total ignorance. Results obtained by the both systems are nearly similar. We can detect the moments when the driver gets ready to pass from one state to another. However, the first system allows us to better understand the maneuver progress by using many measures. For example, when the ignorance mass is equal to 1, we can deduct that the system could not recognize the current state; consequently, we could say that the rules set does not contain all necessary knowledge. There are two perspectives of this work. First, we should generalize the developed systems in order to recognize other maneuvers. We can imagine that the system would have a maneuvers database which contains all maneuvers description. Then, we can develop a specific tool for the evidence theory, by analogy with FUZZYCLIPS, in order to facilitate the expert system design.

Reference [1] [2] [3]

Atanassov K.: Intuitionistic fuzzy sets. FSS, 20: 87-96, 1986. Beynon M., Cosker D., Marshall D.: An expert system for multi-criteria decision making using Dempster-Shafer theory. Expert Systems with Applications 20, p.357 -367, 2001. Beynon M., Curry B., Morgan P.: The Dempster-Shafer theory of evidence: an alternative approach to multi-criteria decision modeling. The International Journal of Management Science, p. 37 -50, 2000.

228

T. Benouhiba and J. M. Nigro

[4]

Bloch I.: Some aspects of Dempster-Shafer evidence theory for classification of multimodality medical images taking partial volume effect into account. Pattern Recognition Letters 17, p.905-919, 1996. CLIPS : www.ghg.net/clips/download/documentation Dongping Z., Conners T., Schmoldt D., Araman P.: A prototype vision system for analyzing CT imagery of hardwood logs. IEEE Transactions on Systems, Man and Cybernetics: B 26(4), p.522-532, 1996. Fagin R., Halpern J. Y.: Reasoning about Knowledge and Probability. Proceedings of the Second Conference on Theoretical Aspects of Reasoning about Knowledge, Morgan Kaufmann, p.277 -293, 1988. FuzzyCLIPS : ai.iit.nrc.ca/cgi-bin/FuzzyCLIPS_ log Heilpern S.: Representation and application of fuzzy numbers. Fuzzy Sets and Systems 91, p.259-268, 1997. Kak A., Andress K., Lopez-Abadia C., Carol M., Lewis R.: Hierarchical evidence accumulation in the PSEIKI system. Uncertainty in Artificial Intelligence vol. 5, North-Holland, 1990. Loriette S., Nigro J.M., Jarkass I.: Rule-Based Approaches for the Recognition of Driving Maneuvers. Canberra (Australia), In ISTA'2000 (International Conference on Advances in Intelligent Systems: Theory and Applications), Volume59 of the series Frontiers in Artificial Intelligence, Ed. IOS Press 2-4 February 2000. Nigro J. M., Loriette-Rougegrez S., Rombaut M., Jarkass I.: Driving situation recognition in the CASSICE project – Towards an uncertainty management. ITSC 2000, 71-76, October 2000. Nigro J. M., Loriette S.: Characterization of driving situation. MS'99 (International Conference on Modeling and Simulation, May17-19, p.287-297. Nigro J. Marc, Loriette-Rougegrez S., Rombaut M.: Driving situation recognition with uncertainty management and rule-based systems. Engineering Applications of Artificial Intelligence Journal, Volume 15, Issue 3-4, pp. 217228, June - August 2002. Shafer G.: Mathematical theory of evidence. Princeton University Press, 1976. Shortliffe E. H., Buchanan B.: A model of inexact reasoning in medicine. Mathematical Biosciences, 23:351-379, 1975. Wang J., Wu Y.: Detection for mobile robot navigation based on multisensor fusion. Proceeding of the SPIE, The International Society for Optical Engineering vol.2591, p.182-192, 1995. Zadeh L. A.: Fuzzy sets. Information and control – n°8, p.338-353, 1965.

[5] [6] [7] [8] [9] [10] [11]

[12] [13] [14]

[15] [16] [17] [18]

Fuzzy Coefficients and Fuzzy Preference Relations in Models of Decision Making Petr Ekel1 , Efim Galperin2 , Reinaldo Palhares3, Claudio Campos1 , and Marina Silva1 1

3

Graduate Program in Electrical Engineering, Pontifical Catholic University of Minas Gerais Ave. Dom Jos´e Gaspar, 500, 30535-610, Belo Horizonte, MG, Brazil [email protected] [email protected] [email protected] 2 Department of Mathematics, University of Quebec at Montreal C.P. 8888, Succ. Centre-Ville, Montreal, Quebec, Canada H3C 3P8 [email protected] Department of Electronics Engineering, Federal University of Minas Gerais Ave. Antˆ onio Carlos, 6627, 31270-010, Belo Horizonte, MG, Brazil [email protected]

Abstract. Analysis of < X, R > models is considered as part of a general approach to solving a wide class of optimization problems with fuzzy coefficients. This approach consists in formulating and solving one and the same problem within the framework of interrelated models to maximally cut off dominated alternatives. The subsequent contraction of the decision uncertainty region is based on reduction of the problem to multiobjective decision making in a fuzzy environment with applying techniques based on fuzzy preference relations. The results of the paper are of a universal character and are already being used to solve power engineering problems.

1

Introduction

Investigations of recent years show the benefits of applying fuzzy set theory to deal with various types of uncertainty, particularly for optimization problems where there are advantages of a fundamental nature (the possibility of validly obtaining less “cautious” solutions) and computational character [1]. The uncertainty of goals is the notable kind of uncertainty related to a multiobjective character of many optimization problems. It is possible to classify two types of problems, which need the use of a multiobjective approach [2]: (a) problems in which solution consequences cannot be estimated on the basis of a single criterion, that involves the necessity of analyzing a vector of criteria, and (b) problems that may be solved on the basis of a single criterion but their unique V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 229–236, 2003. c Springer-Verlag Berlin Heidelberg 2003


230

Petr Ekel et al.

solutions are not achieved because the uncertainty of information produces socalled decision uncertainty regions, and the application of additional criteria can serve as a means to reduce these regions [3]. According to this, two classes of models (so-called < X, M > and < X, R > models) may be constructed. When analyzing < X, M > models, a vector of objective functions is considered for their simultaneous optimization. The lack of clarity in the concept of “optimal solution” is the fundamental methodological complexity in analyzing < X, M > models. When applying the Bellman-Zadeh approach [4], this concept is defined with reasonable validity because the maximum degree of implementing goals serves as the optimality criterion. This conforms to the principle of guaranteed result and provides a constructive line in obtaining harmonious solutions [1, 5]. Some specific questions of using the approach are discussed in [1, 2]. Taking this into account, the paper considers primarily < X, R > models.

2

Optimization Problems with Fuzzy Coefficients

Many problems related to complex system design and control may be formulated as follows: (1) maximize F
(x1 , . . . , xn ) , subject to the constraints
j , gj (x1 , . . . , xn ) ⊆ B


j = 1, . . . , m ,

(2)

where the objective function (1) and constraints (2) include fuzzy coefficients, as indicated by the ∼ symbol. The following problem can also be defined: minimize F
(x1 , . . . , xn ) ,

(3)

subject to the same constraints (2). An approach [3] to handle constraints such as (2) involves replacing each of them by a finite set of nonfuzzy constraints. Depending on the essence of the problem, it is possible to convert constraints (2) to constraints gj (x1 , . . . , xn ) ≤ bj ,

j = 1, . . . , d ≥ m ,

(4)

gj (x1 , . . . , xn ) ≥ bj ,

j = 1, . . . , d ≥ m .

(5)

or to constrains

Problems with fuzzy coefficients only in the objective functions can be solved by modifying traditional methods [2, 3]. For example, the algorithms [1, 3] of solving discrete problems (1), (4) and (3), (5) are based on modifying the methods of [6]. In their use, the need arises to compare alternatives on the basis of relative fuzzy values of the objective function. This may be done by applying the methods classified in [7]. One of their groups is related to building fuzzy

Fuzzy Coefficients and Fuzzy Preference Relations

231

preference relations providing the most justified way to compare alternatives [8]. Considering this, it is necessary to distinguish the choice function or fuzzy number ranking index based on the conception of a generalized preference relation [9]. If the membership functions corresponding to the values F
1 and F
2 of the objective function to be maximized are µ(f1 ) and µ(f2 ), the quantities η{µ(f1 ), µ(f2 )} =

sup f1 , f2 ∈ F f1 ≥ f2

min {µ(f1 ), µ(f2 )} ,

(6)

min {µ(f1 ), µ(f2 )} ,

(7)

and η{µ(f2 ), µ(f1 )} =

sup f1 , f2 ∈ F f2 ≥ f1

are the degrees of preferences µ(f1 )  µ(f2 ) and µ(f2 )  µ(f1 ), respectively. Applying (6) and (7), it is possible to judge the preference of any of the alternatives compared. However, if the membership functions µ(f1 ) and µ(f2 ) correspond to so-called flat fuzzy numbers, the use of (6) and (7) can lead to η{µ(f1 ), µ(f2 )} = η{µ(f2 ), µ(f1 )}. In such situations the algorithms do not allow one to obtain a unique solution. This also occurs with other modifications of optimization methods when the uncertainty and relative stability of optimal solutions produce decision uncertainty regions. In this connection other choice functions (for example, [10, 11]) may be used. However, these functions occasionally result in choices, which appear intuitively inconsistent, and their use does not permit one to close the question of building an order on a set of fuzzy numbers [3]. An approach, which is better validated and natural, is associated with transition to multiobjective choosing alternatives. The application of additional criteria can serve as a convincing means to contract the decision uncertainty region.

3

Multiobjective Choice and Fuzzy Preference Relations

First of all, it should be noted that considerable contraction of the decision uncertainty region may be obtained by formulating and solving one and the same problem within the framework of mutually interrelated models:(a) maximizing (1) while satisfying constraints (4) interpreted as convex down and (b) minimizing (3) while satisfying constraints (5) interpreted as convex up [1, 3]. Assume we are given a set X of alternatives (from the decision uncertainty region) that are to be examined by q criteria of quantitative and/or qualitative nature to make an alternative choice. The problem is presented as < X, R > where R = {R1 , . . . , Rq } is a vector of fuzzy preference relations. Therefore Rp = [X × X, µRp (Xk , Xl )],

p = 1, . . . , q ,

Xk , X l ∈ X ,

(8)

where µRp (Xk , Xl ) is a membership function of fuzzy preference relation. The availability of fuzzy or linguistic estimates F
p (Xk ), p = 1, . . . , q, Xk ∈ X with the membership functions µ[fp (Xk )], p = 1, . . . , q, Xk ∈ X permits one to build Rp , p = 1, . . . , q on the basis of correlations similar to (6) and (7).

232

Petr Ekel et al.

If R is a single preference relation, it can be put in correspondence with a strict fuzzy preference relation RS = R\R−1 [9] with the membership function µSR (Xk , Xl ) = max{µR (Xk , Xl ) − µR (Xl , Xk ), 0} .

(9)

Since µSR (Xl , Xk ) describes a fuzzy set of alternatives, which are strictly dominated by Xl , its complement by 1 − µSR (Xl , Xk ) gives the set of nondominated alternatives. To choice all these alternatives, it is enough to find the intersection of all 1 − µSR (Xl , Xk ), Xk ∈ X on all Xl ∈ X with µR (Xk ) = inf [1 − µSR (Xl , Xk )] = 1 − sup µSR (Xl , Xk ) . Xl ∈X

Xl ∈X

(10)

Because µR (Xk ) is the degree of nondominance, it is natural to choice X  = {Xk | Xk ∈ X, µR (Xk ) = sup µR (Xk )} . Xk ∈X

(11)

If we have the vector fuzzy preference relation, expressions (9)-(11) can serve as the basis for a lexicographic procedure of step-by-step introducing criteria. It generates a sequence X 1 , X 2 , . . . , X q so that X ⊇ X 1 ⊇ X 2 ⊇ · · · ⊇ X q with the use of the following expressions: µp R (Xk ) = 1 −

sup Xl ∈X p−1

µSRp (Xl , Xk ),

p X p = {Xkp | Xkp ∈ X p−1 , µp R (Xk ) =

p = 1, . . . , q ,

sup Xk ∈X p−1

µp R (Xk )} ,

(12)

(13)

constructed on the basis of (10) and (11), respectively. If a uniquely determined order is difficult to build, it is possible to apply another procedure. In particular, the expressions (9)-(11) are applicable if we q take R = p=1 Rp , i.e., µR (Xk , Xl ) = min µRp (Xk , Xl ), l≤p≤q

Xk , Xl ∈ X .

(14)

The use of this procedure leads to the set X  that fulfils the role of the Pareto set [12]. If necessary, its contraction is possible using the convolution µT (Xk , Xl ) =

q 

λp µRp (Xk , Xl ),

Xk , Xl ∈ X ,

(15)

p=1

q where λp > 0, p = 1, . . . , q are importance factors normalized as p=1 λp = 1. The construction of µT (Xk , Xl ), Xk , Xl ∈ X allows one to obtain the membership function µT (Xk ) of the subset of nondominated alternatives using an expression similar to (9). Its intersection with µT (Xk ) defined as µ (Xk ) = min{µR (Xk ), µT (Xk )},

Xk ∈ X ,

(16)

Fuzzy Coefficients and Fuzzy Preference Relations

233

provides us with X  = {Xk | Xk ∈ X, µ (Xk ) = sup µ (Xk )} . Xk ∈X

(17)

Finally, it is possible to apply the third procedure based on presenting (9) as µRp (Xk ) = 1 − sup µSRp (Xl , Xk ), Xl ∈X

p = 1, . . . , q

(18)

to build the membership functions of the subset of nondominated alternatives for all preference relations. Since the membership functions (18) play a role identical to membership functions replacing the objective functions in < X, M > models [1, 2], it is possible to build µ (Xk ) = min µRp (Xk ) l≤p≤q

(19)

to obtain X  in accordance with (17). If necessary to differentiate the importance of different fuzzy preference relations, it is possible to transform (19) as follows: µ (Xk ) = min [µRp (Xk )]λp . l≤p≤q

(20)

The use of (20) does not require the normalization of λp , p = 1, . . . , q. It is natural that the use of the second procedure may lead to the solutions different from the results obtained on the basis of the first procedure. However, solutions based on the second procedure and the third procedure (that is more preferential from the substantial point of view) relating to a single generic basis may at time also be different. Considering this, it should be stressed that the possibility of obtaining different solutions is natural, and the choice of the approach is a prerogative of a decision maker. All procedures have been implemented within the framework of a decision making system DMFE (developed in C++ in the Builder-Borland environment). Its flexibility and user-friendly interaction with a decision maker makes it possible to use DMFE for solving complex problems of multiobjective choosing alternatives with the use of criteria of quantitative and qualitative nature.

4

Illustrative Example

The results described above have found applications in solving power engineering problems. As an example, we dwell on using the multiobjective choice of alternatives for substation planning. Without discussing substantial considerations (they are given in [13]), the problem consists in comparing three alternatives, which could not be distinguished from the point of view of their total costs, using criteria “Alternative Costs”, “Flexibility of Development”, and “Damage to Agriculture”. The fuzzy preference relations corresponding to these criteria,

234

Petr Ekel et al.

built with the use of (6) and (7) on the basis of alternative membership functions presented in [13], are the following:   1 1 1 1 1  , µR1 (Xk , Xl ) =  1 (21) 1 0.912 1 

 1 1 0.909 µR2 (Xk , Xl ) =  1 1 0.909  , 1 1 1 and

(22)



 1 1 1 1 1  . µR3 (Xk , Xl ) =  0.938 0.625 0.938 1

(23)

Consider the application of the first approach if the criteria are arranged, for example, in the following order: p = 1, p = 2, and p = 3. Using (21), it is possible to form the strict fuzzy preference relation with   0 0 0 µSR1 (Xk , Xl ) =  0 0 0.088  . (24) 0 0 0 Following (12) and (13), we obtain on the basis of (24)

1 1 0.912 µ1 R (Xk ) =

(25) 1 1 and X 1 = {X1 , X2 }. From (22) we have: µ2R2 (Xk , Xl ) = , that leads 1 1  1 1 2 3 , to X = {X1 , X2 }. Finally, from (23) we can obtain: µR2 (Xk , Xl ) = 0.938 1 providing us with X 3 = {X1 }. Consider the application of the second approach. As a result of an intersection of (21), (22), and (23), we obtain   1 1 0.909 1 0.909  . µR (Xk , Xl ) =  0.938 (26) 0.625 0.912 1 

It permits one to construct 

 0 0.062 0.284 0 0  . µSR (Xk , Xl ) =  0 (27) 0 0.003 0

Following (11), we obtain µR (Xk ) = 1 0.938 0.716 and X  = {X1 }. Let us consider the use of the third approach. The membership function of the subset of nondominated alternatives for µR1 (Xk ) is (25). Using (22) and (23),

Fuzzy Coefficients and Fuzzy Preference Relations

235





we obtain µR2 (Xk ) = 0.909 0.909 1 and µR2 (Xk ) = 1 0.938 0.625 , respectively. As a result of their intersection, we have X  = {X1 , X2 }. Thus, the use of the first and second approaches leads to choosing the first alternative. The last approach casts away the third alternative but cannot distinguish the first and second alternatives on the basis of information reflected by fuzzy preference relations (21)-(23).

5

Conclusion

Two classes of problems that need the application of a multiobjective approach have been classified. According to this, < X, M > and < X, R > models may be constructed. The use of < X, R > models is based on applying a general approach to solving optimization problems with fuzzy coefficients. It consists in analyzing one and the same problem within the framework of mutually interrelated models. The contraction of the obtained decision uncertainty region is based on reducing the problem to multiobjective choosing alternatives with applying procedures based on fuzzy preference relations. The results of the paper are of a universal character and can be applied to the design and control of systems and processes of different nature as well as the enhancement of intelligent decision support systems. These results are already being used to solve power engineering problems.

References [1] Ekel, P.Ya.: Fuzzy Sets and Models of Decision Making. Int. J. Comp. Math. Appl. 44 (2002) 863-875. 229, 230, 231, 233 [2] Ekel, P.Ya.: Methods of Decision Making in Fuzzy Environment and Their Applications. Nonlin. Anal. 47 (2001) 979-990. 229, 230, 233 [3] Ekel, P., Pedrycz, W., Schinzinger, R.: A General Approach to Solving a Wide Class of Fuzzy Optimization Problems. Fuzzy Sets Syst. 97 (1998) 49-66. 230, 231 [4] Bellman R., Zadeh L. A.: Decision Making in a Fuzzy Environment. Manage. Sci. 17 (1970) 141-164. 230 [5] Ekel, P.Ya., Galperin, E. A.: Box-Triangular Multiobjective Linear Programs for Resource Allocation with Application to Load Management and Energy Market Problems. Math. Comp. Mod., to appear. 230 [6] Zorin, V. V., Ekel, P.Ya.: Discrete-Optimization Methods for Electrical Supply Systems. Power Eng. 18 (1980), 19-30. 230 [7] Chen, S. J., Hwang, C. L.: Fuzzy Multiple Attribute Decision Making: Methods and Applications. Springer-Verlag, Berlin Heidelberg New York (1992). 230 [8] Horiuchi, K., Tamura, N.: VSOP Fuzzy Numbers and Their Fuzzy Ordering. Fuzzy Sets Syst. 93 (1998) 197-210. 231 [9] Orlovsky, S. A.: Decision Making with a Fuzzy Preference Relation, Fuzzy Sets Syst. 1 (1978) 155-167. 231, 232 [10] Fodor, J., Roubens, M.: Fuzzy Preference Modelling and Multicriteria Decision Support. Kluwer, Boston Dordrecht London (1994). 231

236

Petr Ekel et al.

[11] Lee-Kwang, H.: A Method for Ranking Fuzzy Numbers and Its Application to Decision-Making. IEEE Trans. Fuzzy Syst. 7 (1999) 677-685. 231 [12] Orlovsky, S. A.: Problems of Decision Making with Fuzzy Information. Nauka. Moscow.: Nauka (1981), in Russian. 232 [13] Ekel, P.Ya., Terra, L. D. B., Junges, M. F. D.: Methods of Multicriteria Decision Making in Fuzzy Environment and Their Applications to Power System Problems. Proceedings of the 13th Power Systems Computation Conference, Trondheim (1999) 755-761. 233, 234

Mining Fuzzy Rules for a Traffic Information System Alexandre G. Evsukoff and Nelson F. F. Ebecken COPPE/Federal University of Rio de Janeiro P.O.Box 68506, 21945-970 Rio de Janeiro RJ, Brazil {Evsukoff,Nelson}@ntt.ufrj.br

Abstract. This work presents a fuzzy system for pattern recognition in a real application: the selection of traffic information messages to be displayed in Variable Message Signs located at the main routes of the city of Rio de Janeiro. In this application, flow and occupancy rate data is used to fit human operators' evaluation of traffic condition, which is currently done from images of strategically located cameras. The fuzzy rule-base mining is presented considering the symbolic relationships between linguistic terms describing variables and classes. The application presents three classifiers built from data.

1

Introduction

The work presents the development of a fuzzy system for the selection of messages to be displayed in Variable Message Signs (VMS). Such devices were initially conceived to provide drivers with information about traffic incidents, environmental problems, special events etc [1], [4]. However, due to the increasing levels of congestion in big cities the VMS have been used mainly to display current traffic conditions. In the city of Rio de Janeiro, the VMS are systematically used to inform the traffic conditions on the main routes located downstream each panel. Operators in the traffic control centre analyse images from strategically located cameras and classify the conditions on each route in three categories: fluid, dense and slow. The classification is not standardised and depends on the operator's evaluation, which can be different for similar situations. On the other hand, many streets in Rio are equipped with inductive loop detectors. These sensors provide flow and occupancy rates, of which data is used for traffic lights planning but not to support VMS operation. Fuzzy reasoning techniques are a key for human-friendly computerised devices, allowing symbolic generalisation of high amount of data by fuzzy sets and providing its linguistic interpretability [2], [3], [5]. The application described in this work uses flow and occupancy data to mine fuzzy rules to support operators' evaluation in the selection of the messages to be displayed in VMS. The following section introduces the current approach. The third section describes the mining of fuzzy rules for the fuzzy model. The fourth section presents and discusses the results achieved by three classifiers built with this methodology. Finally, some concluding remarks are presented. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 237-243, 2003.  Springer-Verlag Berlin Heidelberg 2003

238

Alexandre G. Evsukoff and Nelson F. F. Ebecken

2

Fuzzy Systems for Pattern Recognition

Consider a pattern recognition problem where observations are described as a Ndimensional vector x in a feature space X N and classes are represented by the set C = {C1 K C m } . The solution consists in assigning a class label C j ∈ C to an observation x(t ) ∈ X N . The problem of designing a fuzzy system for pattern recognition is to build a classifier, which should executes correctly the mapping X N → C . Each input variable x(t ) ∈ X is described using ordered linguistic terms in a descriptor set A = {A1 , K , An } . The meaning of each term Ai ∈ A is given by a fuzzy set. The collection of fuzzy sets used to describe the input variable forms a fuzzy partition of the input variable domain. For a given input x(t ) ∈ X , the membership vector (or fuzzification vector) u(t ) is computed by the fuzzy sets in the fuzzy partition of the input variable domain as:

(

)

u(t ) = µ A1 ( x(t )) K µ An ( x(t )) .

(1)

In pattern recognition applications, the rules' conclusions are the classes in the set C = {C1 K C m } . The fuzzy rule base has thus to relate input linguistic terms Ai ∈ A to the classes C j ∈ C , in rules such as: if x(t ) is Ai then class is C j with cf = ϕ ij

(2)

where ϕ ij ∈ [0,1] is a confidence factor (cf) that represents the rule certainty. The confidence factor weights for all rules define the symbolic fuzzy relation Φ , defined on the Cartesian product A × C . A value µ Φ ( Ai , C j ) = ϕ ij represents how much the term Ai is related to the class C j in the model described by the rule base. A value ϕ ij > 0 means that the rule (i, j ) occurs in the rule base with the confidence

[ ]

factor ϕ ij . The rule base can be represented by the matrix Φ = ϕ ij

, of which the

lines i = 1...n are related to the terms in the input variable descriptor set A and the columns j = 1...m are related to classes in the set C . The output of the fuzzy system is the class membership vector (or the fuzzy model output vector) y (t ) = µ C1 ( x(t )), K , µ Cm ( x(t )) , where µ C j ( x(t )) is the output mem-

(

)

bership value of the input x(t ) to the class C j . The class membership vector y (t ) can also be seen as a fuzzy set defined on the class set C . It is computed from the input membership vector u(t ) and the rule base weights in Φ using the fuzzy relational composition: y (t ) = u(t ) o Φ

(3)

Mining Fuzzy Rules for a Traffic Information System

239

Adopting strong and normalised fuzzy partitions, and using the sum-product composition operator for the fuzzy inference, the class membership vector y (t ) is easily computed as a standard vector matrix product as: y (t ) = u(t ).Φ .

(4)

When two or more variables occur in the antecedent of the rules, input variables are the components of the feature vector x ∈ X N and the rules are written as: if x(t ) is Bi then class is C j with cf = ϕij

(5)

Each term Bi in the multi-variable descriptor set B = {B1 , K , B M } represents a combination of terms in the input variables' descriptor sets. It is a symbolic term used for computation purposes only and does not necessarily need to have a linguistic interpretation. All combination must be considered in such a way that the model is complete, i.e. it produces an output for whatever input values. The multi-variable fuzzy model described by a set of rules (5) is analogous to the single variable fuzzy model described by rules of type (2) and the model output y (t ) = µ C1 (x(t )), K , µ Cm (x(t )) is computed as:

(

)

y (t ) = w (t ).Φ

(6)

The combination of terms in multiple antecedent rules of type (5) implies in an exponential growth of the number of rules in the rule base. For problems with many variables, reasonable results [2] can be obtained considering the aggregation of partial conclusions computed by single antecedent rules of type (2). The final conclusion is the membership vector y (t ) = µ C1 (x(t )), K , µ Cm (x(t )) ,

(

)

which is computed by the aggregation of partial conclusions y i (t ) as in multi-criteria decision-making [6]. Each component µ C j (x(t )) is computed as:

(

µ C j (x(t )) = H µ C j ( x1 (t )), K , µ C j ( x N (t ))

)

(7)

where H : [0,1]N → [0,1] is an aggregation operator. The best aggregation operator must be chosen according to the semantics of the application. Generally, a conjunctive operator, such as the “minimum” or the “product”, gives good results to express that all partial conclusions must agree. A weighted operator like OWA may be used to express some compromise between partial conclusions. The final decision is computed by a decision rule. The most usual decision rule is the “maximum rule”, where the class is chosen as the one with greatest membership value. The “maximum rule” is often used to determine the most suitable solution. Nevertheless, other decision rules can be used including risk analysis and reject or ambiguity distances. When all variables are used in single variable rules as (2) and their outputs are aggregated as (7), the fuzzy classifier behaves as a standard Bayesian classifier. The

240

Alexandre G. Evsukoff and Nelson F. F. Ebecken

current approach is flexible enough so that some partial conclusions can be computed from the combination of two or three variables in multi-variable rules (5). An aggregation operator computes a final conclusion from partial conclusions obtained from all sub-models. A decision rule (the “maximum” rule) computes the final class as shown in Fig. 1. x1 (t )

A1

u 1 (t )

xk (t )

Bk

M x N (t )

AN

y 1 (t )

M

M xk +1 (t )

Φ1

w k (t )

u N (t )

Φk

M ΦN

y k (t )

H

y (t ) decision

C0

y N (t )

Fig. 1. The fuzzy system approach for pattern recognition

The rule base is the core of the model described by the fuzzy system and its determination from data or human knowledge is described in the next section.

3

Fuzzy Rule Base Mining

Consider a labelled data set T , where each sample is a pair (x(t ), v (t ) ) , of which x(t ) is the vector containing the feature values and v(t ) is the vector containing the assigned membership values of x(t ) to each class. For each fuzzy sub-model k , the rule base weights matrix Φ k is computed by the minimisation of the rule output error in each sub-model, defined as: J = N1

∑ (y k (t ) − v(t ) )2

t =1.. N

(8)

where y k (t ) = u k (t ).Φ k if the sub-model k is described by single antecedent rules as (2) or y k (t ) = w k (t ).Φ k if the sub-model is described by multiple antecedent rules as (5). The rule base weights matrix Φ k as the solution of the linear system: Uk Φk = V

(9)

where the matrix U k is the fuzzification matrix, of which lines are the input membership vector for each sample. Each line of the matrix U k can be the single variable input membership vector u k (t ) or the multi-variable input membership vector w k (t ) , if the sub-model k is described by single or multiple antecedent rules. The

Mining Fuzzy Rules for a Traffic Information System

241

matrix V stores, at each line, the class membership vector for the corresponding sample. Equation (9) can be computed by any standard numerical method. Fuzzy rule base weights are computed for each sub-model and used to compute partial conclusions on class descriptions. Final conclusions are computed by aggregation of partial conclusions of each sub-model. Heuristic and experimental knowledge can be combined as different sub-models and their outputs can be aggregated to compute the final conclusions.

4

Application

The application described in this work use flow and occupancy rate data to automatically select messages to support operators' evaluation. One of the data sets used in this application is shown in Fig. 2. The interpolation of the data represented by the solid line shows a typical relationship between flow and density. Density is not directly measured but it can be approximated by the occupancy rate. For low occupancy rate and flow values, traffic condition is classified as clearly “fluid”, since the road is being operated under its capacity. As flow increases the occupancy rate also increases until a saturation point, represented by the limit of the road capacity, where the traffic is classified as “dense”. After such limit, the effects of congestion reduce the traffic flow, and thus the mean velocity, while the occupancy rates still increases. In this situation, the traffic condition is classified as “slow”.

Fig. 2. A training data set

Operators classify traffic conditions visually from camera images. Thus, for some close data values, the classification can be completely different, due to the subjectivity of operators' evaluation. The methodology presented above allowed to build different classifiers; three of them are discussed in this section. The first classifier is obtained considering flow/occupancy ratio as the input variable. Using this analytical relation, the rule base design becomes trivial, since a low mean velocity implies a slow traffic, a medium mean velocity implies a intense traffic

242

Alexandre G. Evsukoff and Nelson F. F. Ebecken

and high velocity implies a fluid traffic. The membership functions were centered in the mean values of flow/occupancy ratio for each class in the training set. The output class membership vector was computed simply by fuzzification. The operators have agreed that flow can be described by three terms and occupancy is better described by four symbolic terms. Fuzzy meanings have been defined by equally spaced fuzzy sets with triangular membership functions. No optimization of fuzzy sets location was performed. The second classifier was obtained considering features separately as a sub-model in single antecedent rules like (2). The rule base weights were computed as (9) for each feature. The partial conclusions were aggregated by the t-norm “minimum” operator. The third classifier considers all the combination of terms in the variables' descriptor set in two-variable antecedent rules like (5). The fourth classifier is a standard Bayes classifier considering normal probability distribution. The collection of the data set was not a simple task. As the operators have many other occupations than monitor traffic conditions, each sample data (acquired at 1/15 minutes rate) was included into the data set only if the operator has just changed the message in the VMS. One data set was collected for each route. As routes are different in capacity, a different classifier was also developed for each route. The data sets were divided into a training data set and a test data set. The classification error rates in the test data set are shown in Table 2. The large error rates in the third route are due to inconsistent data in the training set. Table 2. Classification Results

Classifier #1 #2 #3 #4

Route 1 34.78 31.87 35.16 32.96

Route 2 27.83 31.19 23.85 31.19

Route 3 47.83 46.09 43.48 44.34

In the real VMS application, many detectors are available for each route. In this case, the solution provided by each data collection station should be considered as an independent partial conclusion and final decision could be obtained by weighted aggregation of all data collection stations in each route, in a data fusion scheme.

5

Conclusions

This work has presented a fuzzy system approach to represent knowledge elicited from experts and mined from a data set. The main advantage of this approach is its flexibility to combine single or multiple-variable fuzzy rules and to derive fuzzy rules from heuristic and/or experimental knowledge. In the application, three classifiers were discussed for a simple but real application of a traffic information system. It has been shown that the operator evaluation from visual inspection is very subjective and corresponds roughly to measured data. The classification results reflect the lack of standardization in operators' evaluation. The results obtained with fuzzy classifiers (# 1 to # 3) were more or less similar

Mining Fuzzy Rules for a Traffic Information System

243

to the results of the Bayes classifier (#4), although the rule bases in fuzzy classifiers are readable by domain experts. The rule bases obtained in the data-driven classifiers (classifiers #2 and #3) correspond roughly to the operators' expectation on how flow and occupancy are related to traffic condition, expressed in the heuristic classifier (classifier #1). An automatic procedure to support operators' evaluation could help to standardize the messages to be displayed in VMS, allowing a better understanding of traffic messages by the drivers. The parameters of the fuzzy partitions for each variable are important design issues that can improve classification rates and have not been considered in this work. It is considered for further development of the symbolic approach developed in this work.

Acknowledgements The authors are grateful to the kind co-operation of the traffic control operators and to CET-Rio that has provided the data for this work.

References [1] [2] [3] [4] [5] [6]

Dudek, C. L. (1991). Guidelines on the Use of Changeable Message Signs – Summary Report. Publication No. FHWA-TS-91-002. U.S. Department of Transportation, Federal Highway Administration. Evsukoff, A.; A. C. S. Branco and S. Gentil (1997). A knowledge aquisition method for fuzzy expert systems in diagnosis problems. Proc. 6th IEEE International Conference on Fuzzy Systems – FUZZIEEE'97, Barcelona. Iserman, R. (1998). On fuzzy logic applications for automatic control, supervision and fault diagnosis. IEEE Trans. on Systems Man Cybernetics – Part A: Systems and Humans, 28 (2), pp. 221-234. Sethi, V. and N. Bhandari (1994). Arterial incident detection using fixed detector and probe vehicle data. Transportation Research. Zadeh, L. (1996). Fuzzy logic = computing with words. IEEE Trans. on Fuzzy Systems, 4 (2), pp. 103-111. Zimmermann, H.-J. (1996). Fuzzy Set Theory and its Applications, Kluwer.

Possibilistic Hierarchical Fuzzy Model Paulo Salgado Universidade de Trás-os-Montes e Alto Douro 5000-911 Vila Real, Portugal [email protected]

Abstract. This paper presents a Fuzzy Clustering of Fuzzy Rules Algorithm (FCFRA) with dancing cones that allows the automatic organisation of the sets of fuzzy IF … THEN rules of one fuzzy system in a Hierarchical Prioritised Structure (HPS). The algorithm belongs to a new methodology for organizing linguistic information, SLIM (Separation of Linguistic Information Methodology), and is based on the concept of relevance of rules. The proposed FCFRA algorithm has been successfully applied to the clustering of an image.

1

Introduction

A fuzzy reasoning model is considered as a set of rules in the IF-THEN form to describe input-output relations of a complex system. We use rules to describe this relation instead of classical function approximation techniques mainly due to the transparency of the resulting fuzzy model. When building fuzzy systems from experts or automatically from data we need some procedures that divide the input space into fuzzy granules. These granules are the building blocks for the fuzzy rules. To keep interpretability we usually require that the fuzzy sets be specified in local regions. If this requirement is not fulfilled, many rules must be applied and aggregated simultaneously, such that the final result becomes more difficult to grasp. Moreover, aiming for a high approximation quality, we tend to use a large number of rules. In these last cases, it is not possible to interpret a fuzzy system any longer. The main objective of this article is to show that automated modelling techniques can be used to obtain not only accurate, but also transparent rule-based models from system measurements. This is obtained by organizing the flat fuzzy system f(x) into a set of n fuzzy sub-systems f1(x), f2(x), ..., fn(x), each corresponding to a readable and interpretable fuzzy system that may contain information related with particular aspects of the system f(x). This objective can be reached by using an algorithm that implements Fuzzy Clustering of Fuzzy Rules (FCFRA) with dancing cones [1]. The proposed algorithm allows the grouping of a set of rules into c subgroups (clusters) of similar rules, producing a HPS representation of the fuzzy systems. The paper is organized as follows. A brief introduction of the hierarchical fuzzy systems HPS is made and the concept of relevance is reviewed in section 2. In sec-

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 244-250, 2003.  Springer-Verlag Berlin Heidelberg 2003

Possibilistic Hierarchical Fuzzy Model

245

tion 3 the FCFR strategy is proposed. An example is presented in section 4. Finally, the main conclusions are outlined in section 5.

2

The Hierarchical Prioritized Structure

The SLIM methodology organizes a fuzzy system f(x) as a set of n fuzzy systems f1(x), f2(x), …, fn(x). A clustering algorithm is used in this work to implement the separation of information among the various subsystems [2]. The HPS structure, depicted in figure 1, allows the prioritization of the rules by using a hierarchical representation, as defined by Yager [3]. If i < j the rules in level i will have a higher priority than those in level j. Consider a system with i levels, i=1,…, n-1, each level with Mi rules: I. II.

If U is Aij and Vˆi −1 is low, then Vi is Bij and rule II is used V1 is Vˆ i −1

Rule I is activated if two conditions are satisfied: U is Aij and Vˆi −1 is low. Vˆi −1 , which is the maximum value of the output membership function of Vi-1, may be interpreted as a measure of the satisfaction of the rules in the previous levels. If these rules are relevant, i.e. Vˆi −1 is not low, the information conveyed by these rules will not be used. On the other hand, if the rules in the previous levels are not relevant, i.e. Vˆ is i −1

low, this information is used. Rule II states that the output of level i is the union of the output of the previous level with the output of level i.

Fig. 1. One Hierarchical Prioritized Structure (HPS)

The fuzzy system of figure 1 is a universal function approximator. With the appropriated number of rules it can describe any nonlinear relationship. The output of a generic level i is given by the expression [4]: Mi   Gi =  (1 − α i −1 ) ∧ ∪ Fi l  ∪ Gi −1 l =1  

(1)

246

Paulo Salgado

where Fi l = Ail ( x* ) ∧ Bil , l = 1,

, M i is the output membership function of rule l in

level i. The coefficient αi translates the relevance of the set of rules in level i (for level 0: α0=0, G0 =∅). Equation (1) can be interpreted as an aggregation operation of the relevance, for the hierarchical structure. For the characterization of the relative importance of sets of rules, in the modelling process, it is essential to define a relevance function. Depending on the context where the relevance is to be measured, different metrics may be defined [5]. Definition 1. Consider ℑ a set of rules from U into V, covering the region S = U × V in the product space. Any function of relevance must be of the form ℜ S : P ( ℑ ) → [ 0 , 1]

(2)

where P ( ℑ) is the power set of ℑ. As a generalization of equation (1), we propose a new definition of relevance for the HPS structure [5]. Definitions 2. (Relevance of fuzzy system just i level) Let Si be the input-output region covered by the set of rules of level i. The relevance of the set of rules in level i is defined as:

(

αi = S αi −1 , T (1 − αi −1 ) , ℜS  i

({

where ℜSi = S ℜSi ( Fi l ) , l = 1, , M i

)

(3)

}) and S and T are, respectively, S-norm e T-

norm operations. ℜSi ( Fi l ) represents the relevance of rule l in level i.

Using the product implication rule, and considering Bil a centroid of amplitude δil centred in y=ÿ, then ℜ Si ( Fi l ) = Ail ( x* ) ⋅ δ il

(4)

When the relevance of a level i is 1, the relevance of all the levels below is null. The relevance of the set of rules just i level, as definition 2, is here simplified by using the following expression:

αi −1 + ℜS ; if αi −1 + ℜS ≤ 1 αi =  i

1

i

; otherwise

(5)

Mi

where ℜSi = ∑ ℜSi ( Fi l ) . l =1

In next section we present a new method to cluster the fuzzy rules of a fuzzy system. The result of this process corresponds to transforming the original fuzzy system (a flat structure) into a HPS fuzzy model.

Possibilistic Hierarchical Fuzzy Model

3

247

The Possibilistic Clustering Algorithm of Fuzzy Rules

The objective of the fuzzy clustering partition is the separation of a set of fuzzy rules ℑ={R1, R2,..., RM} into c clusters, according to a “similarity” criterion, finding the optimal clusters centre, V, and the partition matrix, U. Each value uik represents the membership degree of the kth rule, Rk, belonging to the ith cluster i, Ai, and obeys simultaneously to the equations (6) and (7): n

0 < ∑ uik < n, ∀i ∈ { 1, 2,L , c}

(6)

k =1

and c

M

∑∑ ℜ ( x ) ⋅ u l

k

il

⋅ α i = 1 , ∀xk ∈ S

(7)

i =1 l =1

Equation (7) can be interpreted as the sum of the products between the relevance of the rules l in the xk point with the degree of the rule l belonging to the cluster i, uil α i . This last item reflects the joint contribution of rule l to the ith hierarchical system, uil , with the relevance of the hierarchical level i, α i . Thus, for the Fuzzy Clustering of Fuzzy Rules Algorithm, FCFRA, the objective is to find a U=[uik] and V = [v1 , v2 ,L , vC ] with vi ∈ R p where: n m  c M J (U , V ) = ∑  ∑∑ ( uil ℜl ( xk ) ) xk − vi k =1  i =1 l =1

2 A

m + ηi (1 − uil )  

(8)

is minimized, with a weighing constant m>1, with constrain of equation (7). The ⋅ is an inner product norm x

2 A

= x T A x . The parameter ηi is fixed for each cluster with

membership distance of ½. It can be shown that the following algorithm may lead the pair (U*,V*) to a minimum. The models specified by the objective function (8) were minimized using alternating optimization. The results can be expressed in following algorithm:

Possibilistic Fuzzy Clustering algorithms of fuzzy rules – P-FCAFR Step 1: For a set of points X={x1,..., xn}, with xi∈S, and a set of rules ℑ={R1, R2,..., RM}, with relevance ℜl ( xk ) , k= 1, … , M, keep c, 2 ≤ c < n, and initialize U(0)∈ Mcf. Step 2: On the rth iteration, with r= 0, 1, 2, ... , compute the c mean vectors.  ∑ ( u ( ) ) ⋅ ∑ ( ℜ ( x ) ) M

vi( ) = r

l =1



r il

m

 (r ) ∑  uil l =1  M

( )

 ⋅ xk  k =1  , where u ( r )  = U ( r ) , i=1, ... ,c. np  il  m m ⋅ ∑ ( ℜl ( x k ) )  k =1  np

l

m

k

(9)

248

Paulo Salgado

Step 3: Compute the new partition matrix U(r+1) using the expression: uil( r +1) =

1  ∑  ∑ (ℜ ( x )) np

c

j =1

 k =1

l

m

k

⋅ xk − vi( r )



ηi 

2 m −1

, with 1≤ i ≤ c , 1≤ l ≤ M.

(10)



A

In this algorithm, we interpret the values of {ui ( xk )} (discrete support) as observations from the extended membership function µi ( l ) : µi ( l ) = ui ( l ) , l=1,…, M (continuous support). Step 4: Compare U(r) with U(r+1): If || U(r+1)-U(r)|| < ε then the process ends. Otherwise let r = r+1 and go to step 2. ε is a small real positive constant. The applications of the P-FCAFR algorithm to fuzzy system rules with singleton fuzzifier, inference product and centroid defuzzifier result in a fuzzy system with HPS structure, i.e., in the form: c M  f ( x ) = ∑ α i  ∑ θ l ⋅ µ l ( x ) ⋅ uil  i =1  l =1 

 ∑ α  ∑ µ ( x ) ⋅ u c

M

i

i =1

l

il

l =1

  

(11)

If the rules describe one region S, instead of a set of points, and the membership of relevance function of the rules was symmetrical the equation (10) will be reformulate ( r +1)

µil

=1

∑( c

j =1

(r )

xl − vi

A

η

2 * m −1 i

)

(12)

where xl is the centre of rule l. The shapes of this membership function are determined by the update of below iterative procedure in order to minimize the objective function(8). However, the user might be interested in choosing memberships functions shapes that are considered more useful for a given application, although he abandons the objective function. The P-FCAFR algorithm and the Alternating Cluster Estimation Fuzzy rules [1][6], with Cauchy membership function, were used in the identification of the Abington Cross.

4

Experimental Results

The Abington Cross image (Fig. 2a) was used to illustrate the proposed approach. It is a critical image for obtaining the cross clustering due to the high amount of noise. The image is a 256×256 pixels, grey level image. The intensity dynamic range has been normalized to the [0,1] interval. The image is a set of R3 p points, with coordinates (x,y,I), where I (x,y) is the intensity of image at point (x, y). I = 0 corresponds to the white colour, and I = 1 corresponds to the black colour. The aim is to partition the fuzzy system into five clusters. In the first step, the system is modelled in a set of rules, using the nearest neighbourhood method [7]. The

Possibilistic Hierarchical Fuzzy Model

249

resulting system after the learning process has 1400 fuzzy rules. The output of the system at this stage is shown in Fig. 2.b. (b)

(a)

Fig. 2. Grey scale image, with the shape of Abington Cross: a) original image. b) Fuzzy image

The second step consists in the segmentation of the image into 5 clusters (with m=2), each representing a fuzzy system in a HPS structure, using the FCAFR algorithm presented in section 3. Four of them represent the background (each representing the background rectangle in each corner of the image) and one represents the area containing the white cross. The degree of membership of each rule to each rule cluster has been measured using the weighed distance of the centre of the rule to the centre of the cluster, η. Figs. 3.a.-3.e. show the individual outputs response of each hierarchical fuzzy model. The original image can be described as the aggregation (equation (11)).

5

Conclusions

In this work, the mathematical fundaments for Possibilistic fuzzy clustering of fuzzy rules were presented, where the relevance concept has a significant importance. Based in this concept, it is possible to make a fuzzy clustering algorithm of fuzzy rules, which is naturally a generalization of fuzzy possibilistic clustering algorithms. Now, many others clusters algorithms (crisp) can be applied to the fuzzy rules. The P-FCAFR Algorithm organizes the rules of fuzzy systems in the HPS structure, based on the SLIM methodology, where the partition matrix can be interpreted as containing the values of the relevance of the sets of rules in each cluster. The proposed clustering algorithm has demonstrated the ability to deal with an image strongly corrupted by noise, the Abington Cross image. The test image was first identified using the Nearest Neighbourhood Method. The resulting fuzzy rules were successfully grouped into 5 clusters using the proposed P-FCAFR Algorithm. Each resulting cluster represents one meaningful region of the image. This helps in the corroboration of the proposed approach.

250

Paulo Salgado

(a)

(c)

(b)

(d)

(e)

Fig. 3. Images representing the output result of the five background clusters

References [1] [2] [3] [4] [5] [6] [7]

Hoppner, Frank, and al.: Fuzzy Cluster Analysis, Methods for classification data analysis and image recognition, Wiley, (1999). Salgado, Paulo, Clustering and hierarchization of fuzzy systems, submitted to Soft Computer Journal. Yager, R.: On a Hierarchical Structure for Fuzzy Modeling and Control, IEEE Trans. On Syst., Man, and Cybernetics, 23, (1993), 1189-1197. Yager, R.: On the Construction of Hierarchical Fuzzy Systems Models, IEEE Trans. On Syst., Man, and Cyber. –Part C, 28, (1998), 55-66. Salgado, P.: Relevance of the fuzzy sets and fuzzy systems. In: “Systematic Organization of Information in Fuzzy Logic”, NATO Advanced Studies Series, IOS Press, In publication. Runkler, T. A, Bezdek, C.: Alternating Cluster Estimation: A new Tool for Clustering and Function Approximation, IEEE Trans. on Fuzz. Syst., 7, (1999), 377-393. Wang, Li-Xin: Adaptive Fuzzy System and Control, Design and stability analysis, Prentice Hall, Englewood Cliffs, NY 07632 (1994).

Fuzzy Knowledge Based Guidance in the Homing Missiles Mustafa Resa Becan and Ahmet Kuzucu System Dynamics and Control Unit, Mechanical Engineering Faculty Istanbul Technical University, Turkey

Abstract. A fuzzy knowledge based tail pursuit guidance scheme is proposed as an alternative to the conventional methods. The purpose of this method is to perform the tracking and interception using a simplified guidance law based on the fuzzy controller. Noise effects in homing missile guidance must be reduced to get an accurate control. In this work, developed fuzzy tail pursuit control algorithm reduces the noise effects. Then, a knowledge based algorithm is derived from the fuzzy control behavior to obtain a simplified guidance law. Simulation results show that the new algorithm led to satisfactory performance for the purpose of this work.

1

Introduction

The tail pursuit is the one of the most common conventional guidance methods in the homing missile area. But, the homing missile guidance has some uncertain parameters for the target maneuver and target behavior is observed through noisy measurements. In these cases, conventional guidance methods may not be sufficient to obtain the tracking and interception. Fuzzy control has suitable properties to eliminate such difficulties. Fuzzy controller has been used in the systems where variables have fuzzy characteristics or deterministic mathematical models are difficult or impossible [1], [2]. Recently, developed neuro-fuzzy techniques serve as possible approaches for the nonlinear flight control problems [3-5]. However, a limited number of papers have been adressed to the issue of fuzzy missile guidance design [6-8]. Fuzzy tail pursuit guidance is designed considering the noise sourced from the thermal and radar detection sources at the system input. This noise affects the guidance system entirely. Fuzzy tail pursuit (FTP) guidance is applied to a homing missile with known aerodynamic coefficients and their variations. The purpose of this paper is not only to perform the FTP application but also is to search a simplified knowledge based guidance inspired by the fuzzy control results. The objective of this research is to obtain a rule based low cost guidance control free from the computational complexities that a full fuzzy controller requires.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 251-257, 2003.  Springer-Verlag Berlin Heidelberg 2003

252

Mustafa Resa Becan and Ahmet Kuzucu

2

Definition of the Guidance Control System

Figure 1 shows the interception geometry for the missile model and it presents some symbols used in this study. Missile motions are constrained within the horizontal X-Y plane. State variables from Fig.1 are the flight path angle γ , missile velocity Vm, attitude angle θ, angle of attack α , pitch rate q, missile and target positions in the inertial space Xm, Ym, XT, YT and the target angle β. The missile is represented by an aerodynamic model with known coefficients.

Fig. 1. Interception Geometry

2.1

Conventional Tail Pursuit (CTP) Design

Line of sight (LOS) is the distance between the missile and target. The purpose of the tail pursuit guidance is to keep the flight path angle γ equal to the line of sight angle σ. Τhis goal leads to the design of a closed loop control around LOS angle.

Fig. 2. Block Diagram of CTP

Fig. 2 shows the simple block diagram of CTP. Controller inputs are the error e and first derivative of error e& . Noise effect is symbolized as w and σ& is the variation of LOS angle. The CTP output can be written in PD form as shown below: u = K p .e + K v .e&

(1)

where Kp and Kv are the control coefficients chosen for a desired system performance.

Fuzzy Knowledge Based Guidance in the Homing Missiles

3

253

Fuzzy Tail Pursuit (FTP) Design

Target position measurement is not precise and some noise effect is superposed to it. This particularity allows the fuzzy tail pursuit as an alternative to the conventional version. The noise is modeled as a gaussian density function in this study . The input and output variables of the fuzzy controller are linguistic variables taking linguistic values. The input linguistic variables are the error ( e ) and change of error ( e& ) and the linguistic output variable is the control signal u. The linguistic variables are expressed by linguistic sets. Each of these variables is assumed to take seven linguistic sets defined as negative big (NB), negative medium (NM), negative small (NS), zero (ZE), positive small (PS), positive medium (PM), positive big (PB). Triangular membership functions (MF) for error input are scaled in synchronization with the gaussian density function distribution shown on Fig. 3. This approach is preferred to match the membership function distribution with the noisy measurement distribution. Universe of discourse of MF (e/emax) is choosen in regular form at first, but the satisfactory results could not be obtained. Then it is changed using trial and error to provide the accordance with the density function and performed the best results. Consequently, the most reasonable way to design MF is determined as presented on Fig.3.

Fig. 3. Density Function and MF Design

A set of fuzzy PD rules has been performed in a real time application [9]. Table 1 shows the fuzzy PD rules set following this approach.

254

Mustafa Resa Becan and Ahmet Kuzucu Table 1. Rule Table of Fuzzy Tail Pursuit

Minimum (Mamdani type) inference is used to obtain the best possible conclusions [8]. This type of inference allowed easy and effective computation and it is appropriate for real time control applications.The outputs of the linguistic rules are fuzzy, but the guidance command must be crisp. Therefore, the outputs of the linguistic rules must be defuzzified before sending them to the plant. The crisp control action is calculated here using the center of gravity (COG) defuzzification method.

4

Control Applications

CTP and FTP strategy is applied in PD form to a homing missile model. Aerodynamic coefficients are evaluated from tabulated experimental values in the simulation environment. Fuzzy control coefficients are scaled according to the system limitations. Fig. 4 display the interception trajectory and LOS performance of the application. Results show that the FTP perform the interception without time delay.

Fig. 4. FTP Behaviors

Fuzzy Knowledge Based Guidance in the Homing Missiles

255

Fig. 5. Comparison of the Control Behaviors

Control behavior obtained by CTP and FTP are shown on Fig. 5. PD controller coefficients have been determined considering a second order system model with a natural freqency of wn= 6 Hz. and a damping ratio of ζ = 0.7 in CTP application. It is clear that better noise filtering performance may be obtained using prefilters or more sophisticated PD type controller design, so the comparison may seem unfair. But still the improvement in noise filtering of the fuzzy controller is impressing. Taking into account the distribution of measurements instead of instantaneous measurement during fuzzification , and the use of COG method in defuzzification both having some ‘averaging' property, reduce considerably the propagation of noise to the control variable behavior. On the other hand it is clear that a pure theoretical derivative effect in the CTP scheme will amplify the noise effect if a suitable filtering is not added to controller for the frequency band under consideration.

5

Derivation of Knowledge Based Control

Homing missiles are launched for once only. Therefore low cost controllers are needed in such applications. A knowledge based algorithm is developed using the fuzzy results to obtain a guidance controller less equipped than the full fuzzy control. Our goal is to obtain a guidance scheme related to the fuzzy tail pursuit, a new algorithm requiring less sophisticated computing power. Fig. 6 shows the block diagram of the proposed guidance control system.

Fig. 6. Knowledge Based Control Block Diagram

256

Mustafa Resa Becan and Ahmet Kuzucu

Fuzzy surface of fuzzy tail pursuit controller obtained through off-line simulation studies is shown on Fig. 7 Membership functions for knowledge based control are rearrenged using this surface. Table 2 presents the rules rewritten using the FTPcontrol behavior on Fig. 5 Control values are crisp and changed from –3 to +3 to match the control variable variation on Fig. 5. Fig. 7 shows that the derivated guidance law performed the interception within 7.55 s. using the proposed knowledge based discontinuous control. This value is very close to the interception time obtained with the fuzzy tail pursuit guidance.

Fig. 7. Fuzzy Surface Between the Input-Output Values

Table 2. Knowledge Based Rules

Fig. 8. Knowledge Based Guidance Behaviors

Fuzzy Knowledge Based Guidance in the Homing Missiles

6

257

Conclusion

A fuzzy tail pursuit guidance for a homing missile has been presented in this work. The simulation results showed the proposed method achieved a relatively noise free guidance without interception time delay using the filtering property of the fuzzy control. Then, a knowledge based guidance law is derived based on the fuzzy tail pursuit results, to obtain a simple discontinuous control. This work shows that the guidance can be performed using knowledge based algorithm because the line of sight behavior is quite satisfactory. The results encourage future studies through the fuzzy tail pursuit and knowledge based control on the homing missile guidance area.

References [1] [2] [3]

[4] [5] [6] [1] [8] [9]

Lee C.C., “Fuzzy Logic in Control Systems: Fuzzy Logic Controller: Part I & Part II”, IEEE Trans. Syst. Man and Cybernetics, 20, 404-435, 1990 Williams T., “Fuzzy Logic Simplifies Complex Control Problems”, Computer Design, 90- 102, 1991 Huang C., Tylock J., Engel S., and Whitson J., “Comparison of Neural Network Based , Fuzzy Logic Based, and Numerical Nonlinear Inverse Flight Control”, Proceedings of the AIAA Guidance, Navigation and Control Conference, AIAA Washington DC, 1994, pp. 922-929 Geng Z.J., and McCullough C.L., “Missile Control Using Fuzzy Cerebellar Model Arithmetic Computer Neural Networks”, Journal of Guidance, Control and Dynamics, Vol. 20, No.3, 1997, pp. 557-565 Lin C.L., and Chen Y.Y., “Design of an Advanced Guidance Law Against High Speed Attacking Target”, Proceedings of the National Science Council, Part A, Vol.23, No.1, 1999, pp.60-74 Mishra K., Sarma I.G., and Swamy K.N., “Performance Evaluation of the Fuzzy Logic Based Homing Guidance Schemes”, Journal of Guidance, Control and Dynamics, Vol.17, No.6, 1994, pp. 1389-1391 Gonsalves P.G., and Çağlayan A.K., “Fuzzy Logic PID Controller for Missile Terminal Guidance”, Proceedings of 1995 IEEE International Symposium and Intelligent Control, IEEE Publications, Piscataway, NJ.1995, pp.377-382 Lin C.L., and Chen Y.Y., “Design of Fuzzy Logic Guidance Law Against High Speed Target”, Journal of Guidance, Control and Dynamics, Vol.23, No.1, January-February 2000, pp. 17-25 Hoang L.H., and Maher H., “Control of a Direct-Drive DC Motor by Fuzzy Logic”, IEEE Industry Applications Society, Vol. VI, 1993, pp. 732-738

Evolutionary Design of Rule Changing Cellular Automata Hitoshi Kanoh1 and Yun Wu2 1

Institute of Information Sciences and Electronics, University of Tsukuba Tsukuba Ibaraki 305-8573, Japan [email protected] 2 Graduate School of Systems and Information Engineering University of Tsukuba, Tsukuba Ibaraki 305-8573, Japan [email protected]

Abstract. The difficulty of designing cellular automatons' transition rules to perform a particular problem has severely limited their applications. In this paper we propose a new programming method of cellular computers using genetic algorithms. We consider a pair of rules and the number of rule iterations as a step in the computer program. The present method is meant to reduce the complexity of a given problem by dividing the problem into smaller ones and assigning a distinct rule to each. Experimental results using density classification and synchronization problems prove that our method is more efficient than a conventional one.

1

Introduction

Recently, evolutionary computations by parallel computers have gained attention as a method of designing complex systems [1]. In particular, parallel computers based on cellular automata (CAs [2]), meaning cellular computers, have the advantages of vastly parallel, highly local connections and simple processors, and have attracted increased research interest [3]. However, the difficulty of designing CAs' transition rules to perform a particular task has severely limited their applications [4]. The evolutionary design of CA rules has been studied by the EVCA group [5] in detail. A genetic algorithm (GA) was used to evolve CAs. In their study, a CA performed computation means that the input to the computation is encoded as the initial states of cells, the output is decoded from the configuration reached at some later time step, and the intermediate steps that transform the input to the output are taken as the steps in the computation. Sipper [6] has studied a cellular programming algorithm for non-uniform CAs, in which each cell may contain a different rule. In that study, programming means the coevolution of neighboring, non-uniform CAs' rules with parallel genetic operations. In this paper we propose a new programming method of cellular computers using genetic algorithms. We consider a pair of rules and the number of rule iterations as a V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 258-264, 2003.  Springer-Verlag Berlin Heidelberg 2003

Evolutionary Design of Rule Changing Cellular Automata

259

step in the computer program, whereas the EVCA group considers an intermediate step of transformation as a step in the computation. The present method is meant to reduce the complexity of a given problem by dividing the problem into smaller ones and assigning a distinct rule to each one. Experimental results using density classification and synchronization problems prove that our method is more efficient than a conventional method.

2

Present Status of Research

2.1

Cellular Automata

In this paper we address one-dimensional CAs that consist of a one-dimensional lattice of N cells. Each cell can take one of k possible states. The state of each cell at a given time depends only on its own state at the previous time step and the state of its nearby neighbors at the previous time step according to a transition rule R. A neighborhood consists of a cell and its r neighbors on either side. A major factor in CAs is how far one cell is from another. The CA rules can be expressed as a rule table that lists each possible neighborhood with its output bit, that is, the update value of the central cell of the neighborhood. Figure 1 shows an example of a rule table when k=2 and r=1. We regard the output bit “11101000” as a binary number. The number is converted to a decimal, which is 232, and we will denote the rule in Fig. 1 as rule 232. Here we describe CAs with a periodic boundary condition. The behavior of one-dimensional CAs is usually displayed by space-time diagrams in which the horizontal axis depicts the configuration at a certain time t and the vertical axis depicts successive time steps. The term “configuration” refers to the collection of local states over the entire lattice, and S(t) denotes a configuration at the time t.

Fig. 1. An example of a rule table (k=2, r=1)

2.2

Computational Tasks for CAs

We used the density classification and the synchronization tasks as benchmark problems [4, 7]. The goal for the density classification task is to find a transition rule that decides whether or not the initial configuration S(0) contains a majority of 1s. Here ρ(t) denotes the density of 1s in the configuration at the time t. If ρ(0) > 0.5, then within M time steps the CA should go to the fixed-point configuration of all 1s (ρ(t M) =1 ); otherwise, within M time steps it should produce the fixed-point configuration of all 0s (ρ(t M) = 0 ). The value of constant M depends on the task. The second task is one of synchronization: given any initial configuration S(0), the CA should reach a final configuration, within M time steps, that oscillates between all 0s and all 1s in successive time steps.

260

2.3

Hitoshi Kanoh and Yun Wu

Previous Work

The evolutionary design of CA rules has been studied by the EVCA group [3, 4] in detail. A genetic algorithm (GA) was used to evolve CAs for the two computational tasks. That GA was shown to have discovered rules that gave rise to sophisticated emergent computational strategies. Sipper [6] has studied a cellular programming algorithm for 2-state non-uniform CAs, in which each cell may contain a different rule. The evolution of rules is here performed by applying crossover and mutation. He showed that this method is better than uniform (ordinary) CAs with a standard GA for the two tasks. Meanwhile, Land and Belew [8] proved that the perfect two-state rule for performing the density classification task does not exist. However, Fuks [9] showed that a pair of human written rules performs the task perfectly. The first rule is iterated t1 times, and the resulting configuration of CAs is processed by another rule iterated t2 times. Fuks points out that this could be accomplished as an “assembly line” process. Other researchers [10] have developed a real-world application for modeling virtual cities that uses rule-changing, two-dimensional CAs to lay out the buildings and a GA to produce the time series of changes in the cities. The GA searches the sequence of transition rules to generate the virtual city required by users. This may be the only application using rule changing CAs.

3

Proposed Method

3.1

Computation Based on Rule Changing CAs

In this paper, a CA in which an applied rule changes with time is called a rule changing CA, and a pair of rules and the number of rule iterations can be considered as a step in the computer program. The computation based on the rule changing CA can thus operate as follows: Step 1: The input to the computation is encoded as the initial configuration S(0). Step 2: Apply rule R1 to S(0) M1 times; ……; apply rule Rn to S(M1+…+Mn-1) Mn times. Step 3: The output is decoded from the final configuration S(M1+…+Mn). In this case, n is a parameter that depends on the task, and rule Ri and the number of rule iterations Mi (i=1, …, n) can be obtained by the evolutionary algorithm described in the next section. The present method is meant to reduce the complexity of a given task by dividing the task into n smaller tasks and assigning (Ri, Mi) to the i-th task. 3.2

Evolutionary Design

Each chromosome in the population represents a candidate set of Ri and Mi as shown in Fig. 2. The algorithm of the proposed method is shown in Fig. 3.

Evolutionary Design of Rule Changing Cellular Automata

261

R1 | M1| ……… | Rn | Mn Fig. 2. Chromosome of the proposed method (Ri: rule, Mi: the number of rule iteration) procedure main initialize and evaluate population P(0); for g = 1 to the upper bound of generations { apply genetic operators to P(g) and create a temporary population P'(g); evaluate P' (g); get a new population P(g+1) by using P(g) and P'(g); } procedure evaluate for i = 1 to the population size { operate CA with the rules on the i-th individual; calculate fitness of the i-th individual; } Fig. 3. Algorithm of the proposed method (procedure main and procedure evaluate)

3.3

Application

In this section we describe an application that uses the CA with two rules for density classification and synchronization tasks. Chromosome Encoding In the case of the CA with two rules, a chromosome can generally be expressed by the left part in Fig. 4. The right part in Fig. 4 shows an example of the chromosome when k=2 and r=1. Each rule and the number of iterations are expressed by the output bit of the rule table and a nonnegative integer, respectively. R1 | M1| R2 | M2 = 1 1 1 0 1 0 0 0 | 45 | 0 1 0 0 1 0 0 1 | 25 Fig. 4. Example of the chromosome of the proposed method for the CA with two rules

Fitness Calculation The fitness of an individual in the population is the fraction of the Ntest initial configurations in which the individual produced the correct final configurations. Here the initial configurations are uniformly distributed over ρ(0)∈[0.0, 1.0]. Genetic Operations First, generate Npop individuals as an initial population. Then the following operations are repeated for Ngen generations. Step 1: Generate a new set of Ntest initial configurations, and calculate the fitness on this set for each individual in the population. Step 2: The individuals are ranked in order of fitness, and the top Nelit elite individuals are copied to the next generation without modification. Step 3: The remaining (Npop-Nelit) individuals for the next generation are formed by single-point crossovers between the individual randomly selected from Nelit

262

Hitoshi Kanoh and Yun Wu

elites and the individual selected from the whole population by roulette wheel selection. Step 4: The rules R1 and R2 on the offspring from crossover are each mutated at exactly two randomly chosen positions. The numbers of iterations M1 and M2 are mutated with probability 0.5 by substituting random numbers less than an upper bound.

4

Experiments

4.1

Experimental Methods

The experiments were conducted under the following conditions: the number of states k=2, the size of lattice N=149, and M1+M2=149 for the CA; the population size Npop=100, the upper bound of generations Ngen=100, the number of elites Nelit=20, and the number of the initial configuration for test Ntest=100 for the GA. 4.2

Density Classification Task (r=1)

Table 1 shows the best rules in the population at the last generation. The obtained rules agree with the human written rules shown by Fuks [9], which can perform this task perfectly. Table 1. Rules obtained by the genetic algorithm for the density classification task (r=1, k=2, N=149)

Experiment-1 Experiment-2 4.3

R1 184 226

M1 124 73

R2 232 232

M2 25 76

Density Classification Task (r=3)

Figure 5 shows the comparison of the proposed method with the conventional method that uses only one rule as the chromosome [5]. Each point in the figure is the average of ten trials. It is seen from Fig. 5 that the fitness of the former at the last generation (0.98) is higher than that of the latter (0.91). Table 2 shows the best rules obtained by the proposed method, and Fig. 6 shows examples of the space-time diagrams for these rules. The rules in Table 2 have been converted to hexadecimal. Table 2. The best rules obtained by the genetic algorithm for the density classification task (r=3, k=2, N=149)

R1 01000100111C0000 000004013F7FDFFB

M1 101

R2 97E6EFF6E8806448 4808406070040000

M2 48

Evolutionary Design of Rule Changing Cellular Automata

263

Fig. 5. The best fitness at each generation for density classification tasks (r=3, k=2, N=149)

Fig. 6. Examples of space-time diagrams for a density classification task (r=3, k=2, N=149)

4.4

Synchronization Task (r=3)

Figure 7 shows the results of the same comparison as in Fig 5 for the synchronization task. The fitness of the proposed method reaches 1.0 by the 3rd generation, whereas the conventional method requires more than 20 generations.

Fig. 7. The best fitness at each generation for the synchronization task (r=3, k=2, N=149)

264

Hitoshi Kanoh and Yun Wu

5

Conclusions

In this paper we proposed a new programming method of cellular computers. The experiments were conducted using a rule changing CA with two rules. In a forthcoming study, the performance of a CA with more than two rules will be examined.

References [1]

Alba, E. and Tomassini, M.: Parallelism and Evolutionary Algorithms. IEEE Transactions on Evolutionary Computation, Vol. 6, No. 5 (2002) 443-462. [2] Wolfram, S.: A New Kind of Science. Wolfram Media Inc. (2002). [3] Sipper, M.: The Emergence of Cellular Computing. IEEE Computer, Vol. 32, No. 7 (1999). [4] Mitchell, M., Crutchfield, J., and Das, R.: Evolving Cellular Automata with Genetic Algorithms: A Review of Recent Work. Proceedings of the First International Conference on Evolutionary Computation and Its Applications (1996). [5] Mitchell, M., Crutchfield, J. P., and Hraber, P.: Evolving Cellular Automata to Perform Computations: Mechanisms and Impediments. Physica D 75 (1994) 361-391. [6] Sipper, M.: Evolving of Parallel Cellular Machines: The Cellular Programming Approach. Lecture Notes in Computer Science Vol. 1194. Springer-Verlag (1997). [7] Das, R., Crutchfield, L., Mitchell, M., and Hanson, J.: Evolving Globally Synchronized Cellular Automata. Proceedings of the 6-th ICGA (1995) 336-343. [8] Land, M. and Belew, R.: NO Two-State CA for Density Classification Exists. Physical Review Letters 74 (1995) 5148. [9] Fuks, H.: Solution of the Density Classification Problem with Two Cellular Automata Rules. Physical Review E, 55(3) (1997) 2081-2084. [10] Kato, N., Okuno, T., Suzuki, R., and Kanoh, H.: Modeling Virtual Cities Based on Interaction between Cells. IEEE International Conference on Systems, Man, and Cybernetics (2000) 143-148.

Dynamic Control of the Browsing-Exploitation Ratio for Iterative Optimisations L. Baumes1, P. Jouve1, D. Farrusseng2, M. Lengliz1, N. Nicoloyannis1, and C. Mirodatos1 1

Institut de Recherche sur la Catalyse-CNRS 2, Avenue Albert Einstein - F-69626 Villeurbanne Cedex France 2 Laboratoire Equipe de Recherche en Ingénierie des Connaissances Univ. Lumière Lyon2 -5, Avenue Pierre Mendès-France 69676 Bron Cedex France [email protected]

Abstract. A new iterative optimisation method based on an evolutionary strategy. The algorithm is proposed, which proceeds on a binary search space, combines a Genetic Algorithm and a knowledge extraction engine that is used to monitor the optimisation process. In addition of the boosted convergence to the optima in a fixed number of iterations, this method enables to generate knowledge as association rules. The new algorithm is applied for the first time for the design of heterogeneous solids in the frame of a combinatorial program of catalysts development.

1

Introduction

Today, the Genetic Algorithm (GA) approach is a standard and powerful iterative optimisation concept mainly due to its unique adaptation skills to highly diverse contexts. In a sequential manner, it consists at designing population of individuals that shall exhibit superior fitness, generation after generation. When it proceeds, the population shall focus on the optima of the space parameters. The optimisation process is over when criteria on the objective function are achieved. Very recently, this approach has been successfully applied for the optimisation of advanced materials (e.g. catalysts) for the petro-chemistry [1,2]. In this frame, the population consists at around 50 catalysts that are prepared and tested by automated working stations within a few days [3]. In addition of a quick optimisation demand, the total number of iteration was imposed. In this case, convergence of GAs may not be reached due to the "time" constraint. In turn, we face a particular problem: “Knowing that the number of iterations is low and a priori fixed, can efficient optimisations be carried out?” Thus, our strategy is to “act” on the stochastic character of genetic operators in order to guide the GA, without any modification of its algorithmic structure. It comes clear that the relevance of the population design shall be closely linked to the V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 265-270, 2003.  Springer-Verlag Berlin Heidelberg 2003

266

L. Baumes et al.

remaining time (e.g. numbers of iterations left). Indeed the Browsing-Exploitation ratio shall be monitored by the number of iterations left that allows the algorithm to get the best out of all the iterations at its disposal. Except Case Based Reasoning [4] and Supervised Learning methods [5] a very limited numbers of works deal for this purpose. This work presents a learning method based on extraction of equivalence classes, which accumulates strategic information on the whole previously explored search space, and that enables the dynamic control of the Browsing/Exploitation ratio.

2

Context and of the Catalysts Developments and Optimisation Requirements

Because industrial equipments are designed to prepare and test collection of catalysts (called libraries), iterative optimisation and more specifically the evolutionary approach is very well suited. The goal is to optimise properties of catalysts (produce quality). The optimisation cycle consists at designing a new population of individuals (catalysts) in light of previous individuals testing. In the context of heterogeneous catalysis, it does exist neither strong general theories nor empirical models that allow optimising a catalysts formulation for new specificity and requirements in a straightforward manner. Moreover, the probable behaviour of individuals cannot be estimated by simulation. In this study, we face the problem that the number of iterations is a priori fixed and fixed at a low value. It is not the purpose of this study to seek efficient algorithms that reduce the number of iterations. In addition, since the synthesis and testing of catalysts takes half-day to a few days, the calculation time is not a requirement. The context can be depicted in an industrial design process (Figure 1). Final Product

Specifications (compounds, processes, knowledge, objective function, time, …)

Cycle end Yes/No

Population Design

Population Production

Data analysis & decision

Population Test

Fig. 1. Iterative Design Process

On a general point of view, the data analysis and decision step, which follows each population test, emphases new guidelines for individuals improvement and also to update previous guidelines, which have been already considered. Thus, knowledge about the individuals (catalysts) and their properties (yields) is created, up-dated in the course of the screening and re-injected in the design process. In turn, each population design takes benefits of previous experiments. The knowledge mainly

Dynamic Control of the Browsing-Exploitation Ratio for Iterative Optimisations

267

arises from the discovery of relationships between catalysts variables and the respective yields. In addition, the discovery of relationships between the properties of the variables and the yields can provide occasionally very useful guidelines. Indeed, it allows passing form categorical variables to such as the elements of the Mendeleeï v periodic table to continuous variables such as their respective electronegativity values. When introducing these additional information at the starting point, the optimisation process can be boosted because of the wider knowledge accumulated. Because of the very limited iterations number, usual optimisation methods such as GAs, would not be efficient to address this issue. Furthermore, GAs do not take into account the information linked to the catalysts (variables properties) so that the knowledge that would be extracted in the course of the screening could not be re-injected in the optimisation process. In addition, the stochastic feature of genetic operators can become a major problem when the number of iteration is very limited. In the industrial process, the browsing on the search space shall be promoted at the early stage of the screening whereas the exploitation shall take more and more priority when the end of the optimisation process is approaching. The genetic operators work simultaneously on both browsing and exploitation. However, this BrowsingExploitation ratio cannot be directly and dynamically controlled by usual AG in the course of the screening.

3

A New Hybrid Algorithm for Iterative Optimisation

In order to extract and manage information in the optimisation cycle, we have developed a new approach that consists at hybridising a GA with a Knowledge Based System (KBS), which extracts empirical association rules on the whole set of previously tested catalysts and re-injects these strategic information to guideline the optimisation process. The guideline is performed by monitoring the Browsing/Exploitation ratio with the KBS, which enables to consider dynamically the number of iterations. 3.1

AG and Knowledge-Based System Search Spaces

The algorithm operates with all kinds of GAs. The GA search space can consider ordered and non-ordered qualitative variables that can be are either pure binary variable (presence or absence of elements), or discretized continuous variable coded by a set of binary value (element concentration) or nominal variable coded by a set of binary value (Figure 2). The KBS search space includes the GA search space with additional information, which are inherent features of the GA variables such as the size of the pore or the elemental composition of discrete type of zeolites. Table 1. GA and KBS representation GA search space P re pa ra tion va ria ble s Ze olite Ca tion/m e thod KL ZSM-5 NaY NaX Na-beta Li/imp Li/ie 1 0 1 0 0 0 1 0 2 0 0 0 1 0 0 1 3 1 0 0 0 1 0 0

Extended search space … … … … …

Ca tion cha rge "+1" "+2" 1 0 1 0 0 1

Additionna l va ria ble s O-ring Ele ctrone ga tivity 10 12 0.79-0.82 0.89-0.95 0.98-1 1.31 1 0 0 0 1 0 0 1 0 0 1 0 1 1 1 1 0 0

… … … … …

268

L. Baumes et al.

3.2

The Knowledge-Based System

The mechanism of the KBS cannot be extensively depicted in this extended abstract; details can be found elsewhere [6-8]. The KBS search space is binning in different zones that form a lattice. The space zone extraction consists in obtaining equivalence classes by the maximum square algorithm . Each of these zones is characterized by two coefficients B and E, which are characteristics of the Browsing and Exploration level, respectively. The relevance R of an individual x that is in a zone Zi is calculated a function of B and E. We define the followings:

R(h1 ( x), h2 ( x), t , µ) with h = ( f o g )

h1 = f1 (B(Zi )), h2 = f 2 (E (Zi )) and B(Zi ) = g1 ( x, t , µ),

E(Zi ) = g 2 ( x, t , µ)

For example, we can define B, E and R as below:

B( Z i ) =

(number of individuals x previously tested in Zi ) card(Zi )

E ( Z i ) = mean fitness of individuals x previously tested in Zi R(xk ) =

∑ (E (Z ) × (1 − B(Z )) i

i

∀Z i , xk ∈ Z i

i

In this example R is actually the Browsing-Exploitation ratio, others more complex function can be used. The total number of iteration (µ) can be here injected. The Browsing coefficient function enables to estimate the “real” diversity of an individual with respects to the occupancy rates of all the classified zones and quantifies its contribution to the extent of browsing of the total search space. The second coefficient E that evaluates the performances, takes into account the results of all individuals x in the same classification zones (for example the mean value). This second function quantifies the relevance of an individual for improving exploitation. In order to get a unique selection criteria, a third coefficient which combines the two former ones and the number of remaining individuals still to be tested t is defined R(h1(x),h2(x),t). It enables to evaluate the global relevance of an individual with respect to its interest for both the browsing and exploitation. In this manner, the relevance of individuals constituting each new generation can be increased and controlled as a function of time. 3.3

Hybridization Mechanism

The individual that exhibits the highest Relevance shall constitute the next generation. Within the traditional framework, GAs take as input a population of size K and proposes a new population of identical size (figure 4). In order to increase the number of individuals in the population of output (size K) with high T(x) values, one forces the GA to propose a number µK (µ>1, µ∈IN) of individuals, among which K valuers showing the highest relevance R are selected. The choice of the various functions f1, f2, g1, g2 and R is of major importance and completely defines the manner of controlling the optimisation process.

Dynamic Control of the Browsing-Exploitation Ratio for Iterative Optimisations Initial population

269

K individuals selection of K individuals by the KBS

Evaluation

µ x K individuals

Final population of K individuals

end criteria GA

Fig. 2. Mechanism of the AG-KBS coupling

4

Tests and Discussions

A validation of the algorithm has been carried out on the following benchmark, which simulate a real case of industrial optimisation. • •

•

AG Space: ∀i, {x1,…,x6}∈ℜ+ ; xi ∈[0...1] - binary coding on 4 bits KBS Space: xi, min(xi≠0), max(xi) in {[0..0,25[ [0,25..0,5[ [0,5..0,75[ [0,75..1[} x1, x4 and (x1&x4) presence or absence ratio = max / min in {[0..1] ]1..2] ]2..4] ]4..8] ]8..16]} Number of element (NbE) in {1;2;3;4;5;6} Fitness Function: (4Σ(x1+x2+x3)+2Σ(x4+x5)-x6) × promotor × factor × feasible with: If (x1 or x4)>0.15 then promotor = 0 else 1 If NbE < 4 then factor = NbE else (7-NbE) If ratio > 10 then feasible = 0 else 1

A conventional GA is used for comparison purpose for which the features are: one-point crossover (number of parents: 2), roulette wheel selection, 24 (6*4) of chromosome size and a generation size (K) of 20. For the GA-KBS algorithm two studies have been carried out: with µ1 fixed at 10 (figure 5) and µ2=2 which increases of one unity each 10 generation. Fitness

Fitness

Fitness

Generation 25000

50000

75000

Generation

Generation 5000

10000

100

200

500

1000

Fig. 3. Test results, SGA (left), AG-KBS with µ=10 (middle), zoom on both (right)

270

L. Baumes et al.

In figure 3, the curves represent the mean value for maxima found on 100 runs. The areas around the curves show the dispersion. The hybrid algorithm gives much better results even if the parameters are not optimised. We noticed that with µ2 the algorithm shows better results, KBS cannot learn ‘good' rules on few data at the beginning (low µ) but needs many different chromosomes (high µ) in order to choose whereas GA generations often converge to local optima. Indeed, the KBS decreases the dispersion of GA runs and increases fitness results. The SGA converges to local optima and does not succeed to reach the global maximum (72) before a long time. At the same time, KBS tries to keep diversity in generations. Therefore higher mutation and crossover rates are required in order to get individuals which belong to different KBS zones.

References [1] [2] [3]

[4] [5] [6] [7] [8]

Wolf, D.; Buyevskaya, O. V.; Baerns, M. Appl. Catal., A 2000, 200, 63-77. Corma, A.; Serra, J. M.; Chica, A. In Principles and methods for accelerated catalyst design and testing; Derouane, E. G., Parmon, V., Lemos, F., Ribeiro, F. R., Eds.; Kluver Academic Publishers: Dordrecht - NL, 2002, pp 153-172. Farrusseng, D.; Baumes, L.; Vauthey, I.; Hayaud, C.; Denton, P.; Mirodatos, C. In Principles and methods for accelerated catalyst design and testing; Derouane, E. G., Parmon, V., Lemos, F., Ribeiro, F. R., Eds.; Kluver Academic Publishers: Dordrecht - NL, 2002, pp 101-124. Ravisé, C.; Sebag, M.; Schoenauer, M. In Artificial Evolution; Alliot, J. M., Lutton, E., Ronald, Schoenauer, M., Snyers, D., Eds.; Springer-Verlag, 1996, pp 100-119. Rasheed, K.; Hirsh, H. In The Seventh International Conference on Genetic Algorithms (ICGA'97), 1997. Norris, E. M. Revue Roumaine de Mathématiques Pures et Appliquées 1978, 23, 243-250. Agrawal, R.; Imielinski, T.; Swami, A. In Proceedings of the ACM SIGMOD Conference, 1993. Belkhiter, N.; Bourhfi, C.; Gammoudi, M. M.; Jaoua, A.; Le Thanh, N.; Reguig, M. INFOR 1994, 32, 33-53.

Intelligent Motion Generator for Mobile Robot by Automatic Constructed Action Knowledge-Base Using GA Hirokazu Watabe, Chikara Hirooka, and Tsukasa Kawaoka Chikara Hirooka and Tsukasa Kawaoka Dept. of Knowledge Engineering & Computer Sciences, Doshisha University Kyo-Tanabe, Kyoto, 610-0394, Japan

Abstract. Intelligent robots, which can become a partner to us humans, need ability to act naturally. In actions of the robot, it is very important to recognize environment and move naturally. In this paper, the learning algorithm to construct knowledge of action in order to achieve tasks that is given to the mobile robot is proposed. Conventionally, the knowledge of action for the robot was mostly constructed by humans, but the more complex the information of the environment, action, and task becomes, the more trial and error must be repeated. Therefore, the robot must learn and construct the knowledge of action by itself. In this study, the action to achieve a task in an environment is generated by genetic algorithm, and it is shown that by repeatedly extracting knowledge of action, the construction of the Action Knowledge-Base concerning a task of any situation is possible.

1

Introduction

In the field of mobile robot control, many approaches have been tried to generate robot control program or strategy by giving only the objective task [2][3][4][5][6][7][8]. To achieve such a problem is not so difficult if the size of problem is small or a number of states of the robot is small. It becomes, however, very difficult when the size of problem is big or the number of states of the robot is big. It is assumed that a mobile robot has some sensors and acts some actions. The robot can properly acts, in some environment, if all action rules are given. The action rule is a kind of IF-THEN rule. A part of IF is one of the states of sensors and a part of THEN is one of the actions that the robot can perform. The problem treated in this paper is to decide all action rules automatically. If the number of states of sensors is small and the number of possible actions is also small, the problem is not so difficult. But, if these numbers are relatively big, the problem becomes difficult to solve because the size of solution space is proportional to the combination of those numbers. The problem becomes too difficult to solve if the number of states of sensors is infinite. Originally, the number of states of sensors is infinite since the V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 271-278, 2003.  Springer-Verlag Berlin Heidelberg 2003

272

Hirokazu Watabe et al.

value of each sensor is a continuous variable. However, in conventional methods, each sensor variable must be quantitized and the number of states of sensors must be finite and as small as possible. This constraint is hard to solve the practical problems. In this paper, the learning method to construct the action knowledge base, which consists of all proper action rules to achieve the task, by which the robot can properly act in any situation different from learning situation, is proposed. The learning method is basically constructed by genetic algorithm (GA)[1]. GA automatically generates some action rules in the learning environments. To evaluate the proposed method, the mobile robot simulator is made, and expected results are demonstrated. subgoal

G

S Fig. 1. Road Like Environment

d

subgoal b

L

a

R Fig. 2. Mobile Robot

Intelligent Motion Generator for Mobile Robot

2

Target Environment and Mobile Robot

2.1

Assuming Environment

273

In this paper, the environment for the robot is road like one shown in figure 1. That is, there are two places. One place is the place where the robot can move and the other place is the place where the robot cannot move. These two places are separated by some kinds of wall. In this environment, the robot can generate subgoals which are intermediate points from start point to goal point using his sonar sensors. Each subgoal has position and direction.

Fig. 3. Subgoal generator

2.2

Definition of Mobile Robot

In this paper, Pioneer 2 type mobile robot is assumed. Figure 2 shows the structure of the robot. It is assumed that the robot recognizes the following five parameters. (Unit length is a distance between left and right wheel of the robot.) 1. 2.

A distance to the subgoal: d (0<=d, and d is floating point number) An angle between the robot direction and the direction from the robot to the subgoal: a (-180degree < a <= +180degree, and a is a floating point number) 3. An angle between the robot direction and the objective direction: b (-180degree < b <= +180 degree, and b is a floating point number) 4. A speed of left wheel: L (-10 <= L <= +10, L is a integer number) 5. A speed of right wheel: R (-10 <= R <= +10, R is a integer number) And it is also assumed that the robot output the following two parameters. 1. 2.

Acceleration of left wheel: X (-3<=X<=+3, X is an integer number) Acceleration of right wheel: Y (-3<=Y<=+3, Y is an integer number)

Therefore, the action rule of this robot is formed as the following. IF (d, a, b, L, R) THEN (X, Y)

274

Hirokazu Watabe et al.

b1

SubGoal

a1

d1 L1 R1

X1 Y1 X2 Y2 Chromosome1 3 0 -2 1 E Chromosome2 -1 -2 0 1 E E

Xm Ym E E E 0 -3 E E E E1 1 E E

ChromosomeN

E E E 3 -3

2 -1

Fig. 4. Initial state

1

3

Fig. 5. Genotype X2

Y2

Y3 X3

X1 Y1

X1 Y1 X2 Y2 Best -3 3 2 0

EEE

Xm Ym * *

Fig. 6. Best actions

3

Planning and Moving Algorithm

The robot makes a plan and moves by the following algorithm. 1.

Generate one subgoal on the road by observing the environment around the robot using sonar sensors. (Fig. 3) 2. Move to the subgoal by using the Action Knowledge-Base (AKB). 3. Moving task is completed if the robot reaches to the final goal otherwise go to step 1.

Intelligent Motion Generator for Mobile Robot

4

275

Learning Method

To realize the proposed algorithm, AKB is required. AKB is a knowledge base, which has action rules. Each action rule is an IF-THEN type rule described in section 2.2. In this chapter, the method to generate the proper action rules automatically is described. 4.1

Learning Model

Learning process is performed in the mobile robot simulator. In this paper, the given task is to reach the subgaol (target place and target direction) as fast as possible and minimum energy. The following is an outline of the learning process. 1. 2.

Set the target place and target direction in the virtual environment. Set the initial states of the robot. These are position, orientation, and speed of left and right wheels. 3. Apply GA and obtain the best individual (consecutive actions) for the given initial states of the robot. 4. Extract action rules from the best individual (consecutive actions) and store these action rules into AKB. 5. Repeat from step 1 to 4 until enough action rules are collected. 4.2

Applying GA

In this learning model, GA is applied to obtain one consecutive actions from an initial state of the robot to the objective state of the robot (position and direction). The following is the algorithm that is obtained such a consecutive actions. 1. 2.

Set the initial state of the robot (d1, a1, b1, L1, R1) at random. (Fig. 4) Generate N chromosomes at random. (Fig. 5) Each chromosome is an mconsecutive action of the robot. For example, the chromosome (3,0, -2,1, 0,3) represents three steps of actions. Left wheel is accelerated for 3 in the first step, left wheel is accelerated for -2 and right wheel is accelerated for 1 in the second step, and right wheel is accelerated for 3 in the third step. 3. Perform simple GA process using an elite strategy, tournament selection of size 2, two-point crossover and mutation. The fitness function is as the following. fitness=w1T+w2D+w3B+w4E

Where, T is the time to reach the subgoal if the robot reaches the subgoal in msteps, otherwise T=m. D is the distance between the robot and the subgoal place in the final step when the robot did not reach the target in the m-step. B is the difference between robot direction and subgoal direction in the m-step. E is the total energy from initial state to final state. It is assumed that the robot uses unit energy by 3 accelerations for each wheel. Each wi is weight parameter. 4. Obtain the best chromosome after GA process. (Fig. 6)

276

Hirokazu Watabe et al.

4.3

Extract Action Rules

After a GA process is finished, some action rules are generated from the best chromosome of GA process. Figure 7 shows the image of action rules extraction. The robot is set in the initial state, and IF (the initial state) THEN (action of first step of best chromosome) becomes one of the action rules, i.e. IF (d1, a1, b1, L1, R1) THEN (X1, Y1). And this action rule is stored in the AKB. After the robot performed the action (X1, Y1), the state of the robot changed to (d2, a2, b2, L2, R2), then next action rule is the following. IF (d2, a2, b2, L2, R2) THEN (X2, Y2) And by repeating this process, maximum m action rules are obtained. (d3,a3,b3,L3,R3 | X3, Y3)

X2

Y2

Y3 X3 (d2,a2,b2,L2,R2 | X2, Y2) X1 Y1 Action Knowledge Base

(d1,a1,b1,L1,R1 | X1, Y1)

Fig. 7. Extract action rules

5

Experimental Results

To evaluate the proposed method, computer experiments are performed. Table 1. GA parameters Maximum generation Population size Length of chromosome Selection Elite Crossover Crossover rate Mutation rate

500 50 40 Tournament (size 2) 2% 2 point 60% 10% per a gene

Intelligent Motion Generator for Mobile Robot

277

4000 3500 3000

Fitness

2500 2000 1500

‚ U

1000 500

‚ U

0 0

50

100

150

200 250 300 Generation

350

400

Fig. 8. GA result (fitness)

5.1

450

500

Fig. 9. GA result (motion)

GA's Performance

The given task is to reach the subgoal (place and orientation) with minimum time and energy. Table 1 shows GA parameters used in the learning stages. Figure 8 shows an example of GA's performance by fitness graph and figure 9 shows the robot motion of the best individual of the same example. 5.2

Performance of Learning Method

To evaluate the proposed learning method, the following experiment is performed. 1.

Execute learning stage 1,500 times and obtain action rules only when the motion of best individual was success. In the learning stage, one subgoal was placed in the environment and the robot was set in the initial state. The initial state is chosen at random in each learning stage. The total number of obtained action rules was about 50,000. 2. After repeat the learning stage, the evaluation stage is performed. In the evaluation stage, 100 initial states of the robot are generated at random, and the robot acts using the action knowledge base constructed in the learning stage. The robot uses the action rule, which is most similar to current situation, in each step. The success rate was 85%. By these results, the proposed method is efficient and about 1,500 learning times is enough for this task.

6

Conclusion

In this paper, the learning method to construct the action knowledge base by which the robot can act properly to achieve the given tasks in any situation was proposed. The learning method is basically constructed by GA. However, applying GA simply to the problem, the robot did not properly act in any situation. GA can be applied only to the fixed situations. Therefore, GA was used to obtain the action rules to achieve

278

Hirokazu Watabe et al.

each given task and the action knowledge base was constructed by the collection of these action rules. Experimental results show that the robot can acts properly in almost any situation using the action knowledge base, which has enough and proper action rules.

Acknowledgements This work was supported with the Aid of Doshisha University's Research Promotion Fund.

References [1] [2] [3] [4] [5] [6]

[7] [8]

D. E. Goldberg: Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley (1989) T. Ito, H. Iba and M. Kimura: Robustness of Robot Programs Generated by Genetic Programming, Proc. of GP-96 (1996) 321-326 P. Maes, Behaviour-Based Artificial Intelligence, Proc. of the Second International Conference on Simulation of Adaptive Behaviour (SAB-92), MIT-Press (1993) 2-10 T. Minato and M. Asada, Environmental Change Adaptation for Mobile Robot Navigation, Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (1998) 1859-1864 C. W. Reynolds, Evolution of Obstacle Avoidance Behaviour: Using Noise to Promote Robust Solutions, Advances in Genetic Programming, Vol.1, chapter 10, MIT Press (1994) 221-241 K. Terada, H. Takeda and T. Nishida, An Acquisition of the Relation between Vision and Action using Self-Organizing Map and Reinforcement Learning, Proc. of Second International Conference on Knowledge-Based Intelligent Electronic Systems (KES98), Vol.1, IEEE (1998) 429-434 S. Yamada, Evolutional Learning of Behaviours for Action-Based Environment Modelling, Journal of Japanese Society for Artificial Intelligence, Vol.14, No.5 (1999) 870-878 H. Watabe and T. Kawaoka, Automatic Generation of Behaviours for Mobile Robot by GA with Automatically Generated Action Rule-Base, Proc. of IEEE International Conference on Industrial Electronics, Control and Instrumentation (IECON-2000) (2000) 1668-1674

Population-Based Approach to Multiprocessor Task Scheduling in Multistage Hybrid Flowshops Joanna J¸edrzejowicz1 and Piotr J¸edrzejowicz2 1

2

Institute of Mathematics, Gda´ nsk University Wita Stwosza 57, 80-952 Gda´ nsk, Poland [email protected] Department of Information Systems, Gdynia Maritime University Morska 83, 81-225 Gdynia, Poland [email protected]

Abstract. The paper considers multiprocessor task scheduling in multistage hybrid flowshops. To solve the above problem a population based approach is suggested. The population learning algorithm based on several local search procedures has been proposed and implemented. The algorithm has been evaluated by means of a computational experiment in which 160 benchmark instances have been solved and compared with the available upper bounds. It has been possible to improve 45% of previously known upper bounds.

1

Introduction

In this paper multiprocessor task scheduling in multistage hybrid flowshops is considered. Multiprocessor task scheduling has received growing attention in recent years. Already in the early eighties it was observed that there exist scheduling problems where some tasks have to be processed on more than one processor at a time [1], [2], [4]. During an execution of these multiprocessor tasks communication among processors working on the same task is implicitly hidden in a ”black box” denoting an assignment of this task to a subset of processors during some time interval. Multiprocessor task scheduling in multistage hybrid flowshops has numerous practical applications including fault-tolerant computing, multiprocessor computations, design of operations system for parallel environments, traffic control in restricted areas, and many others. Unfortunately the general, multiprocessor, flowshop problem to minimize schedule length (makespan) is NP-hard [5]. It should be also noted that even in a classical case of flow shop scheduling, only a few particular cases are efficiently solvable [3]. In view of the complexity of the considered scheduling problem it is unlikely that efficient exact algorithms for solving it could ever be found. Hence, various approximation algorithms have been studied and proposed. For example, well known metaheuristics - tabu search and genetic algorithm were proposed in, respectively [10] and [12]. This paper proposes applying a relatively new V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 279–286, 2003. c Springer-Verlag Berlin Heidelberg 2003


280

Joanna J¸edrzejowicz and Piotr J¸edrzejowicz

technique - the population learning algorithm to obtain solutions to multistage hybrid flowshop problems. It is shown that the approach is effective and promising, moreover, in numerous instances it has been possible to improve previously announced upper bounds for benchmark problems.

2

Problem Formulation

The discussed problem involves scheduling of n jobs composed of tasks in a hybrid flowshop with m stages. All jobs have the same processing order through the machines, i.e. a job is composed of an ordered list of multiprocessor tasks where the i-th task of each job is processed at the i-th flowshop stage (the number of tasks within a job corresponds exactly to the number of flowshop stages). Processing order of tasks flowing through stages is the same for all jobs. At each stage i, i = 1, . . . , m, there are available mi identical parallel processors. For the processing at stage i, task i being a part of the job j, j = 1, . . . , n, requires sizei,j processors simultaneously. That is, sizei,j processors assigned to task i at stage i start processing the task of the job j simultaneously and continue doing so for a period of time equal to processing time requirement of this task, denoted pi,j . Each subset of available processors can process only the task assigned to it at a time. The processors do not break down. All jobs are ready at the beginning of the scheduling period. Preemption of tasks is not allowed. The objective is to minimize makespan, i.e. the completion of the lastly scheduled task in the last stage.

3

Population Learning Algorithm

Population learning algorithm [7] is a population-based method inspired by analogy to a phenomenon of social education processes in which a diminishing number of individuals enters more and more advanced learning stages. In PLA an individual represents a coded solution of the considered problem. Initially, a number of individuals, known as the initial population, is randomly generated. Once the initial population has been generated, individuals enter the first learning stage. It involves applying some, possibly basic and elementary, improvement schemes. These can be based on some local search procedures. The improved individuals are then evaluated and better ones pass to subsequent stages. A strategy of selecting better or more promising individuals must be defined and duly applied. In the following stages the whole cycle is repeated. Individuals are subject to improvement and learning, either individually or through information exchange, and the selected ones are again promoted to a higher stage with the remaining ones dropped-out from the process. In the final stage the remaining individuals are reviewed and the best represents a solution to the problem at hand. At different stages of the process, different improvement schemes and learning procedures are applied. These gradually become more and more sophisticated and time consuming as there are less and less individuals to be taught.

Population-Based Approach to Multiprocessor Task Scheduling

281

Although PLA shares some features with evolutionary algorithms and greedy randomised adaptive search procedures (GRASP), introduced in [6], it clearly has its own distinctive characteristics. Main features of the PLA as compared with evolutionary algorithms are shown in Table 1.

Table 1. Population learning versus evolutionary algorithms Features

PLA

EA

Underlying phenomena Processing unit Population size Iterative search mode Intensification

Social education and Natural evolution learning processes Population of solutions Population of solutions Decreasing Constant Selection Generation replacement Variety of learning and Some implementations allow improvement procedures for intensification through embedded local search procedures Information exchange Different schemes including Information exchange based crossover allowed but none on crossover mechanisms is obligatory is a must Diversification Mostly achieved by setting Mutation is a primary massive initial population mechanism for size but additional schemes diversification including mutation can be used

PLA shares also some features with other population-based methods. In fact the idea of refining the population of solutions during the subsequent computation stages is common to PLA and memetic algorithms [9]. The latter, however, assume a constant population size and a single local search procedure and rely to a greater extend on typical genetic/evolutionary algorithms operators. There are also some similarities between PLA and cultural algorithms where an inheritance process operates at the micro and the micro-evolutionary levels [11].

4

PLA Implementation

To solve the multiprocessor task scheduling in a multistage hybrid flowshop problem, the population learning algorithm, denoted as PLA-MPFS has been implemented. It is an extension of the algorithm proposed originally in [8]. The algorithm makes use of different learning and improvement procedures, which, in turn, are based on five neighbourhood structures shown in Table 2. In what follows π = (π(1), . . . , π(n)) denotes an individual (a permutation of tasks) and g(π) its fitness function. Fitness function of an individual is calculated according to the following algorithm:

282

Joanna J¸edrzejowicz and Piotr J¸edrzejowicz

FITNESS: begin for i = 1 to m do for j = 1 to n do allocate task π (j) at stage i to the required number of processors, scheduling it as early as feasible; end for end for output:= finishing time of task number π (n) at stage m end

Table 2. Neighbourhood structures Notation Move N1 (x) N2 (x) N3 (x)

Neighbourhood space

Exchange of two consecutive tasks in x All possible exchanges Exchange of two non-consecutive tasks in x All possible exchanges Finding optimal order of four consecutive All fourtuples of consecutive tasks in x by enumeration tasks

All learning and improvement procedures operate on the population of individuals P, and perform local search algorithms based on the neighbourhood structures shown in Table 2. LEARN (i, P): for each individual x in P do Local_search (i) end for Local search algorithms are defined in Table 3. Within the discussed implementation a simple procedure SELECT(P,s) is used, where s is a percentage of best individuals from a current population promoted to higher stages. For any population P, let LEARNING(P) stand for the following: LEARNING(P): for i = 1 to 5 do LEARN(i,P) SELECT(P, s) end for The last learning procedure LEARN REC(x) is recursive and operates on overlapping partitions of the individual x, which form a ”small” population of individuals P’. To such population LEARNING(P’) is applied. Now, the structure of the implemented population learning algorithms can be shown as:

Population-Based Approach to Multiprocessor Task Scheduling

283

PLA-MPFS: begin generate randomly initial_population P := initial_population LEARNING(P) for each individual x in P do LEARN_REC (x) end for output the best individual from P end

Table 3. Local search algorithms used in the PLA-MPFS i

Description of the local search algorithm

1

Perform all moves from the neighborhood structure N1 (x); accept moves improving g(x); stop when no further improvements of g(x) are possible Mutate x producing x
(mutation procedure is selected randomly from the two available ones the two point random exchange or the rotation of all chromosomes between two random points); perform all moves from the neighborhood structure N1 (x
); accept moves improving g(x
); stop when no further improvements of g(x’) are possible Repeat k’ times (k’ is a parameter set at the fine-tuning phase; in the reported experiment k’ = 3 * initial population size); generate offspring y and y
by a single point crossover from x and random x
; perform all moves from the neighborhood structure N1 (y) and N1 (y
); accept moves improving g(y) and g(y
); stop when no further improvements of g(y) and g(y
) are possible; adjust P by replacing x and x
with the two best from {x, x
, y, y
} Perform all moves from the neighborhood structure N2 (x); accept moves improving g(x); stop when no further improvements of g(x) are possible Perform all moves from the neighborhood structure N3 (x); accept moves improving g(x); stop when no further improvements of g(x) are possible

2

3

4

5

284

5

Joanna J¸edrzejowicz and Piotr J¸edrzejowicz

Computational Experiment

The proposed algorithm has been evaluated experimentally. The experiment has involved 160 instances of the multiprocessor task scheduling in a multistage hybrid flowshop. The dataset includes 80 instances with 50 jobs each and 80 instances with 100 jobs each. Each group of 80 problem instances is further partitioned into four subgroups of 20 instances each with 2, 5, 8 and 10 stages, respectively. Processor availability at various stages and processor requirements per task varied between 1 and 10. All problem instances as well as their currently best upper bounds have been obtained through OR-LIBRARY (http://www.ms.ic.ac.uk/info.html). The experiment has been carried on a PC with Pentium III, 850 Mhz processor. The initial population size for the experiment has been set to 200 and the selection factor s to 0.5. Mean relative error (MRE), standard deviations of errors and mean computation times calculated after a single run of the PLA-MPFS are shown in Table 4. Negative values of MRE show how much it has been possible to improve upper bounds within respective clusters of benchmark dataset. Table 4. MRE, standard deviation of errors and mean computation time Measure

Number of jobs 2

MRE (%) St. deviation of errors (%) Mean computation time (min.)

50 100 50 100 50 100

-0.9814% -0.6127% 0.0174% 0.0116% 1.20 3.55

Number of stages 5 8 -0.4036% -0.3180% 0.2971% 0.2253% 3.35 8.80

-0.3367% -0.2535% 0.4104% 0.3042% 5.76 14.15

10 0.3712% -0.4540% 0.5992% 0.2122% 10.40 27.42

Application of the PLA-MPFS has resulted in improving the total of upper bounds of all considered cases by 0.34%. Out of 160 instances solved it has been possible to improve currently known upper bounds in 73 instances, which is more than 45% of all considered instances. The results of the reported experiments are available at http://manta.univ.gda.pl/˜jj/mpfs.txt.

6

Conclusions

Results of the experiments allow to draw the following conclusions: – Population learning algorithms can be considered as an effective metaheuristic for solving problems of multiprocessor task scheduling in multistage hybrid flowshops. The reported experiment has contributed to improving 73 upper bounds out of 160 solved benchmark instances.

Population-Based Approach to Multiprocessor Task Scheduling

285

– Quality of results obtained by applying PLA in terms of both - standard deviation of error and mean relative error produced in a single run, is quite satisfactory. – Whereas the quality of solutions obtained using PLA algorithms is satisfactory, there is still a lot of room for improvements with respect to computational effort required. – Future research should concentrate on parallel PLA schemes with a view of increasing quality of results or decreasing computation time or achieving both goals simultaneously. – Critical for the successful application of the PLA approach is the fine-tuning phase where a proper balance between computation time and solution quality has to be found. Since there are several parameters, which can be controlled by the user (population size, selection criterion, number of iterations for the embedded local search procedures etc.), further research is needed to identify interactions between these parameters and to establish methodology for setting their values. – The experiment has confirmed both - value and strength of the populationbased approach, which remains one of the most effective computational intelligence techniques.

References [1] Bla˙zewicz J., M. Drozdowski, J. W¸eglarz: Scheduling independent 2-processor tasks to minimize schedule length, Information Processing Letters, 18 (1984) 267– 273 279 [2] Bla˙zewicz J., M. Drozdowski, J. W¸eglarz: Scheduling multiprocessor tasks to minimize schedule length, IEEE Transactions on Computers, C-35(1986) 81–96 279 [3] Bla˙zewicz J., K. H.Ecker, E.Pesch, G.Schmidt, J.W¸eglarz: Scheduling Computer and Manufacturing Processes, Springer, Berlin (1996) 279 [4] Lloyd E. L.: Concurrent task systems, Operations Research, 29 (1981) 189–201 279 [5] Brucker P., B. Kramer: Shop Scheduling Problems with multiprocessor tasks on dedicated processors, Annals of Operations Research, 50 (1995) 13–27 279 [6] Feo T. A., M. G. C. Resende: A probabilistic heuristic for computationally difficult set covering problems, Operations Research Letters, 8 (1989) 706–712 281 [7] J¸edrzejowicz P.: Social Learning Algorithm as a Tool for Solving Some Difficult Scheduling Problems, Foundation of Computing and Decision Sciences, 24 (1999) 51–66 280 [8] J¸edrzejowicz J., P. J¸edrzejowicz: Permutation Scheduling Using Population Learning Algorithm, in: Knowledge-based Intelligent Information Engineering Systems and Allied Technologies, E.Damiani et al. (Eds.), IOS Press, Amsterdam (2002) 93–97 281 [9] Moscato P.: Memetic Algorithms: A short introduction. In: D.Corne, M.Dorigo, F.Glover (Eds.), New Ideas in Optimization, McGraw-Hill, New York (1999) 219– 234 281 [10] Oˇ guz, C., Y. Zinder, V. Ha Do, A. Janiak, M. Lichtenstein: Hybrid flow-shop scheduling problems with multiprocessor task systems, Working paper, The Hong Kong Polytechnic University, Hong Kong SAR (2001) 279

286

Joanna J¸edrzejowicz and Piotr J¸edrzejowicz

[11] Reynolds R. G.: An Introduction to Cultural Algorithms. In: A. V. Sebald, L. J. Fogel (Eds.), Proceedings of the Third Annual Conference on Evolutionary Programming, World Scientific, River Edge, (1994) 131–139 281 [12] Sivrikaya-S ¸ erifoˇ glu, F., G. Ulusoy: A genetic algorithm for multiprocessor task scheduling in multistage hybrid flowshops, Working paper, Abant Izzet Baysal University, Bolu (2001) 279

A New Paradigm of Optimisation by Using Artificial Immune Reactions M. K¨ oster1 , A. Grauel1 , G. Klene1 , and H. Convey2 1

University of Applied Sciences – South Westphalia, Soest Campus L¨ ubecker Ring 2, D-59494 Soest, Germany {koester,grauel,klene}@coin.soest.fh-swf.de 2 Bolton Institute of Higher Education, Deane Campus Bolton, BL3 5AB, England [email protected]

Abstract. This paper presents an implementation of an artificial immune system in order to realise multi-modal function optimisation. The main paradigms of an artificial immune system are clonal selection and affinity maturation. Therefore these paradigms are reviewed briefly. The optimisation is based on the opt-aiNet algorithm. This algorithm is described theoretically. Furthermore, the Griewank function has been used to test the algorithm with different parameters. The simulation results are discussed.

1

Introduction

The architecture and mechanisms of the human immune system can be explained by several theories and mathematical models [1, 2]. The immune system consists of a complex set of cells and molecules that protect the body against infections. The features of an immune system such as learning, memory, and adaptation are currently being used in the development of Artificial Immune Systems. Beside the classical procedures and the computational approaches we propose AIS as a useful optimisation strategy. Publications in this relatively new research field show, that the concept is interesting for e.g. data mining [3], pattern recognition [4] and optimisation [5]. It is still unclear what characterises an AIS, but the principles clonal selection and affinity maturation can be found in most of the works related to AIS. These principles are going to be reviewed in Section 2. Based on clonal selection and affinity maturation de Castro and Timmis developed the opt-aiNet algorithm [6] which is suitable for multimodal optimisation problems. Section 3 discusses the concept of this algorithm. In order to test the performance of the algorithm the Griewank function [7] has been applied to the algorithm. The results of this test function are described in Section 3.2.

2

Artificial Immune Systems

Compared to other well-established computational intelligence paradigms such as artificial neural networks, evolutionary algorithms and fuzzy systems, the field V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 287–292, 2003. c Springer-Verlag Berlin Heidelberg 2003


288

M. K¨ oster et al.

of artificial immune systems is still a new paradigm. The two major principles used in artificial immune systems are clonal selection and affinity maturation. 2.1

Clonal Selection and Affinity Maturation

When an antigen (pathogen) enters the human body, the B-cells (B-lymphocytes) of the human immune system respond by secreting antibodies. Antibodies are molecules which are able to recognize and bind to a certain part of the antigen called epitope. Each antibody can only bind to one type of epitope, whereas an antigen can have more than one epitope. Thus, the antigen can be recognized by several antibodies. Once an antibody of a B-cell has recognized the eptitope of an antigen, the antigen stimulates the B-cell to proliferate (clone) and maturate into a plasma cell or a memory cell. Plasma cells are the most active antibody secretors. Memory cells are the cells having high affinity with the antigens. They circulate through the blood, lymph and tissues for a faster response to previously recognized antigens. As those B-cells that recognize an antigen improve their affinity to the selective antigen, this principle is called affinity maturation. And since these B-cells also proliferate, while those who don’t recognize an antigen waste away, this principle is called clonal selection [8]. It is similar to the natural selection found in the evolution theory except that it occurs on a rapid time scale within the order of days.

3

Implementation and Test of the opt-aiNet Algorithm

Based on the principles mentioned in Section 2 de Castro and Timmis developed the opt-aiNet algorithm [6] to realise a multimodal function optimisation. Like evolutionary algorithms the opt-aiNet algorithm is an adaptive optimisation technique. With each iteration the solution gradually improves until the local or the global optimum is found. The algorithm is also a stochastic optimisation method. Thus, it enables the exploration of the fitness landscape. It is not only capable of locating the global optimum, but also local optima which might also be of interest. 3.1

Algorithm

The components used in the algorithm can be summarised as follows: Network Cell: A network cell c is one feasible solution to the optimisation problem. Each cell can be represented by a real-valued vector c ∈ IRn . Population: The population Pt = {c0 , ..., c|Pt |−1 } is the set of all current network cells at the time t. Fitness: The fitness of a cell describes the quality of the solution. In the case of finding the minimum or the maximum, the fitness value can be derived by the objective function itself. Ft is the average fitness of the population Pt at the time t.

A New Paradigm of Optimisation by Using Artificial Immune Reactions

289

Affinity: The affinity measure is given by the Euclidean distance between two cells. Clone: A clone is an identical copy of a parent network cell. The opt-aiNet algorithm can be summarised as follows: (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (22) (23) (24) (25) (26) (27) (28) (29) (30) (31) (32) (33) (34) (35) (36) (37) (38)

t=0 P0 = randomly initialise population with np cells
|P0 |−1 F0 = |P10 | i=0 f itness(ci ) while not abort do repeat t=t+1 Pt = Pt−1 for i ∈ {0, ..., |Pt | − 1} do for j ∈ {1, ..., nc } do mc = hypermutate(ci, f itness(ci )) if f itness(mc) > f itness(ci ) then ci = mc end if end for end for
|Pt |−1 Ft = |P1t | i=0 f itness(ci ) until Ft < Ft−1 + σf i=0 Pmem = {c0 } for j ∈ {0, ..., |Pt | − 1} do k=0 repeat if af f inity(cj , ck ) < σs if f itness(cj ) > f itness(ck ) replace ck in Pmem by cj end if else k = k+1 end if until (k > i) or (af f inity(cj , ck ) < σs ) if k > i Pmem = Pmem ∪ {cj } i=i+1 end if end for Pt = Pmem Pt = add (|Pmem | ∗ d) randomly generated cells end while

In this algorithm the optima can be treated as the antigens and the network cells as the B-cells of the immune system. The fitness of the network cell is a measure of the affinity to the antigen (optimum). The algorithm combines a local search in the steps (5)-(17) and a global search in the steps (18)-(35). In the local search the fitness of the currently

290

M. K¨ oster et al.

available network cells improve gradually until the degree of improvement is less than the predefined fitness threshold σf . The affinity mutation in step (10) is realized by the following equation: mc = c +

N −fn (c) e β

.

(1)

In the above equation mc is the mutated network cell, c is the parent network cell, N is a Gaussian random number of zero mean and standard deviation σ = 1, β is a decay parameter, and fn (x) is the normalized fitness value of c which is defined by the following equation. fn (c) =

(f (c )) f (c) − min
c ∈P

max (f (c )) − min (f (c ))

c ∈P

;

f (c) : f itness value .

(2)

c ∈P

Since the mutation is based on the fitness and thus on the affinity of the parent network cell, it describes the affinity maturation process. If the fitness of a mutated clone is better than the parent cell, the parent cell is replaced by this clone in the steps (11)-(13) of the algorithm. This mechanism describes the clonal selection in the local search. The suppression mechanism in the steps (24)-(26) describes the clonal selection in the global search. 3.2

Simulation Results

The opt-aiNet algorithm has been applied to the Griewank function, which is defined as follows:   n n   xi x2i − cos √ (3) f (x) = 1 + ; x = (x1 , ..., xn ) ∈ IRn . 4000 i i=1 i=1 The griewank function is a widely used multimodal test function with a very rugged landscape and large number of local optima. In the following the degree of the function is set to n = 2. For the local search the number of clones are set to nc = 20 and the fitness threshold σf = 0.01. The factor of newcomers is set to d = 5. The algorithm is aborted after 200 generations. To find the global maximum of the Griewank function in an input range of x1 , x2 ∈ [−100, 100] the suppression threshold can be set to σs = 100 and the mutation decay to β = 2. The corresponding result is shown in Figure 1. Only the best network cell survives in each iteration as the suppression area covers nearly the whole input space. Since the decay β is small, the possible range of the affinity mutation, defined by Equation (1), is high and therefore the network cells don’t get stuck in a local minima. It was able to locate the global optima (x=(0,0)) in most of the test runs after approximately 116 generations. In a second example the input range of the Griewank function is set to x1 , x2 ∈ [−10, 10]. The factor of newcomers is set to d = 0.8. By setting the suppression threshold to σs = 0.5 and the mutation decay to β = 10 the

A New Paradigm of Optimisation by Using Artificial Immune Reactions

291

Fig. 1. The Griewank function, defined by Equation (3), with an input range of x1 , x2 ∈ [−100, 100]

Fig. 2. The Griewank function, defined by Equation (3), with an input range of x1 , x2 ∈ [−10, 10] algorithm discovers not only the global optimum but also local optima as shown in Figure 2. After approximately 83 generations the algorithm was able to locate all 17 optima including the global optimum. In this example the higher mutation decay β reduces the possible range of the affinity mutation. Therefore, the network cells converge towards their next local optimum.

292

4

M. K¨ oster et al.

Conclusion

The presented AIS is a new computational intelligent method based on biological metaphors for optimisation. We have tested this method on different multi-modal function optimisation problems. Main advantages in an engineering perspective are the handling and the simplicity of the clonal selection principle and the affinity maturation. In contrast to Evolutionary Algorithms (EAs) no encoding of cells (individuals) is required, the cardinality of the population is automatically determined by the suppression mechanism and the capability of combining exploitation with exploration of the fitness landscape with good stabilisation of the population. The convergency speed for functions with many extremities can be a strong limitation to the practical application of artificial neural networks (ANN).

References [1] N. K. Jerne: Towards a network theory of the immune system. In: Ann. Immunol. (Inst. Pasteur), Vol. 125C. (1974) 373–379 287 [2] Alan. S. Perelson: Theoretical Immunology, Volume 1-2. Addison Wesley, Reading, MA, USA (1988) 287 [3] T. Knight and J. Timmis: A Multi-Layered Immune Inspired Approach to Data Mining. In A Lotfi, J Garibaldi, and R John, eds.: Proceedings of the 4th International Conference on Recent Advances in Soft Computing, Nottingham, UK. (2002) 266–271 287 [4] L. N. de Castro and J. Timmis: Artificial Immune Systems: A Novel Approach to Pattern Recognition. In L Alonso J Corchado and C Fyfe, eds.: Artificial Neural Networks in Pattern Recognition. University of Paisley (2002) 67–84 287 [5] D. Dasgupta, ed.: Artificial Immune Systems and Their Applications. SpringerVerlag, Berlin (1999) 287 [6] L. N. de Castro and J. Timmis: An Artificial Immune Network for Multimodal Function Optimization. In D. B. Fogel, M. A. El-Sharkawi, Xin Yao, G. Greenwood, H. Iba, P. Marrow, and M. Shackleton, eds.: Proceedings of the 2002 Congress on Evolutionary Computation CEC2002, IEEE Press (2002) 699–704 287, 288 [7] D. Whitley, K. Mathias, S. Rana, and J. Dzubera: Building Better Test Functions. In Larry Eshelman, ed.: Proceedings of the Sixth International Conference on Genetic Algorithms, San Francisco, CA, Morgan Kaufmann (1995) 239–246 287 [8] F. M. Burnet: Clonal Selection and After. In G. I. Bell, A. S. Perelson, and G. H. Pimbley Jr., eds.: Theoretical Immunology, Marcel Dekker Inc. (1978) 63–85 288

Multi-objective Genetic Programming Optimization of Decision Trees for Classifying Medical Data Ernest Muthomi Mugambi1 and Andrew Hunter2 1

Science Lab, Computer Science Dept., Sunderland University South Rd., Durham DH1 3LE [email protected] 2 Durham University

Abstract. Although there has been considerable study in the area of trading- off accuracy and comprehensibility of decision tree models, the bulk of the methods dwell on sacrificing comprehensibility for the sake of accuracy, or fine-tuning the balance between comprehensibility and accuracy. Invariably, the level of trade-off is decided a priori. It is possible for such decisions to be made a posteriori which means the induction process does not discriminate against any of the objectives. In this paper, we present such a method that uses multi-objective Genetic Programming to optimize decision tree models. We have used this method to build decision tree models from Diabetes data in a bid to investigate its capability to trade-off comprehensibility and performance.

1

Introduction

Decision trees have been popularly univariate (also known as axis-parallel) in nature which implies that they use splits based on a single attribute at each internal node. Even though several methods have been developed for constructing multivariate trees, this body of work is not well-known [1]. Most of the work on multivariate splits considers linear (oblique) trees [10]. The problem of finding an optimal linear split is known to be intractable [11] hence the need to find good albeit suboptimal linear splits. While there exist methods used for finding good linear splits ranging from linear discriminant analysis [12], hill-climbing search [13] to perceptron training [14] amid others, the use of evolutionary algorithms which are known to be powerful global search mechanisms, is little-known. In this paper, we use a Genetic Programming algorithm to optimize polynomial (non-linear) decision structures. For many practical tasks, the trees produced by tree-generation algorithms are not comprehensible to users due to their size and complexity. It is desirable that the classifier “provide insight and understanding into the predictive structure of the data” [16] as well as explanations of its individual predictions [15]; in medical data mining such characteristics are a prerequisite for model acceptability. It can be argued that the incomprehensibility of some models is caused by the V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 293–299, 2003. c Springer-Verlag Berlin Heidelberg 2003


294

Ernest Muthomi Mugambi and Andrew Hunter

model induction process being primarily based on predictive accuracy or performance. To address this concern, we use a multi-objective Genetic Programming algorithm to optimise decision trees for both classification performance and comprehensibility, without discriminating against either. There is a lack of a proper empirically tested theory on comprehensibility; the comprehensibility ideas used are based on some limited studies in the literature [17][18].

2

Decision Trees

Decision trees are a way to represent underlying data hierarchically by recursively partitioning the data. The Decision tree formalism has been found to be intuitively appealing and comprehensible because the way they perform classification by a sequence of tests is easy for a domain expert to understand. For this reason among others, decision trees have found their way into many a data mining researcher’s / practitioners tool box. For a comprehensive survey on decision trees refer to [1]. One way to classify decision trees is by looking at the way they split data. The most common type of decision tree are referred to as univariate or axis-parallel. This type of decision trees carry out tests on a single variable at each non-leaf node. Their mode of splitting the data is equivalent to using axis-parallel hyperplanes in the feature space. Another class of decision trees tests a linear combination of features at each internal node and are referred to as multivariate linear or oblique trees because the tests are geometrically equivalent to hyperplanes at an oblique orientation to the axis of the feature space. The last and least known decision tree type is the non-linear multivariate decision trees. Although often a linear tree can describe the data well, studies [3] have shown that non-linear trees are more accurate. Ittner and Schlosser [2] conceived a non-linear multivariate decision tree that produces partitioning in the form of curved hypersurfaces of the second degree. Their method is based on combination of primitive features (constructive induction) in the form of polynomials which are induced by a linear (oblique) tree generation algorithm. We introduce a novel multivariate non-linear decision tree that is optimized by use of Genetic Programming. The optimization is aimed at maximizing the performance and improving the comprehensibility of the decision tree in tandem.

3

Structure and Comprehensibility

There are three types of operators used that dictate the structure of the decision tree. Base operators comprise multiplication (*) and addition (+) operators that are used to combine several terminal functions or variables into a polynomial function. Fuzzy operators are fuzzy membership functions (sigmoid functions and bell functions) used to squash continuous variables. For example we could have a polish notation polynomial function of the form: (∗(+x1 x2 )φx3 ), where φ represents a sigmoid membership function. Intermediate operators are novel high level operators that are used to partition the feature space. Model comprehensibility is determined by syntactic and semantic simplicity.

Multi-objective Genetic Programming Optimization of Decision Trees

295

– Syntactic simplicity - This refers to the size of a pattern or number of interacting terms in a model; also known as complexity. For a polynomial model it is determined by the number of nodes in the decision tree. – Semantic simplicity - The type and order of operators used in the decision trees. The model is easier to comprehend when the intermediate operators are at the top followed by the base operators in the middle and lastly the fuzzy operators at the leaf nodes.

4

Genetic Programming(GP)

Genetic Programming is a global, random search method for solving genetic treelike programs. It is suitable for solving NP-hard computational problems such as those that arise in non-linear decision tree modeling. GP has been successful in such areas as symbolic regression, pattern recognition and concept learning among others. GP offers a very natural way to represent the decision tree structure. GP uses several operators such as crossover and mutation to optimize the expression tree.

5

Coefficient Optimization

While GP is good at optimizing the structure of the tree, it is known to be deficient in the capability to optimize the coefficients in the expression tree [8]. We have incorporated a Quasi-Newton optimization technique to augment the power of the GP coefficient optimization. This technique uses an error propagation algorithm that efficiently calculates the gradient of the error function with respect to the coefficients embedded in the GP expression tree.

6

Multi-objective Optimization (MOO)

Multi-objective optimization is defined as “a vector of decision variables which satisfies constraints and optimizes a vector function whose elements represent the objective functions”. Most decision trees use classification performance as the objective of the induction process. While performance is an important objective other characteristics such as tree size and comprehensibility might not be any less important. MOO therefore offers a suitable way to optimize the decision tree without discriminating against any of the objectives by using Pareto optimality concept. According to this concept a feature solution A is better than B if A is better than B in at least one objective and not worse-off in any of the remaining objectives. A set of solutions where none is better than any other is known as a non-dominated set and if such a set dominates all other solutions it is reffered to as the Pareto front. MOO algorithms [7][19] attempt to find this Pareto front. There are three main objectives we are interested in: tree size, classification performance and degree of the polynomial. Tree size is a straightforward measure of the size of the decision tree, which we aim to minimize. It is accepted that

296

Ernest Muthomi Mugambi and Andrew Hunter

smaller decision tree are more comprehensible and have better generalization capabilities. The second objective we mean to maximize is classification performance, which is measured using the Receiver Operating Characteristic curve. This is a standard technique, widely used in medical applications, which characterises the performance of a classifier as the decision threshold is altered, by displaying the sensitivity1 versus 1-specificity2 . Finally the degree of the polynomial structure determines how the feature space is partitioned. Second degree polynomials describe quadratic hypersurfaces which can be used to model complex non-linear behaviour. Therefore the third objective is to use second degree polynomial terms where appropriate; for minimizing use of an excessive number of linear terms.

7

Multi-objective Genetic Programming Algorithm (MOGP)

The algorithm commences with the initialisation of the decision tree structures in the form of Genetic Programming expression trees. The trees can be stronglytyped to ensure that only specific orders of operators are allowed. This is in line with the need to develop perspicuous tree structures. Although the generation of terminals and non-terminals is purely random, there are stricts controls to ensure that the expression trees are valid. During initialization the training set is also set to half the data set, the cross validated test set is set to the remaining data set and the terminal set to the variables contained in the data set. After initialization the initial population is evaluated and the elite set which is made up of the “best” individuals of that population is constituted. MOGP is an elitist evolutionary algorithm but great caution is taken to ensure that the elite set does not have undue influence on the ability of the algorithm to explore diverse feature spaces. The selection of the elite is based on three objectives: the performance on the Receiver Operating Characteristic curve, the size of the decision tree and lastly the percentage of the second degree polynomial functions. The Pareto optimality concept is used for the multi-objective evaluation after the coefficient optimization of the expression tree. The mating pool is made up of individuals from population and the elite set chosen by binary tournament selection where the winner is determined by fitness sharing. Reproduction is carried out by the mating pool using crossover and mutation. Crossover and mutation operations are strictly controlled to ensure that they give valid offsprings.

8

Experiments and Results

We conducted ten experiments using the algorithm. Control parameters for the runs are given in Table 1. The data set constitutes 2304 records of patients that attend a diabetic clinic. Each patient corresponds to a record in the data set. 1 2

Percentage of positively classified instances that the classifier gets right. Percentage of negatively classified instances that the classifier gets right.

Multi-objective Genetic Programming Optimization of Decision Trees

297

Table 1. MOGP Settings Population Non-dominated set size Tournament size Dominance group size Generations Crossover rate Mutation rate

100 25 2 10 200 0.3 0.3

Each record contains readings taken from the patients and their medical history. The 29 patient attributes constitute factors that are associated with Diabetes and the type of complications the patients may have suffered in the period they have had diabetes. The aim of this classification exercise is to build decision tree models that can predict patients whose diabetic status is likely to deteriorate leading to complications that are normally associated with the disease. The training set is made up of 1300 cases while the test set constitutes the remaining 1004 cases. Figure 1 shows the ROC Curves of some of the decision tree models found in the elite set which comprises the non-dominated models in the population. The complexity of the elite set models range from 1 to 39. To assess the efficacy of the algorithm, we compared the results with a Radial Basis Function neural network using all the 29 variables. The Radial Basis Function network is a powerful but

1

1

1

1

0.5

0.5

0.5

0.5

c=1 c=3 c = 11 c=5 0 0 0 0 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 1 1 1 1 0.5

0.5

0.5 0.5 c = 21 c = 35 c = 17 c = 27 0 0 0 0 0 0.5 1 0 0.5 1 0 0.5 1 0 0.5 1 1 1 KEY MOGP 0.5 0.5 RBF c = 37 c = 39 0 0 0 0.5 1 0 0.5 1

Fig. 1. ROC Performance of MOGP models compared with RBF models for different complexities

298

Ernest Muthomi Mugambi and Andrew Hunter

poorly comprehensible technique and hence forms a good basis for comparison with our model. It is evident from the ROC curves in figure 1 that the low complexity models (1-35) perform as well as the RBF network. As expected the higher complexity models perform very poorly - it is known that complex models have poor generalization capability. From the results it is also evident that our algorithm is capable of producing a wide range of model complexities that are of good peformances. In addition to this, very comprehensible (small and compact) models such as 0.111x2 ∗ 0.34x18 and 0.02x2 ∗ 2.3x23 + 1.2x6 ( see the first and second ROC curves in figure 1) that perform as well as an RBF network, are easily obtained.

9

Conclusion

We have introduced and described a method for inducing decision tree models without determining the level of trade-off between comprehensibility and ROC curve performance before hand. The decision tree models are optimised using a multi-objective Genetic Programming technique. The coefficients within the polynomial decision tree models are optimised using an efficient error propagation algorithm. Future work will involve comparing the performance of our decision tree framework mainly with other known linear and non-linear decision tree induction algorithms.

References [1] [2]

[3] [4]

[5]

[6] [7]

[8] [9]

S.K Murthy. Automatic construction of decision trees from data: a multidisciplinary survey.Kluwer academic publishers,Boston.1-49,1998. 293, 294 A.Ittner and M.Schlosser. Discovery of relevant new features by generating non-linear decision trees.Proc. of 2nd International Conference on Knowledge Discovery and Data Mining,108-113.AAAI Press,Menlo Park,CA,Portland,Oregon,USA,1996. 294 A.Ittner. Non-linear decision trees NDT.International conference on machine learning,1996. 294 N.Nikolaev and V. Slavov. Concepts of inductive genetic programming.EuroGP:First European workshop on genetic programming,lecture notes in computer science. LNCS 1391, Springer,Berlin,1998,pp.49-59. M. C.J Bot and W. B. Langdon. Application of genetic programming to induction of linear classification trees.EuroGP 2000:European conference,Edindburgh,Scotland,UK,April 2000. L.A Breslow and D. W. Aha. Simplifying decision trees: a survey. Navy Center for Applied Research in Knowledge Engineering Review Technical Report , 1998. C.Emmanouillidis. Evolutionary multi-objective feature selection and ROC analysis with applocation to industrial machinery fault diagnosis.Evolutionary methods for design,optimization and control,Barcelona 2002. 295 A.Hunter. Expression Inference - genetic symbolic classification integrated with non-linear coefficient optimization.AISC 2002:117-127. 295 J. Hanley and B. McNeill. The meaning and use of the area under a receiver operator characteristic curve. Diagn.Radiology,143,29-36,(1982).

Multi-objective Genetic Programming Optimization of Decision Trees [10]

[11] [12]

[13] [14] [15]

[16] [17]

[18] [19]

299

A. Ittner, J. Zeidler, R. Rossius, W. Dilger and M. Schlosser. Feature space partitioning by non-Linear and fuzzy decision trees.Chemnitz University of Technology ,Department of Computer Science,Chemnitz,1997. 293 L. Hyafil and R. L. Rivest. Constructing optimal binary decision trees is NPcomplete.Information Processing Letters,5(1):15-17,1976. 293 W. Loh and N. Vanichsetakul. Tree structured classification via generalized discriminant analysis.J.of the American Statistical Association,83(403):715728,1988. 293 S. Murthy,S. Kasif, S. Salzberg and R. Beigel. A system for induction of oblique decision trees.J. of Artificial Intelligence Research,2:1-33,August 1994. 293 S.E Hampson and D. J. Volper. Linear function neurons:Structure and training.Biological Cybernetics,53(4):203-217,1986. 293 D. Michie. Inducing knowledge from data;First Principles.Unpublished manuscript for a talk given at the Seventh International Conference on Machine Learning.Austin,Texas. 293 L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and regression trees.Belmont,CA:Wadsworth International Group. 293 C. Pena-Reyes, and M. Sipper. Fuzzy CoCo:Balancing accuracy and interpretability of fuzzy models by means of coevolution.Logic Systems Laboratory, Swiss Federal Institute of Technology in Lausanne, CH-1015 Lausanne,Switzerland. 294 N. Lavrac. Selected techniques for data mining in medicine. Artificial Intelligence in Medicine 16(1999) 3-23. 294 A. Hunter. Using multiobjective genetic programming to infer logistic polynomial regression models. 15th European Conference on Artificial Intelligence,Lyon,France.2002. 295

A Study of the Compression Method for a Reference Character Dictionary Used for On-line Character Recognition Jungpil Shin Department of Computer Software, University of Aizu Aizu-Wakamatsu City, Fukushima, 965-8580, Japan tel.: +81 (242) 37-2704; fax: +81 (242) 37-2731 [email protected]

Abstract. This paper reports on a compression method for a reference character dictionary used for on-line Chinese handwriting character recognition, which is based on pattern matching between input and reference patterns. First, one reference pattern for each category is generated from the training data. Second, the reference dictionary is optimized by using the Generalized Learning Vector Quantization (GLVQ) algorithm. Third, the reference dictionaries are compressed to approximately one tenth of the original data using Vector Quantization with the K-means clustering algorithm. Then, the compressed reference dictionaries are again optimized by the GLVQ algorithm. Experimental results for Chinese character recognition show that the dictionary can be successfully compressed without decreasing the recognition rate, and the calculation time of distance between strokes can be reduced by a factor of approximately five.

1

Introduction

On-line recognition of handwritten cursive characters is a key issue in state-ofthe-art character recognition research [8, 16], and extensive research has been conducted to accommodate the variation seen in stroke order, stroke number and stroke deformation [7, 9, 10, 11, 12, 13, 14, 15]. The recognition framework is usually based on pattern matching between the input and a reference pattern. However, the accuracy of character recognition is greatly influenced by the quality of the reference characters. In the case of Chinese characters, their reference dictionary consists of approximately one thousand different characters. To realize high quality recognition on a small microprocessor with built-in memory, for example a Personal Digital Assistant (PDA), the computational resources and recognition time required must be reduced. In addition, each character change by those who write the reference characters needs to be optimized. The aim of this paper is the compression and quality improvement of reference characters. This aim can be realized by making a smaller dictionary

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 300–309, 2003. c Springer-Verlag Berlin Heidelberg 2003


A Study of the Compression Method for a Reference Character Dictionary

301

based on stroke elements, not on character elements. First, one reference pattern for each category is generated from the training data. Second, the reference dictionary is optimized using the Generalized Learning Vector Quantization (GLVQ) algorithm [1],[2]. Third, the reference dictionary is compressed considerably compared with the original data using the Vector Quantization with K-means Clustering algorithm. Then, the GLVQ algorithm again optimizes the compressed reference dictionaries. In this paper, a suitable compression rate is investigated, because a trade-off exists between the recognition rate and recognition time/compression rate. Experimental results for Chinese character recognition show that the dictionary can be successfully compressed without decreasing the recognition rate, and the calculation time of distance between strokes can be considerably reduced.

2

Reference Pattern Generation System

Figure 1 shows the block diagram of the compression algorithm employed for the reference dictionary. First, each stroke of a preliminary reference pattern is generated from the training data by storing the average values of the loci of feature points in each stroke. One reference pattern for each category is made. Second, the vector of the preliminary reference stroke is corrected to the optimal vector by repeating learning using the GLVQ algorithm. Third, using the K-means clustering algorithm, the radicals composed by strokes having similar position and shape are combined to form a single compressed radical. Four, for all the strokes, the compression process is performed and a new reference dictionary is generated using the K-means clustering algorithm. Five, each compressed stroke in the new reference dictionary is again optimized using the GLVQ algorithm. Then, the final reference character patterns are created.

3

Average Pattern Generation

The experimental data consists of 2965 categories of Chinese characters, i.e., specified characters in the first level of the Japanese Industry Standard (JIS) code set, written by 90 university students (a total of 258603 characters having up to 26 strokes each). Students were directed to write cursively in a normal manner. The data were obtained using a stylus pen on a Liquid Crystal Display (LCD) tablet. The input character is transformed into a planar 256x256 mesh by preprocessing. The steps are redundancy elimination, smoothing, size normalization,

Average Pattern Generation

Optimization by GLVQ

Compression by Radical Units

Compression by Stroke Units

Fig. 1. Generation flow of the reference dictionary

Optimization by GLVQ

302

Jungpil Shin

(a)

(b)

Fig. 2. Examples of (a) input pattern, and (b) preliminary reference pattern

and feature point extraction. As feature information, the x-y coordinates and movement direction vector between one point and the next are extracted from the character data. Preliminary reference patterns are made as follows: First, the non-connected strokes are extracted by rearranging the strokes in the correct stroke order. Then, each stroke of the preliminary reference pattern is generated from the training data by storing the average values of the loci of the feature points in each stroke. One reference pattern for each category is generated. The generated reference patterns are divided stroke by stroke and numbered sequentially. The numbers are memorized in a code table and the actual recognition process is performed by referring to the table. The average of an original stroke that is referred to in the code table is denoted by C a , And the total number of C a is 32398. The typical examples of input and reference patterns of ” ” are shown in Figure 2. Automatic searching for correct stroke-correspondence between input pattern and reference pattern is not trivial, because the analysis of stroke order and stroke connection should be performed simultaneously with high accuracy. First, using the method of Cube Search [14, 15], correct stroke-correspondence between input pattern and reference pattern are searched for automatically by backtracking. This algorithm has a novel advantage that enables an efficient search for the optimal stroke-correspondence in spite of: (i) stroke order variation, (ii) stroke number variation due to stroke connection, and (iii) exceptional user-generated stroke deformations. Some of the wrong stroke-correspondence is manually converted into the appropriate stroke-correspondence by observation of these characters. These errors are due to strokes written in substantially different positions.

4

Generalized Learning Vector Quantization

The Generalized Learning Vector Quantization (GLVQ) algorithm [1, 2] is based on the Learning Vector Quantization (LVQ) algorithm proposed by Kohonen [5,

A Study of the Compression Method for a Reference Character Dictionary

303

different pattern w’2 input pattern same pattern w’1 (xl, yl)

(xj,yj)

(xi,yi)

Fig. 3. Sample vectors to match among feature points after changes using GLVQ

6], and it has been shown to be superior to LVQ. The purpose of these methods is to correct a reference vector to the optimal vector by repeating learning. We employ the GLVQ algorithm for reference pattern optimization. Briefly, one reference vector near the input vector v and belonging to the same category as v is set to ω1 , and another reference vector near v belonging to a different category from v is set to ω2 . Each distance is set to d1 and d2 , respectively. Two vectors of ω1 and ω2 are changed using Eqs. (1) and (2), respectively. That is, a similar reference pattern approaches an input pattern and a different pattern deviates from an input pattern. Figure 3 shows intuitively the vector changes from the original vector. ω1 ← ω1 + α

∂f d2 (v − ω1 ) ∂µ (d1 + d2 )2

(1)

ω2 ← ω2 − α

d1 ∂f (v − ω2 ) ∂µ (d1 + d2 )2

(2)

where, α is a small positive constant that controls the vector movement. In this ∂f = 4F (µ, t){1 − F (µ, t)} was used in the experiments, where t is the paper, ∂µ 2 learning time, F (µ, t) is a sigmoid function of 1+e1−µt , and µ = dd11 −d +d2 . The optimized reference stroke by GLVQ is denoted by C g . Examples of C g and its code number are shown in Figure 4.

5

Compressing Reference Characters

Using the preliminary reference pattern optimized by GLVQ, we generate a new compressed reference pattern using the K-means clustering algorithms as follows: 5.1

Compression by Radical Units

The compressed VQ code for radical units, the so-called ”Hen” and ”Tsukuri”, which is denoted by C r , is generated from C g using the K-means clustering algorithm, which performs on those radicals with the same number of strokes.

304

Jungpil Shin

4234

4237

4238

4235 405

407

4239

15052

27567 27566

15053 15059

15054

4236

15055

408 4240

15056

406

15060

15061

15057 15062

27580

4241

27568

27569 27571 27570 27574 27572 27573 27575 27579 27578

27576

27577

27581

15058

Fig. 4. Examples of stroke C g

Fig. 5. Number of strokes in radical The goal is to make C r substitute for similarly shaped strokes in C g using the average of each stroke having the same position and shape. By this process we can reduce the misrecognition, which is determined by a different stroke in spite of the same radical of stroke position and shape. Furthermore, the next clustering stage works more effectively. Figure 5 shows the number of strokes in the radical and the number of its variations. Examples of a reference character with radical ” ” and its compressed strokes C r are shown in Figure 6.

(a)

(b)

Fig. 6. Examples of (a) reference character with the same radical and (b) its compressed strokes C r

A Study of the Compression Method for a Reference Character Dictionary

305

Fig. 7. Examples of original and compressed reference characters

5.2

Compression by Stroke Units

The compressed VQ code for each stroke unit, which is denoted by C s , is again generated from C r using the K-means clustering algorithm. The compressed size of the reference pattern is determined by the threshold value that determines whether a stroke belongs in the same cluster as other strokes. It is expected that the size of a dictionary can be reduced, without decreasing the rate of recognition, by compressing the reference dictionary using vector quantization. Moreover, the computational effort to calculate the distance between the input stroke and a reference stroke can be reduced dramatically as follows. First, the distance values calculated are immediately saved in the table. Then, when using the same value again, it is read from a table and is used without calculation. A suitable compression rate should be investigated because a trade-off exists between recognition rate and recognition time/compression rate. Figure 7 shows the examples of the original reference characters and the compressed reference characters, which are recreated by the VQ code. In the case of Ref 1 of Figure 7, the number of strokes was compressed from 32398 to 7372, a compression rate of 22.8%. A maximum of 96 strokes were compressed into one stroke. Figure 8 shows the number of strokes compressed to one stroke in the case of Ref 3 . Although the reference characters were compressed, compared with the original characters, characters from Ref 1 to Ref 3 did not lose shape significantly. The actual compression rate is shown in Table 1. Using DP matching, the distance between an input stroke and a reference stroke in the K-means algorithm for radical and stroke units is calculated by the weighted sum of (1) the distance between x-y coordinate sequences and (2) the distance between directional vector sequences. An asymmetric DP equation is employed for DP matching.

306

Jungpil Shin

Fig. 8. Compression result Table 1. Total stroke number and compression rate ref. set total number of str. compression rate maximum str. no. in one category Ref 0 32398 Ref 1 7372 22.8 % 96 Ref 2 5657 17.5 % 123 Ref 3 3148 9.7 % 186 Ref 4 1899 5.9 % 305 Ref 5 1203 3.7 % 413 Ref 6 783 2.4 % 494 Ref 7 551 1.7 % 753 Ref 8 317 1.0 % 1310

5.3

Generalized Learning Vector Quantization

Each compressed stroke of C s is again optimized using the GLVQ algorithm, and is denoted by C m . Then, the final reference character patterns are created. Using this compressed reference dictionary of stroke C m with small size, it is expected that the recognition time will be reduced greatly without decreasing the recognition rate compared with the non-compressed dictionary.

6

Recognition Experiment

The usefulness of the presented method was demonstrated by recognition experiments performed on a PC, a Pentium III 700 MHz processor. Training data consisted of 2965 Chinese character categories, used as the investigation data in Sec. 3. Another 20 writers provided test data for the recognition experiment in

A Study of the Compression Method for a Reference Character Dictionary

307

Fig. 9. Recognition rates for each stroke number Table 2. Recognition rate (%) for using strokes (a) C g , (b) C s , and (c) C m ; (d) the improvement (%) between C s and C m ref. set Ref 0 Ref 1 Ref 2 Ref 3 Ref 4 Ref 5 Ref 6 Ref 7 Ref 8

(a) 99.0 -

(b) 98.2 98.1 97.9 97.2 97.0 96.5 95.8 94.2

(c) 98.9 98.9 98.8 98.1 98.0 97.4 96.9 95.6

(d) +0.7 +0.8 +0.9 +0.9 +1.0 +0.9 +1.1 +1.4

a similar fashion to the training data. The data consisted of the same characters as those of the training data and totalled 48631 characters, and were limited to characters with the correct number of strokes. Recognition experiments using C a , C g , C s , and C m as reference patterns were conducted. For the recognition algorithm, we used the method of Cube Search [14, 15]. Table 2 summarizes the recognition experiment results. A considerable improvement in recognition rate is achieved between using stroke C s and C m . The result of (b) is that it succeeded in compression of data without decreasing the recognition rate. The result of (c), compared with VQ references, is that the average recognition rate increased about 1%, and recognition rates from Ref 1 to Ref 3 hardly

308

Jungpil Shin

changed compared with Ref 0 . Therefore, Ref 3 is the most suitable as the reference characters. The stroke distance calculation time in case of Ref 3 can be reduced by a factor of approximately five, from 1.23 sec to 0.25 sec. Moreover, the recognition rate of each stroke was investigated and the result is shown in Figure 9. When the compression rate becomes high, in the case of few stroke characters, the recognition rate tends to fall. For this research, a program, as shown in Figure 10, that can select and display Chinese characters having reference strokes included in the same category was written using the C++ language.

Fig. 10. An application that displays VQ character

7

Conclusion

We have investigated a compression method for a reference character dictionary used for on-line Chinese handwriting character recognition. By a combination of the K-means clustering algorithm and the GLVQ algorithm, the reference dictionary was successfully compressed to one tenth of the original data size without a decrease in the recognition rate. The time used to calculate distance between strokes can be reduced to approximately one fifth. Future work includes applying the proposed method to a reference dictionary in which one category of reference stroke consists of multiple strokes rather than a single stroke. We expect that a more precise recognition algorithm can be found.

A Study of the Compression Method for a Reference Character Dictionary

309

References [1] A. Sato, K. Yamada, ”Character Recognition using Generalized Learning Vector Quantization”, Technical Report of IEICE, PRU95-219, pp. 23-30, March 1996. 301, 302 [2] A. Sato, K. Yamada, ”Generalized Learning Vector Quantization”, Advances in Neural Information Processing Systems 8, The MIT Press, 1996. 301, 302 [3] A. Kitadai, M. Nakagawa, ”A Learning Algorithm for Structural Pattern Representation used in On-Line Handwritten Character Recognition”, Technical Report of IEICE PRMU2000-209(2001-03), pp. 61-68. [4] Mu-King Tsay, Keh-Hwa Shyu, Pao-Chung Chang, ”Feature Transformation with Generalized Learning Vector Quantization for Hand-Written Chinese Character Recognition”, IEICE trans. Vol. E82-D, No.3 pp. 687-692, March 1999. [5] T. Kohonen, ”Self-Organization and Associative Memory”, 3rd ed., SpringerVerlag, 1989. 303 [6] T. Kohonen, ”LVQ PAK Version 3.1 -The Learning Vector Quantization Program Package,” LVQ Programming Team of the Helsinki University of Technology, 1995. 303 [7] M. Nakagawa, “Non-keyboard Input of Japanese Text – On-Line Recognition of Handwritten Characters as the Most Hopeful Approach,” IPSJ Trans., Japan, vol. 13, no. 1, pp. 15-34, April 1990. 300 [8] C. C. Tappert, C. Y. Suen, and T. Wakahara, “The State of the Art in On-Line Handwriting Recognition,” IEEE Trans. Pattern Anal. Machine Intell., vol. 12, no. 8, pp. 787–808, Aug. 1990. 300 [9] M. Nakagawa and K. Akiyama, “A Linear-time Elastic Matching for Stroke Number Free Recognition of On-Line Handwritten Characters,” Proc. 4th Int. Workshop on Frontiers in Handwriting Recognition, pp.48-56, Dec. 1994. 300 [10] A. Hsieh, K. Fan, and T. Fan, “Bipartite Weighted Matching for On-Line Handwritten Chinese Character Recognition,” Pattern Recognition, Vol.28, No.2, pp.143-151, 1995. 300 [11] T. Wakahara, A. Suzuki, N. Nakajima, S. Miyahara, and K. Odaka, “StrokeNumber and Stroke-Order Free On-Line Kanji Character Recognition as One-toOne Stroke Correspondence Problem,” IEICE Trans. Inf. & Syst., vol. E79-D, no. 5, pp. 529–534, May 1996. 300 [12] T. Uchiyama, N. Sonehara, and Y. Tokunaga, “On-Line Handwritten Character Recognition Based on Non-Euclidean Distance,” IEICE Trans. Inf. & Syst., vol. J80-D-II, No. 10, pp. 2705–2712, Oct. 1997 (in Japanese). 300 [13] T. Wakahara and K. Odaka, ”On-Line Cursive Kanji Character Recognition Using Stroke-Based Affine Transformation,” IEEE Trans. Pattern Anal. Machine Intell., vol. 19, no. 12, pp. 1381–1385, Dec. 1997. 300 [14] J. Shin, M. M. Ali, Y. Katayama, and H. Sakoe, “Stroke Order Free On-Line Character Recognition Algorithm Using Inter-Stroke Information,” IEICE Trans. Inf. & Syst., vol. J82-D-II, No. 3, pp. 382–389, Mar. 1999 (in Japanese). 300, 302, 307 [15] J. Shin and H. Sakoe, “Stroke Correspondence Search Method for Stroke-Order and Stroke-Number Free On-Line Character Recognition — Multilayer Cube Search,” IEICE Trans. Inf. & Syst., vol. J82-D-II, No. 2, pp. 230–239, Feb. 1999 (in Japanese). 300, 302, 307 [16] R. Plamondon, D. P. Lopresti, L. R. B. Schomaker, and R. Srihari, “On-line handwriting recognition,” In: J. G. Webster (Ed.). Wiley Ency clopedia of Electrical & Electronics Engineering, 123-146, New York, 1999. 300

Mercer Kernels and 1-Cohomology of Certain Semi-simple Lie Groups Bernd-Jrü gen Falkowski University of Applied Sciences Stralsund Zur Schwedenschanze 15, D-18435 Stralsund, Germany [email protected]

Abstract. For the construction of support vector machines so-called Mercer kernels are of considerable importance. Since the conditions of Mercer's theorem are hard to verify some mathematical results arising from semi-simple Lie groups are collected here to provide concrete examples of Mercer kernels for the real line. Besides an interesting connection to Furstenberg's theory of noncommuting random products comes to light. These results have, in essence, been known for quite some time but are rather technical in nature. Hence a concise treatment is offered here to make them accessible to Neural Network researchers.

1

Introduction

Recently support vector machines, cf. e.g. [Sch], have received much attention. In this context Mercer kernels, cf. e.g. [Cri], p. 35 for Mercer's theorem, which are important building blocks of such machines, have frequently been used. Thus in [Fal2] some mathematical results providing a complete description of all Mercer kernels possessing certain invariance properties under a group action were collected. These were used to derive two out of three basic types of kernels, cf. e.g. [Hay], p. 333, and one less well-known kernel type in a systematic fashion. However, apart from these and some construction rules, cf. e.g. [Cri], pp. 42-44, very little else appears to be known about such kernels amongst Neural Network researchers. This is all the more surprising since the conditions of Mercer's theorem are not easily verifiable in general. Hence it seems worthwhile to point out some results for semi-simple Lie groups that also have, in essence, been known for quite some time, cf. [Fur], [Gan], [Erv], to provide further concrete examples of Mercer kernels. Besides an interesting connection to Furstenberg's theory of noncommuting random products and associated laws of large numbers comes to light. Note that due to lack of space and the very technical nature of the subject only sketch proofs will be given. For technical details and further background information we refer to [Fur], [Gan], [Hel], [Erv].

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 310-316, 2003.  Springer-Verlag Berlin Heidelberg 2003

Mercer Kernels and 1-Cohomology of Certain Semi-simple Lie Groups

2

311

Some Basic Definitions and Results

In this section some basic definitions and results from [Fal2] are going to be recalled for the convenience of the reader. Definition 1: Let G be a topological group and let g → Ug be a weakly continuous unitary representation of G in a Hilbert space H (i.e. in particular a homomorphism). A map δ: G → H is called a first order cocycle (or just a cocycle) associated with U if Ug δ(h) = δ(gh) - δ(g)

∀g, h ∈ G

Example 1: Suppose that v is a fixed vector in H. Then a trivial cocycle (coboundary) associated with U may be defined by setting δ(g):= Ug v – v. In [Fal2] it was shown how Mercer kernels enjoying certain invariance properties may be constructed from positive definite functions. Moreover the logarithms of such functions (so-called conditionally positive definite functions) were described in terms of the 1-cohomology there. The (relevant part of the) basic result is: Theorem 1: If δ is a cocycle satisfying δ(e) = 0, then the function ψ(g) = - c< δ (g), δ (g)> is normalized conditionally positive definite for every c>0 and hence ϕ(g):= exp[ψ(g)] is positive definite in the usual sense. Mercer kernels K may be obtained from these functions for example by setting K(g1, g2):= ϕ(g −1 g1) or 2 1 − K(g1, g2):= ψ (g g1) − ψ (g −1) − ψ (g1) 2 2 Proof: See example 2, lemma 1.2, and theorem3.2 in [Fal2]. Unfortunately the 1-cohomology for topological groups is by no means easy to calculate explicitly in general, although some results are available for ℜn, cf. [Fal2]. However, strangely enough some known facts about non-commuting random products for semi-simple Lie groups prove helpful to obtain very concrete results.

3

Semi-simple Lie Groups with Finite Centre

Note first of all that in theorem 1 above an explicit knowledge of δ may not be required. Indeed, suppose that G is a semi-simple Lie group with finite centre. Let G = KAN, with K compact, A abelian, and N nilpotent, be its Iwasawa decomposition, cf. e.g. [Hel] for the technical details. Suppose further that a nontrivial cocycle of G associated with U satisfies δ(k) = 0 ∀k ∈ K . Then it follows that U cannot have a non-trivial vector vector invariant under K, cf. [Erv], p. 88. From this fact one obtains: Lemma 1: The function ψ given in theorem 1 satisfies the functional equation

312

Bernd-Jürgen Falkowski

∫ ψ(g1k g 2)dk = ψ(g1) + ψ(g 2) ∀ g1 , g 2 ∈ G ------------------------------(*)

K

where dk denotes the Haar measure on K. Proof: See [Erv], p. 90. The equation (*) above is quite remarkable since it appears in many different contexts, cf. e.g. [Fur] for non-commuting random products and laws of large numbers, or [Gan], [Fal1] for Levy-Schoenberg Kernels. If it has a unique nonnegative solution and a non-trivial cocycle is known to exist, then the negative of its solution is conditionally positive definite. In [Fur] non-negative solutions of (*) are described as follows: Let G = KAN as above, and let MAN, where M is the centralizer of A in K, be a minimal parabolic subgroup, cf. e.g. [Fur] for the technical details. Consider B = KAN/MAN. Then K acts transitively on this homogeneous space and in fact there exists a unique K-invariant probability measure m on B, cf. [Fur]. In order to describe the solutions of (*) a further definition of [Fur] is needed. Definition 2: If G acts transitively on a topological space X, then an A-cocycle on X is a real-valued continuous function ρ on G×X satisfying (i) (ii)

ρ(g1g2,x) = ρ(g1,g2x) + ρ(g2,x) ρ(k,x) = 0 ∀k∈K

∀ g1, g2 ∈ G, x ∈ X

Then the following theorem holds, cf.[Fur], [Fal1]: Theorem 2: The function ψρ defined by ψρ(g):= ∫ ρ(g, x )dm( x ) B

is a non-negative solution of (*). Note that the ψρ thus constructed is biinvariant with respect to K, i.e. satisfies the condition ψρ(k1gk2) = ψρ(g) ∀ k1, k2 ∈ K, ∀ g ∈ G, which is rather useful since any element in G may be written as g = k1ak2 with k1, k2 ∈ K, a ∈ A in view of the Cartan decomposition of G, cf. [Hel]. Hence it will suffice to compute ψρ on A.

4

Concrete Examples (SOe(n;1) and SU(n;1))

As examples the real and complex Lorentz groups will be treated in this section (actually in the real case only the connected component of the identity will be dealt with). Definition 3: SOe(n;1) (SU(n;1)) are real (complex) (n+1)×(n+1) matrix groups which leave the real (hermitian) quadratic forms n −1 ∑ x i2 − x 2 n i=0

invariant.

n −1 ( ∑ | x i |2− | x n |2) i=0

Mercer Kernels and 1-Cohomology of Certain Semi-simple Lie Groups

313

The homogeneous G-space B = KAN/MAN in either case turns out to be the (real (complex)) Sphere Sn-1 on which an element g := [gij] for 0 ≤ i, j ≤ n of G acts by fractional linear transformations, i.e. n −1 n −1 (gx)i:= ( ∑ g ik x k + g in ) /( ∑ g nk x k + g nn ) k=0 k=0 n −1 n −1 ∑ x i2 = 1 ( ∑ (| x i | 2 = 1) and x := (x0, x2, …, xn-1). i=0 i=0 It is in fact known, cf. [Fur], that there is, up to scalar multiples, precisely one Acocycle in either case, which is described in:

where

Lemma 2: Let g ∈ SOe(n;1) (SU(n;1)) be given by g := [gij] for 0 ≤ i, j ≤ n. Then ρ(g,x) := log| where

n −1 ∑ g nk x k + g nn |, k=0

x := (x0, x2, …, xn-1), is an A-cocycle.

Proof: Computation (note here that the invariance conditions given in definition 3 of course impose certain restrictions on the gij which have not been given here but must be used in the computation). Q.E.D. Next note that the abelian group A appearing in the Cartan decomposition is given by the special elements g := [gij] for 0 ≤ i, j ≤ n satisfying gij = δij (Kronecker delta) for 0 ≤ i, j ≤ n-2, for some t ∈ ℜ, gn-1n-1 = gnn = cosht, gn-1n = gnn-1 = sinht all other gij being zero. Hence A is isomorphic in either case to the group of hyperbolic rotations in the plane or equivalently (ℜ, +). If one denotes a typical element of A by a(t) in this case, then ρ(a(t),x) = log|xn-1*sinht + cosht|. From this one then immediately obtains the following theorem: Theorem 3: The negative of the function ψρ(t) defined by π

ψρ(t):= ∫ log(sinh t cos u + cosh t ) sin n − 2 u du 0

is conditionally positive definite on the real line for every n ≥ 2. Proof: By theorem 2, lemma 2, and the remarks above the negative of the function fρ(t) originally defined by fρ(t):=

∫ ρ(a ( t ), x )dm( x ) n −1

S

on SOe(n;1) is conditionally positive definite on the real line for every n. But from this one immediately obtains the expression given in theorem 3 (modulo a constant) on transforming to polar coordinates, cf. e.g. [Mag]. Q.E.D.

314

Bernd-Jürgen Falkowski

Note that the integral for n = 3 may be evaluated and one thus obtains an interesting explicit result. Corollary to Theorem 3: The negative of the function ψρ(t) defined by ψρ(t):= t*coth t – 1 is conditionally positive definite on the real line. Hence the kernels K1 and K2 defined by K1(t1,t2):= exp[1-(t1-t2)*coth (t1-t2)] respectively K2(t1,t2):= -(t1-t2)*coth (t1-t2) + t2*coth t2+ t1*coth t1 –1 are Mercer kernels on the real line. π

Proof: Note that ∫ log(sinh t cos u + cosh t ) sin u du = 2*(t*coth t – 1), discard the 0

factor 2 (clearly any non-negative constant factor could appear in the kernels without altering the fact that they are Mercer kernels) and apply theorem 1. Q.E.D. Remark: If one is willing to invest a little more effort, then the case n = 2 can be evaluated explicitly as well, cf. e.g. [Gan] for a similar calculation. This computation has not been performed here since the result is similar to the case discussed in theorem 4 below. In the complex case the calculations are slightly more complicated and thus only an interesting special case will betreated here. For SU(1;1) one obtains the following result: Theorem 4: The negative of the function ψρ(t) defined by ψρ(t):= log(cosht) is conditionally positive definite on the real line. Hence the kernels K3 and K4 defined by K3(t1,t2):= 1/cosh(t1-t2) respectively K4(t1,t2):= log[(cosht1*cosht2)/cosh(t1-t2)] are Mercer kernels on the real line. Proof: For SU(1;1) an invariant measure on the complex unit circle is given by (1/2πi)dx0/x0 where |x0| = 1. Hence 1 / 2πi ∫ log | sinh t x 0 + cosh t | d x 0 / x 0 = log(cosh t) by the Cauchy residue 1

S

theorem, cf. e.g. [Ahl]. Again apply theorem 1 to obtain the result.

Q.E.D.

Mercer Kernels and 1-Cohomology of Certain Semi-simple Lie Groups

5

315

Concluding Remarks

The functions described in theorem 2 describe the Gaussian part of a noncommutative analogue of the Levy-Khintchin formula, cf. e.g. [Gan]. Indeed, if one wishes to consider more general Mercer kernels, then this is perfectly possible, see e.g. [Fal1] for a more thorough discussion of the situation, which would have exceeded the framework of this paper. However, there is also an interesting connection to a law of large numbers which should not be overlooked. If {Xn} is a sequence of independent identically distributed random variables with values in a multiplicative group of positive real numbers, then the law of large numbers can be stated as follows: (1/n)log(X1X2 … Xn) → E(logXn), where E denotes the expectation value. The function ψ(t):= log t is characterized up to a constant multiple by the equation log (t1t2) = log t1 + log t2. However, a semi-simple Lie group does not admit any non-trivial homomorphisms and hence it would be pointless to look for an exact analogue of this function. The relevant analogue is the functional equation (*) given above and using solutions of this functional equation a law of large numbers can indeed be proved in the non-commutative situation prevailing in semisimple Lie groups as discussed above, for details see e.g. [Fur]. Since only Mercer kernels have been described above it may be worthwhile to remind the reader that from these the relevant feature spaces may be constructed as Hilbert spaces. In this case the Mercer kernels are used to define the inner product (that is to say they are reproducing kernels). Alternatively they may be used to define positive integral operators where the feature maps then arise as eigenfunctions belonging to positive eigenvalues. For a more detailed discussion of these aspects the reader is referred to [Sch] chapter 1. Finally, from a practical point of view, for example the very explicit result in the corollary to theorem 3 seems noteworthy, since even for the hyperbolic tangent kernel the positive definiteness criterion is satisfied only for some parameter values, cf. e.g. [Hay], p. 333.

References [Ahl] [Cri] [Erv] [Fal1] [Fal2]

Ahlfors, L.V.: Complex Analysis. Second Edition, (1966) Cristianini, N.; Shawe-Taylor, J.: An Introduction to Support Vector Machines and other Kernel-Based Learning Methods. Cambridge University Press, (2000) Erven, J.; Falkowski, B.-J.: Low Order Cohomology and Applications. Springer Lecture Notes in Mathematics, Vol. 877, (1981) Falkowski, B.-J.: Levy-Schoenberg Kernels on Riemannian Symmetric Spaces of Non-Compact Type. In: Probability Measures on Groups, Ed. H. Heyer, Springer Lecture Notes in Mathematics, Vol. 1210, (1986) Falkowski, B.-J.: Mercer Kernels and 1-Cohomology. In: Proc. of the 5th Intl. Conference on Knowledge Based Intelligent Engineering Systems & Allied Technologies (KES 2001), Eds. N. Baba, R.J. Howlett, L.C. Jain, IOS Press, (2001)

316

[Fur] [Gan] [Hay] [Hel] [Mag] [Sch]

Bernd-Jürgen Falkowski

Furstenberg, H.: Noncommuting Random Products. Trans. of the AMS,Vol. 108, (1962) Gangolli, R.: Positive Definite Kernels on Homogeneous Spaces. In: Ann. Inst. H. Poincare B, Vol. 3, (1967) Haykin, S.: Neural Networks, a Comprehensive Foundation. Prentice-Hall, (1999) Helgason, S.: Differential Geometry and Symmetric Spaces. Academic Press, New York, (1963) Magnus, W.; Oberhettinger, F.: Formeln und Sätze frü die speziellen Funktionen der Mathematischen Physik. Springer, (1948) Schölkopf, B.; Burges, J.C.; Smol a, A.J. (Eds.): Advances in Kernel Methods, Support Vector Learning. MIT Press, (1999)

On-line Profit Sharing Works Efficiently Tohgoroh Matsui1 , Nobuhiro Inuzuka2 , and Hirohisa Seki2 1

2

Department of Industrial Administration, Faculty of Science and Technology Tokyo University of Science, 2641 Yamazaki, Noda-shi, Chiba 278-8510, Japan [email protected] Department of Computer Science and Engineering, Graduate School of Engineering Nagoya Institute of Technology Gokiso-cho, Showa-ku, Nagoya 466-8555, Japan {inuzuka@elcom,seki@ics}.nitech.ac.jp

Abstract. Reinforcement learning constructs knowledge containing state-to-action decision rules from agent’s experiences. Most of reinforcement learning methods are action-value estimation methods which estimate the true values of state-action pairs and derive the optimal policy from the value estimates. However, these methods have a serious drawback that they stray when the values for the “opposite” actions, such as moving left and moving right, are equal. This paper describes the basic mechanism of on-line profit-sharing (OnPS) which is an actionpreference learning method. The main contribution of this paper is to show the equivalence of off-line and on-line in profit sharing. We also show a typical benchmark example for comparison between OnPS and Q-learning.

1

Introduction

Reinforcement learning constructs knowledge containing state-to-action decision rules from agent’s experiences. Most of reinforcement learning methods are based on action-value function which is defined as the expected return starting from state-action pairs, and called action-value estimation methods [5]. Action-value estimation methods estimate the true values of state-action pairs and derive the optimal policy from the value estimates. However, these methods have a serious drawback that they stray when the values for the “opposite” actions, such as moving left and moving right, are equal. This paper describes the basic mechanism of on-line profit-sharing (OnPS). Profit sharing [2, 3] is an action-preference learning method, which maintains preferences for each action instead of action-value estimates. Profit sharing1 has an advantage of the number of parameters to be set, because it only needs a discount rate parameter. Moreover, Arai et al. [1] have shown that profit sharing outperforms Q-learning in a multi-agent domain. 1

Note that profit sharing sometimes converges to a locally optimal solution like EM algorithm or neural networks.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 317–324, 2003. c Springer-Verlag Berlin Heidelberg 2003


318

Tohgoroh Matsui et al.

Profit sharing is usually an off-line updating method, which does not change its knowledge until the end of a task. One drawback is that ordinary profit sharing requires unbounded memory space to store all of the selected stateaction pairs during the task. Another disadvantage is that intermediate rewards until the end of the task are considered to be zero. Although OnPS is one of the profit-sharing methods, it can be implemented with bounded memory space and use intermediate rewards during the task for updating its knowledge. The main contribution of this paper is to show the equivalence of off-line and on-line in profit sharing. We also give the result of a typical benchmark example for comparison between OnPS and Q-learning [6].

2

Profit Sharing

Profit sharing [2, 3] considers the standard episodic reinforcement learning framework, in which an agent interacts with an environment [5]. In profit sharing, state-action pairs in a single episode of T time steps, s0 , a0 , r1 , s1 , a1 , r2 , · · · , rT , sT with states st ∈ S, actions at ∈ A, and rewards rt ∈ , are reinforced at the end of the episode. We consider the initial state, s0 , to be given arbitrarily. In episodic reinforcement learning tasks, each episode ends in a special state called the terminal state. Profit sharing learns the action-preferences, P , of state-action pairs. After an episode is completed, P is updated by P (st , at ) ← P (st , at ) + f (t, rT , T ),

(1)

for each st ∈ S, at ∈ A in the episode, where f is a function, called credit assignment. The rationality theorem [4] guarantees that unnecessary state-action pairs, that is, pairs evermore make up the loop, are always given smaller increments than the necessary pairs, if the credit-assignment function satisfies f (t, rT , T ) > L

t−1


f (i, rT , T ),

(2)

i=0

where L is the maximum number of pairs, except for unnecessary ones, in a state. In many cases, a function f (t, rT , T ) = γ T −t−1 rT ,

(3)

is used, where γ is a parameter, also called the discount rate. This function satisfies Equation (2) with γ ≤ 1/ max |A(s)| ≤ 1/L. We also use this function in this paper. The shape of the function is shown in Figure 1. The increments for P (st , at ) decrease as the state-action pair, st and at , was visited earlier. Profit

On-line Profit Sharing Works Efficiently

319

rT

f (t, r T , T) = γ T − t − 1 r T γ1 rT γ2 rT ...

T− 3

T−2

T−1

Step t

Fig. 1. A credit-assignment function in the form of Equation (3), in profit sharing. The increments of P (st , at ) decrease as the difference between t and T increases

sharing chooses action a with probability in proportion to action preferences at state s: P (s, a) Pr(at = a|st = s) =  .  a
∈A(s) P (s, a ) This is called weighted-roulette. Figure 2 shows the algorithm for the profitsharing method.

Initialize, for all s ∈ S, a ∈ A(s): P (s, a) ← a small constant Repeat (for each episode): Initialize s Repeat (for each step of episode): Choose a from s using weighted-roulette derived from P Take action a; observe reward, r, and next state, s
s ← s
until s is terminal For each pair st , at appearing in the episode: P (st , at ) ← P (st , at ) + f (t, rT , T )

Fig. 2. Off-line profit-sharing algorithm. The function f is a credit-assignment function

3

On-line Profit-Sharing Method

The ordinary profit-sharing method has the problem that its memory and computational requirements increase without bound, because the method must store

320

Tohgoroh Matsui et al. c(s,a) Steps

Visiting to this state-action pair

Fig. 3. The credit trace for a state-action pair. It decays by γ on each step and is incremented by one when the state-action pair is visited in a step, in accordance with Equation (4)

all the experienced states, the selected actions, and the final reward. That is, it requires s0 , a0 , s1 , a1 , · · · , sT −1 , aT −1 , rT to calculate the updates of all of P for state-action pairs st and at in an episode. On the other hand, incremental methods compute the (k + 1)th values from the kth values, so that it is not necessary to keep earlier state-action pairs. Instead, our incremental implementation method requires additional but bounded memory for credit traces. Credit traces, c, maintain credit assignments for each state-action pair. They are based on eligibility traces [5] which are one of the basic mechanisms of reinforcement learning. We apply this technique for implementing the profit-sharing method incrementally. We call this version of the profit-sharing method the online profit-sharing method (OnPS) because updates are done at each step during an episode, as soon as the increment is computed. Because the number of elements in the credit trace, c, is equal to that in the preference, P , the size of required memory is bounded. In each step, the credit traces for all states decay by γ, while the trace for the state-action pair visited on the step is incremented by one:  γct−1 (s, a) + 1 if s = st and a = at ; ct (s, a) = γct−1 (s, a) otherwise, for all non-terminal states s ∈ S, a ∈ A. The credit trace accumulates each time when the state-action pair is visited, then decays gradually while the state-action pair is not visited, as illustrated in Figure 3. The increments are given as follows: ∆Pt (s, a) = rt+1 ct (s, a) for all s, a. Figure 4 shows the algorithm for OnPS.

(5)

On-line Profit Sharing Works Efficiently

321

Initialize, for all s ∈ S, a ∈ A(s): P (s, a) ← a small constant Repeat (for each episode): Initialize s c(s, a) = 0, for all s, a Repeat (for each step of episode): Choose a from s using weighted-roulette derived from P c(s, a) ← c(s, a) + 1 Take action a; observe reward, r, and next state, s
For all s, a: P (s, a) ← P (s, a) + rc(s, a) c(s, a) ← γc(s, a) s ← s
until s is terminal

Fig. 4. On-line profit-sharing (OnPS) algorithm

4

Equivalence of Off-line and On-line Updating in Profit-Sharing Methods

In this section, we show that the sum of all the updates is exactly the same for the off-line and on-line profit-sharing methods over an episode. Theorem 1. If the rewards are zero until the system reaches one of the goal states and Equation (3) is used as the credit-assignment function, then the offline updating given in Equation (1) is equivalent to the on-line updating given in Equation (5). Proof. First, an accumulating eligibility trace can be written non-recursively as ct (s, a) =

t


γ t−k Issk Iaak ,

k=0

where sk and ak are states and actions that appeared in the episode respectively, and Ixy is an identity-indicator function, equal to one if x = y and equal to zero otherwise. The update of a step in the on-line profit-sharing method is Equation (5). Thus, the sum of all the updates can be written as ∆P (s, a) =

T −1


rt+1 ct (s, a) =

t=0

T −1
t=0

rt+1

t


γ t−k Issk Iaak .

(6)

k=0

On the other hand, the sum of updates in the off-line profit-sharing method can be written using the identity-indicator functions as ∆P (s, a) =

T −1
t=0

f (t, rT , T )Isst Iaat .

322

Tohgoroh Matsui et al.

G

F

S

G

F

Fig. 5. 11×11 grid world example. S is the initial state, and G’s are goal states. G’s and F’s are terminal states

Using Equation (3) as the credit-assignment function, f , this can be transformed to T −1 T −1

T −t−1 γ rT Isst Iaat = rT γ T −t−1 Isst Iaat . (7) ∆P (s, a) = t=0

t=0

Now, we rewrite the right-hand side of Equation (6). Because all the intermediate rewards until the end of the episode are zero, r1 = r2 = · · · = rT −1 = 0, only the term for t = T − 1 remains: ∆P (s, a) =

T −1


rt+1

t=0

=r1

0


t


γ t−k Issk Iaak

k=0

γ 0−k Issk Iaak + · · · + rT

k=0

=rT

T −1


T −1


γ (T −1)−k Issk Iaak

k=0

γ T −k−1 Issk Iaak .

k=0

Therefore, the sums of all the updates in the off-line and on-line profit-sharing methods are same, if we use Equation (3) as the credit-assignment function, and all of the intermediate rewards until the end of the episode are zero. ✷

5

An Example

We used 11×11 grid-world shown in Figure 5. The MDP is deterministic and has 4 actions, moving up, down, left, or right. If an agent bumps into a wall, it remains in the same state. The four corner states are terminal. The agent receives a reward of 1 for the actions entering the bottom-right and upper-left corners. All of the other rewards are 0, including for entering the other two corners. The initial state is in the center.

0.1

0.1

0.08

0.08

Rewards per Step

Rewards per Step

On-line Profit Sharing Works Efficiently

0.06 0.04 0.02

Q-learning 1

10

100

OnPS

0.06 0.04 0.02

OnPS

0

1000 1e+4 1e+5 1e+6 1e+7 1e+8 Number of Steps

323

Q-learning

0 1

10

100

1000 1e+4 1e+5 1e+6 1e+7 1e+8 Number of Steps

Fig. 6. The performances of greedy policy (left) and either weighted-roulette or softmax policy which are used during the learning (right) in the 11×11 environment. The optimal performance is 0.1

We have implemented OnPS with the initial preferences are 1/|A| = 0.25, γ = 1/|A| = 0.25, and Q-learning using Gibbs-distribution softmax action-selection with α = 0.01, γ = 0.9 and τ = 0.2. The results are shown in Figure 6. For each evaluation 101 trials are run with a maximum step cutoff at 101 steps. We ran a series of 30 runs and show the average. The left panel shows the performances of greedy policy derived from the same knowledge at each total time step. The right panel shows the performance comparison of probabilistic policies, that is, between weighted-roulette and softmax, which are used during the learning. The results indicate that OnPS outperforms Q-learning for accomplishing the task when they learn. Since the true values for moving all directions are the same at center, actionvalue estimation methods stray as the estimation approaches the true value. Therefore, the performance of Q-learning was poor using the stochastic policy.

6

Discussion and Conclusion

Monte Carlo control method (MC-control) [5] is the most closest method to OnPS. MC-control acquires the average returns per selection for each stateaction pair in order to estimate action-values. The essential difference is that MCcontrol is an action-value estimation method, while OnPS is an action-preference learning method. Hence, MC-control suffers from even-value actions when it explores. We proved the equivalence of off-line and on-line updating in profit sharing, if all of the intermediate rewards are zero. Profit sharing can accomplish the task even during the learning, although the acquired knowledge is not guaranteed to be optimal. Nevertheless, OnPS worked well in several benchmark problems [5]. OnPS is superior to off-line profit-sharing method from the view point of required memory space. Furthermore, OnPS can deal with intermediate rewards during

324

Tohgoroh Matsui et al.

the task, though details of the effects have not examined yet. Those theoretical analysis remains as future work.

References [1] Sachiyo Arai, Katia Sycara, and Terry R. Payne. Experience-based reinforcement learning to acquire effective behavior in a multi-agent domain. In R. Mizoguchi and J. Slaney, editors, Proceedings of The 6th Pacific Rim International Conference on Artificial Intelligence (PRICAI-2000), volume 1886 of Lecture Notes in Artificial Intelligence, pages 125–135. Springer-Verlag, 2000. 317 [2] John J. Grefenstette. Credit assignment in rule discovery systems based on genetic algorithms. Machine Learning, 3:225–245, 1988. 317, 318 [3] John H. Holland. Escaping brittleness: The possibilities of general-purpose learning algorithms applied to parallel rule-based systems. In R. S. Michalski, J. G. Carbonell, and T. M. Mitchell, editors, Machine Learning: An Artificial Intelligence Approach, volume 2. Morgan Kaufmann Publishers, 1986. 317, 318 [4] Kazuteru Miyazaki, Masayuki Yamamura, and Shigenobu Kobayashi. On the rationality of profit sharing in reinforcement learning. In Proceedings of The 3rd International Conference on Fuzzy Logic, Neural Nets and Soft Computing, pages 285–288. Fuzzy Logic Systems Institute, 1994. 318 [5] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 1998. 317, 318, 320, 323 [6] Christopher J. C. H. Watkins and Peter Dayan. Technical note: Q-learning. Machine Learning, 8(3/4):279–292, 1992. 318

Fast Feature Ranking Algorithm
Roberto Ruiz, Jos´e C. Riquelme, and Jes´ us S. Aguilar-Ruiz Departamento de Lenguajes y Sistemas, Universidad de Sevilla. Avda. Reina Mercedes S/N. 41012 Sevilla, Espa˜ na {rruiz,riquelme,aguilar}@lsi.us.es Abstract. The attribute selection techniques for supervised learning, used in the preprocessing phase to emphasize the most relevant attributes, allow making models of classification simpler and easy to understand. The algorithm has some interesting characteristics: lower computational cost (O(m n log n) m attributes and n examples in the data set) with respect to other typical algorithms due to the absence of distance and statistical calculations; its applicability to any labelled data set, that is to say, it can contain continuous and discrete variables, with no need for transformation. In order to test the relevance of the new feature selection algorithm, we compare the results induced by several classifiers before and after applying the feature selection algorithms.

1

Introduction

It is advisable to apply to the database preprocessing techniques to reduce the number of attributes or the number of examples in such a way as to decrease the computational time cost. These preprocessing techniques are fundamentally oriented to either of the next goals: feature selection (eliminating non-relevant attributes) and editing (reduction of the number of examples by eliminating some of them or calculating prototypes [1]). Our algorithm belongs to the first group. Feature selection methods can be grouped into two categories from the point of view of a method’s output. One category is about ranking feature according to same evaluation criterion; the other is about choosing a minimum set of features that satisfies an evaluation criterion. In this paper we present a new feature ranking algorithm by means of Projections and the hypothesis on which the heuristic is based is: ”place the best attributes with the smallest number of label changes (NLC)”.

2

Feature Evaluation

2.1

Description

To describe the algorithm we will use the well-known data set IRIS, because of the easy interpretation of their two-dimensional projections.


This work has been supported by the Spanish Research Agency CICYT under grant TIC2001-1143-C03-02.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 325–331, 2003. c Springer-Verlag Berlin Heidelberg 2003


Roberto Ruiz et al.

5

5

4,5

4,5

4

4

3,5

3,5

sepalwidth

sepalwidth

326

3 2,5 2 1,5

3 2,5 2 1,5

1

1

0,5

0,5 0

0 0

2

(a)

4 sepallength

6

8

10

0

0,5

(b)

1

1,5

2

2,5

3

petalwidth

Fig. 1. Representation of Attributes (a) Sepalwidth-Sepallength and (b) Sepalwidth-Petalwidth

Three projections of IRIS have been made in two-dimensional graphs. In Figure 1(a) it is possible to observe that if the projection of the examples is made on the abscissas or ordinate axis we can not obtain intervals where any class is a majority, only can be seen the intervals [4.3,4.8] of Sepallength for the Setosa class or [7.1,8.0] for Virginica. In Figure 1(b) for the Sepalwidth parameter in the ordinate axis clear intervals are not appraised either. Nevertheless, for the Petalwidth attribute is possible to appreciate some intervals where the class is unique: [0,0.6] for Setosa, [1.0,1.3] for Versicolor and [1.8,2.5] for Virginica. SOAP is based on this principle: to count the label changes, produced when crossing the projections of each example in each dimension. If the attributes are in ascending order according to the NLC, we will have a list that defines the priority of selection, from greater to smaller importance. Finally, to choose the more advisable number of features, we define a reduction factor, RF, in order to take the subset from attributes formed by the first of the aforementioned list. Before formally exposing the algorithm, we will explain with more details the main idea. We considered the situation depicted in Figure 1(b): the projection of the examples on the abscissas axis produces an ordered sequence of intervals (some of then can be a single point) which have assigned a single label or a set of them: [0,0.6] Se, [1.0,1.3] Ve, [1.4,1.4] Ve-Vi, [1.5,1.5] Ve-Vi, [1.6,1.6] Ve-Vi, [1.7,1.7] Ve-Vi, [1.8,1.8] Ve-Vi, [1.9,2.5] Vi. If we apply the same idea with the projection on the ordinate axis, we calculate the partitions of the ordered sequences: Ve, R, R, Ve, R, R, R, R, R, R, R, R, R, R, Se, R, Se, R, Se, where R is a combination of two or three labels. We can observe that we obtain almost one subsequence of the same value with different classes for each value from the ordered projection. That is to say, projections on the ordinate axis provide much less information that on the abscissas axis. In the intervals with multiple labels we will consider the worst case, that being the maximum number of label changes possible for a same value. The number of label changes obtained by the algorithm in the projection of each dimension is: Petalwidth 16, Petallength 19, Sepallenth 87 and Sepalwidth 120. In this way, we can achieve a ranking with the best attributes from the point

Fast Feature Ranking Algorithm

327

Table 1. Main Algorithm Input: E training (N examples, M attributes) Output: E reduced (N examples, K attributes) for each attribute Ai ∈ 1..M QuickSort(E,i) NLCi ← NumberChanges(E,i) NLC Attribute Ranking Select the k first

Table 2. NumberChanges function Input: E training (N examples, M attributes), i Output: number of label changes for each example ej ∈ E with j in 1..N if att(u[j],i) ∈ subsequence of the same value changes = changes + ChangesSameValue() else if lab(u[j]) <> lastLabel) changes = changes + 1 return(changes)

of view of the classification. This result agrees with what is common knowledge in data mining, which states that the width and length of petals are more important than those related to sepals. 2.2

Algorithm

The algorithm is very simple and fast, see Table 1. It has the capacity to operate with continuous and discrete variables as well as with databases which have two classes or multiple classes. In the ascending-order-task for each attribute, the QuickSort [5] algorithm is used. This algorithm is O(n log n), on average. Once ordered by an attribute, we can count the label changes throughout the ordered projected sequence. NumberChanges in Table 2, considers whether we deal with different values from an attribute, or with a subsequence of the same value (this situation can be originated in continuous and discrete variables). In the first case, it compares the present label with that of the following value. Whereas in the second case, where the subsequence is of the same value, it counts as many label changes as are possible (function ChangesSameValue). After applying QuickSort, we might have repeated values with the same or different class. For this reason, the algorithm firstly sorts by value and, in

328

Roberto Ruiz et al.

case of equality, it will look for the worst of the all possible cases (function ChangesSameValue). We could find the situation as depicted in Figure 2(a). The examples sharing the same value for an attribute are ordered by class. The label changes obtained are two. The next execution of the algorithm may find another situation, with a different number of label changes. The solution to this problem consists of finding the worst case. The heuristic is applied to obtain the maximum number of label changes within the interval containing repeated values. In this way, the ChangesSameValue method would produce the output shown in Figure 2(b), seven changes. This can be obtained with low cost. It can be deduced counting the class’ elements. ChangesSameValue stores the relative frequency for each class within the interval. It is possible to be affirm that: if rf i > (nelem/2) them (nelem − rf i) ∗ 2

else nelem − 1

(1)

rfi: relative frequency for each class, with i in {1,. . . ,k} classes. nelem: number of elements within the interval. In Figure 2(a) we can observe a subsequence of the same value with eight elements: three elements are class A, four class B and one C. Applying formula 2 there is no relative frequency greater than half of the elements. Then, the maximum number of label changes is nelem-1, seven. In Figure 2(b) we verify it. Ranking algorithms produce a ranked list, according to the evaluation criterion applied. The methods need an external parameter to take the subset from attributes formed by the first features of the aforementioned list. This parameter produces different results with different data sets. Therefore, in order to establish the number of attributes in each case, we put the range of value of the ranked lists between [0,1], i.e. the punctuation of the first attribute of the list will be 1, and the last attribute 0. Then, we select attributes over the parameter named Reduction Factor (RF). We do not realize an especial analyzed on each data set.

3

Experiments

In order to compare the effectiveness of SOAP as a feature selector for common machine learning algorithms, experiments were performed using sixteen

3

3

3

3

3

3

3

3

values

A

A

A

B

B

B

B

C

classes changes

3

3

3

3

3

3

3

3

values

B

A B

A

B

A B

C

classes changes

Fig. 2. Subsequence of the same value (a) two changes (b) seven changes

Fast Feature Ranking Algorithm

329

Table 3. Data sets, number of selected features, the percentage of the original features retained and time in milliseconds DATA Data Set Inst. Atts N◦ Cl. autos 205 25 7 breast-c 286 9 2 breast-w 699 9 2 diabetes 768 8 2 glass2 163 9 2 heart-c 303 13 5 heart-st 270 13 2 hepatit. 155 19 2 horse-c. 368 27 2 hypothy. 3772 29 4 iris 150 4 3 labor 57 16 2 lymph 148 18 4 sick 3772 29 2 sonar 208 60 2 vote 435 16 2 Average

SOAP Atts ( %)t-ms 2.9 (11.8) 15 1.5 (16.7) 4 5.2 (57.6) 6 2.8 (34.9) 6 3.2 (35.7) 2 6.3 (48.2) 6 5.4 (41.8) 4 2.6 (13.6) 4 2.3 ( 8.6) 16 1.7 ( 5.7) 180 2.0 (50.0) 3 4.3 (27.0) 1 1.8 ( 9.9) 3 1.0 ( 3.4) 120 3.0 ( 5.0) 21 1.6 (10.0) 9 (23.7)

Atts 5.3 4.1 9.0 3.1 4.0 6.4 6.3 8.7 2.0 1.0 1.9 3.3 8.9 1.0 17.8 1.0

CFS ( %)t-ms (21.3) 50 (45.9) 6 (99.7) 35 (38.9) 39 (43.9) 9 (49.1) 10 (48.2) 12 (45.6) 9 ( 7.4) 43 ( 3.4) 281 (48.3) 3 (20.8) 3 (49.2) 7 ( 3.4) 252 (29.7) 90 ( 6.3) 4 (35.1)

Atts 10.9 3.7 8.1 0.0 0.3 6.9 6.3 13.3 2.3 5.2 4.0 8.8 11.8 7.1 3.9 15.5

RLF ( %) t-ms ( 43.7) 403 ( 41.6) 174 ( 89.4) 1670 ( 0.0) 1779 ( 3.6) 96 ( 53.4) 368 ( 48.2) 365 ( 70.0) 135 ( 8.6) 941 ( 18.0)94991 (100.0) 44 ( 55.3) 21 ( 65.8) 109 ( 24.5)93539 ( 6.5) 920 ( 96.9) 651 ( 45.3)

standard data sets from the UCI repository [4]. The data sets and their characteristics are summarized in Table 3. The percentage of correct classification with C4.5, averaged over ten ten-fold cross-validation runs, were calculated for each algorithm-data set combination before and after feature selection by SOAP (RF 0.75), CFS and ReliefF (threshold 0.05). For each train-test split, the dimensionality was reduced by each feature selector before being passed to the learning algorithms. The same fold were used for each feature selector-learning scheme combination. To perform the experiment with CFS and ReliefF we used the Weka1 (Waikato Environment for Knowledge Analysis) implementation. Table 3 shows the average number of features selected and the percentage of the original features retained. SOAP is a specially selective algorithm compared with CFS and RLF. If SOAP and CFS are compared, only in one data set (labor) is the number of characteristics significantly greater than those selected by CFS. In six data sets there are no significant differences, and in nine, the number of features is significantly smaller than CFS. Compare to RLF, only in glass2 and diabetes, SOAP obtains more parameters in the reduction process (threshold 0.05 is not sufficient). It can be seen that SOAP retained 23,7% of the attributes on average. Table 4 shows the results for attribute selection with C4.5 and compares the size (number of nodes) of the trees produced by each attribute selection scheme 1

http://www.cs.waikato.ac.nz/˜ml

330

Roberto Ruiz et al.

Table 4. Result of attribute selection with C4.5. Accuracy and size of trees. ◦, • Statistically significant improvement or degradation (p=0.05) DATA Set Ac. autos 82.54 breast-c 74.37 breast-w 95.01 diabetes 74.64 glass2 78.71 heart-c 76.83 heart-stat 78.11 hepatitis 78.97 horse-c.OR. 66.30 hypothyroid 99.54 iris 94.27 labor 80.70 lymph 77.36 sick 98.66 sonar 74.28 vote 96.53 Average 82.93

Size 63.32 12.34 24.96 42.06 24.00 43.87 34.58 17.06 1.00 27.84 8.18 6.93 28.05 49.02 27.98 10.64 26.36

SOAP Ac. Size 73.37 • 45.84 70.24 • 6.61 94.64 21.28 74.14 7.78 78.96 14.88 77.06 34.02 80.67 ◦ 19.50 80.19 5.62 66.30 1.00 95.02 • 4.30 94.40 8.12 78.25 3.76 72.84 • 7.34 93.88 • 1.00 70.05 • 7.00 95.63 • 3.00 80.98 11.94

◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Ac. 74.54 72.90 95.02 74.36 79.82 77.16 80.63 81.68 66.30 96.64 94.13 80.35 75.95 96.32 74.38 95.63 82.24

CFS Size • 55.66 18.94 24.68 14.68 14.06 29.35 ◦ 23.84 ◦ 8.68 1.00 • 5.90 7.98 6.44 20.32 • 5.00 28.18 • 3.00 16.73

◦ • ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦

Ac. 74.15 70.42 95.02 65.10 53.50 79.60 82.33 80.45 66.28 93.52 94.40 80.00 74.66 93.88 70.19 96.53 79.38

RLF Size • 85.74 • 11.31 24.68 • 1.00 • 1.70 ◦ 28.72 ◦ 14.78 11.26 1.36 • 12.52 8.16 5.88 24.10 • 1.00 • 9.74 10.64 15.79

• ◦ ◦ ◦ ◦ ◦ • ◦ ◦ ◦ ◦ ◦

against the size of the trees produced by C4.5 with no attribute selection. Smaller trees are preferred as they are easier to interpret, but accuracy is generally degraded. The table shows how often each method performs significantly better (denoted by ◦) or worse (denoted by •) than when performing no feature selection (column 2 and 3). Throughout we speak of results being significantly different if the difference is statistically at the 5% level according to a paired two-sided t test. Each pair of points consisting of the estimates obtained in one of the ten, ten-fold cross-validation runs, for before and after feature selection. For SOAP, feature selection degrades performance on seven data sets, improves on one and it is equal on eight. The reason for why the algorithm is not as accurate is the number of attribute selected, less than three feature. Five of these seven data sets obtain a percentage less than 10% of the original features. The results are similar to ReliefF and a little worse than those provided by CFS. Analyzing the data sets in which SOAP lost to CFS, we can observe breast-c, lymph and sonar, where the number of feature selected by SOAP is 25% of CFS (breast-c 4,1 to 1,5 with SOAP, lymph 8,9-1,8 and sonar 17,8-3). Nevertheless the accuracy reduction is small: breast-c 72,9 (CFS) to 70,24 with SOAP, lymph 75,95-72,84 and sonar 74,38-70,05. It is interesting to compare the speed of the attribute selection techniques. We measured the time taken in milliseconds to select the final subset of attributes. SOAP is an algorithm with a very short computation time. The results shown

Fast Feature Ranking Algorithm

331

in Table 3 confirm the expectations. SOAP takes 400 milliseconds2 in reducing 16 data sets whereas CFS takes 853 milliseconds and RLF more than 3 minutes. In general, SOAP is faster than the other methods and it is independent of the classes number. Also it is possible to be observed that ReliefF is affected very negatively by the number of instances in the data set, it can be seen in ”hypothyroid” and ”sick”. Even though these two data sets were eliminated, SOAP is more than 3 times faster than CFS, and more than 75 times than ReliefF.

4

Conclusions

In this paper we present a deterministic attribute selection algorithm. It is a very efficient and simple method used in the preprocessing phase. A considerable reduction of the number of attributes is produced in comparison to other techniques. It does not need distance nor statistical calculations, which could be very costly in time (correlation, gain of information, etc.). The computational cost is lower than other methods O(m n log n).

References [1] Aguilar-Ruiz, Jes´ us S., Riquelme, Jos´e C. and Toro, Miguel. Data Set Editing by Ordered Projection. Intelligent Data Analysis Journal. Vol. 5, n◦ 5, pp. 1-13, IOS Press (2001). 325 [2] Almuallim, H. and Dietterich, T. G. Learning boolean concepts in the presence of many irrelevant features. Artificial Intelligence, 69(1-2):279-305 (1994). [3] Blake, C. and Merz, E. K. UCI Repository of machine learning databases (1998). [4] Hall M. A. Correlation-based feature selection for machine learning. PhD thesis, Department of Computer Science, University of Waikato, Hamilton, New Zealand (1998). [5] Hoare, C. A. R. QuickSort. Computer Journal, 5(1):10-15 (1962). 327 [6] Kira, K. and Rendell, L. A practical approach to feature selection. In Proceedings of the Ninth International Conference on Machine Learning. pp. 249-256, Morgan Kaufmann (1992). [7] Kohavi, R. and John, G. H. Wrappers for feature subset selection. Artificial Intelligence, 97, 273-324 (1997). [8] Kononenko, I. Estimating attibutes: Analisys and extensions of relief. In Proceedings of the Seventh European Conference on Machine Learning. pp. 171-182, Springer-Verlag (1994). [9] Quinlan, J. C4.5: Programs for machine learning. Morgan Kaufmann (1993). ˇ [10] Robnik-Sikonja, M. And Kononenko, I. An adaption of relief for attribute estimation in regression. In Proceedings of the Fourteenth International Conference on Machine Learning. pp. 296-304, Morgan Kaufmann (1997). [11] Setiono, R., and Liu, H. A probabilistic approach to feature selection-a filter solution. In Proceedings of International Conference on Machine Learning, 319327 (1996). 2

This is a rough measure. Obtaining true cpu time from within a Java program is quite difficult.

Visual Clustering with Artificial Ants Colonies Nicolas Labroche, Nicolas Monmarch´e, and Gilles Venturini Laboratoire d’Informatique de l’Universit´e de Tours, ´ Ecole Polytechnique de l’Universit´e de Tours-D´epartement Informatique, 64, avenue Jean Portalis 37200 Tours, France {labroche,monmarche,venturini}@univ-tours.fr http://www.antsearch.univ-tours.fr/

Abstract. In this paper, we propose a new model of the chemical recognition system of ants to solve the unsupervised clustering problem. The colonial closure mechanism allows ants to discriminate between nestmates and intruders by the mean of a colonial odour that is shared by every nestmate. In our model we associate each object of the data set to the odour of an artificial ant. Each odour is defined as a real vector with two components, that can be represented in a 2D-space of odours. Our method simulates meetings between ants according to pre-established behavioural rules, to ensure the convergence of similar odours (i.e. similar objects) in the same portion of the 2D-space. This provides the expected partition of the objects. We test our method against other well-known clustering method and show that it can perform well. Furthermore, our approach can handle every type of data (from numerical to symbolic attibutes, since there exists an adapted similarity measure) and allows one to visualize the dynamic creation of the nests. We plan to use this algorithm as a basis for a more sophisticated interactive clustering tool.

1

Introduction

Computer scientists have successfully used bio mimetic approaches to imagine performing heuristics. For example, Dorigo et al. modelled the pheromone trails of real ants to solve optimisation problems in the well-known Ant Colony Optimization heuristic (ACO [1, 2]). Genetic algorithms have been applied to optimisation problems and clustering problems [3, 4]. The collective behaviours of ants have also inspired researchers to solve the graph partitioning problem with co-evolving species [5] and the unsupervised clustering problem [6, 7]. In these studies, researchers model real ants abilities to sort their brood. Artificial ants move on a 2D discrete grid on which objects are randomly placed. Ants may pick up or carry one or more objects and may drop them according to given probabilities. After a given time, these artificial ants build groups of similar objects that provide the final partition. In [8], the authors present AntClust, an ant-based clustering algorithm that relies on a chemical recognition model of ants. In this work, each artificial ant is associated to an object by the mean of its cuticular odour. This odour is representative of the belonging nest of each ant and coded with a single value. This value may change according to behavioural V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 332–339, 2003. c Springer-Verlag Berlin Heidelberg 2003


Visual Clustering with Artificial Ants Colonies

333

rules and meetings with other ants, until each ant finds the nest that best fits the object it represents. In this paper, we propose an alternative model of this chemical recognition system. Our goal is to see if the previous discrete odour model implies limitations to the method. Furthermore, we want to be able to visualize dynamically the building of the nests. In this case, an expert could interact with the system, while the results are displayed. Our paper is organised as follows: the section 2 briefly describes the main principles of the chemical recognition system of ants and introduces the new model of an artificial ant (the parameters and the behavioural rules for the meetings). The section 3 presents the Visual AntClust algorithm and discuss its parameters settings. Then, the section 4 describes the clustering method that we use to evaluate Visual AntClust and the data sets used as benchmarks. Finally, the section 5 concludes and introduces futures developments of our work.

2 2.1

The Model The Chemical Recognition System of Real Ants

As many other social insects (such as termites, wasps and bees), ants have developed a mechanism of colonial closure that allows them to discriminate between nestmates and intruders. Each ant generates its own cuticular odour, called the label. This label is partly defined by the genome of the ant and partly by chemical substances extracted from its environment (mainly nest materials and food). According to the ”Gestalt odour theory” [9, 10], the continuous chemical exchanges between nestmates lead to the establishment of a colonial odour that is recognised by all the members of the colony. 2.2

Parameters

The main idea of Visual AntClust is to associate one object of the data set to the genome of one artificial ant. The genome allows ants to generate their own odour and is used to learn or update their recognition template. This template is used in every meeting to decide if both ants should accept each others or not. For one ant a, we define the following parameters. The label labela is a vector in the 2D-space of odours. Each of its components reflects the chemical concentration of one compound in the label of the artificial ant and is defined in [0, 1]. As no assumption can be done concerning the final partition, the labels vectors are randomly initialised for every ant. The genome genomea corresponds to an object of the data set and can not evolve during the algorithm. The recognition template ta , is used by each ant during meetings to appreciate the similarities between genomes (i.e. between 2 objects). ta is learned during a fixed number of iterations (set experimentally to 50). During this period, the ant a encounters other randomly chosen ants and estimates the similarity

334

Nicolas Labroche et al.

between their genomes. At the end, the ant a uses the mean and maximal similarities values Sim(a, ·) and max (Sim(a, ·)) observed during this period to set its template threshold ta as follows: ta =

max (Sim(a, ·)) + Sim(a, ·) 2

(1)

The satisfaction estimator sa . At the beginning of the algorithm, it is set to 0 and is increased each time the artificial ant a meets and accepts another ant in a local portion of the 2D-space of odour. sa estimates for each ant if it is well placed in the 2D-odour space. When it increases, the ant limits the area of the 2D-space in which it accepts other ants to help the algorithm to converge. 2.3

Meeting Algorithm

We detail hereafter the meeting process between two ants i and j. This mechanism allows ants to exchange and mutually reinforce their labels if they are close enough and if their genomes are similar enough. Meeting(Ant i, Ant j) (1) (2) (3) (4) (5) (6) (7) (8)

D ← Compute the Euclidian distance between Labeli and Labelj if D ≤ (1 − max(si , sj )) then if there is acceptance between ants i and j then Increase si and sj because ants i and j are well-placed Compute Ri and Rj , the odour exchange rates between ant i and ant j Update Labeli(j) according to Ri(j) and si(j) endif endif

The algorithm determines if both ants labels are similar enough to allow the meeting. It is known that real ants prefer to reinforce their labels with nestmates that have similar odours. We consider also that an ant should have a need to exchange chemical substances to meet another ant. We use si and sj , the satisfaction estimators, to evaluate this need (line 2). An artificial ant i, that is well placed in the odour-space (si is close to 1), already massively accepts its neighbours. That means that the object corresponding to ant i is already assigned to a good cluster. In the opposite case, if si is close to 0, either ant i is alone in a portion of the odour-space, or it is in a region where the neighbours are not similar enough (i.e. the object is misclassified). In this case, the ant needs to encounter other ants to increase its chances to belong to a good nest. If an ant corresponding to the previous criteria is found, there must be acceptance between ants (line 3). In this case, the similarity between genomes of two ants i and j must be greater or equal than both ants’ template thresholds ti and tj as shown in the following equation 2.
 (2) Acceptance(Anti , Antj ) ← Sim(Anti , Antj ) ≥ max (tAnti , tAntj )

Visual Clustering with Artificial Ants Colonies

335

Finally, if both ants are equally satisfied, they mutually reinforce their labels according to an impregnation rate R computed as follows for one ant i that accepts an other ant j. (3) Ri ← Sim(i, j) − ti This ensures the convergence of the algorithm by preventing ants that are wellplaced to update their label. Thus, only the less satisfied ants can change their label consequently to a meeting, which improves the efficiency of the algorithm.

3

Visual AntClust Visual AntClust(data set with N objects, Niter ) (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11)

Initialise N ants according to parameters described in the section 2.2 while (Niter iterations are not reached) do Draw N ants in the 2D odor space for c ← 1 to N do Meeting(i,j) where i and j are randomly chosen ants endfor endwhile Compute Dmax , the maximal distance in odor space that allows an ant to find a nestmate Group in the same nest all the ants within a local perimeter of value Dmax Delete nests n that are too small (size(n) < 0.5 × size∀ nests ) Re-assign the ants that have no more nest to the nest of the most similar ant

Our algorithm depends on two main parameters that are namely, the number of global iterations Niter and the maximal distance under which a nestmate may be found Dmax . The first parameter Niter has been set experimentally to 2.500 iterations. In fact, some data sets need more or less iterations, but we do not develop, for the time being, a heuristic to determine automatically the number of iterations. To the last extent, our goal is to integrate this algorithm in an interactive framework, in which an expert will have the possibility to start and stop the visual clustering process and manipulate the groups. The second parameter Dmax is computed for each ant a according to an estimation of the mean D(i, ·) and minimal min(D(i, ·)) distances (here an Euclidean distance measure) between its label and the labels of the other ants as follows: (4) Dmax ← (1 − β)D(i, ·) + β min(D(i, ·)) β has been set experimentally to 0.95 to get the best results.

4 4.1

Experiments and Results Experiments

We apply a traditional k-Means approach because of its time complexity Θ(N ), N being the number of objects to cluster. The algorithm inputs an initial

336

Nicolas Labroche et al.

Table 1. Main characteristics of the data sets Art1 Art2 Art3 Art4 Art5 Art6 Iris Glass P ima Soybean T hyroid N 400 1000 1100 200 900 400 150 214 798 47 215 M 2 2 2 2 2 8 4 9 8 35 5 K 4 2 4 2 9 4 3 7 2 4 3

partition of k clusters. This partition is then refined at each iteration by the assignment of an object to its nearer cluster center. The cluster centers are updated after each object assignment until the algorithm converges. The method stops when the intra class inertia becomes stable. In our experiment, we randomly generate the initial partition with the expected number of clusters to get the best results with this approach. We also compare our new approach to AntClust, an other ant-based clustering algorithm that also relies on the chemical recognition system of ants and that has been described in [8]. In this method the label of each ant is coded by a single value that reflects the nest the ant belongs to. The algorithm is able to find good partitions without any assumption concerning the number of expected clusters and can treat every type of data. We use several numerical data sets to evaluate the contribution of our method. There are real data sets such as Iris, P ima, Soybean, T hyroid and Glass data sets and also artificial data sets called Art1,2,3,4,5,6 . They are generated according to distinct laws (Gaussian and uniform) and with distinct difficulties. The number of objects N , the number of attributes M and the number of clusters K of each data sets are described in the table 1. We use a clustering error measure that evaluates the differences between the final partition output by each method and the expected one, by comparing each pair of objects and by verifying each time if they are clustered similarly or not in both partitions. 4.2

Results

The table 2 shows that, even with its simple heuristic to create the partition from the 2D-space of odours, Visual AntClust is able to perform well. For some data sets like Art1 , Art4 and Art6 , Iris and Glass, it slightly outperforms AntClust though it shows generally a greater variability in its performances. Furthermore, Visual AntClust is even better than the k-Means approach in two cases for Art6 and Soybean, where the clustering error is equal to 0. We notice also that Visual AntClust does not manage to estimate the number of expected clusters for P ima and T hyroid. This may mean that the parameters of the algorithm are not well suited for these data sets or that the heuristic that finds the nests is not accurate enough in these cases to find a good partition. It is very important to notice that Visual AntClust is slower than AntClust and of course k-Means because at each iteration, it has to draw the on-going partition of the data set. The figure 1

Visual Clustering with Artificial Ants Colonies

Beginning of the algorithm: 0 iteration

337

Convergence: about 100 iterations

Clusters apparition: about 500 iterations Clusters definition: about 1000 iterations

Fig. 1. Four steps of Visual AntClust in the 2D-space of odours to build the final partition for Art1

represents four successive steps of Visual AntClust as it dynamically builds the final partition.

5

Conclusion and Perspectives

In this paper, we introduce a new ant based clustering algorithm named Visual AntClust. It associates one object of the data set to the genome of one artificial ant and simulates meetings between them to dynamically build the expected partition. The method is compared to the k-Means approach and AntClust, an other ant-based clustering method over artificial and real data sets. We show that Visual AntClust performs well generally and sometimes very well. Furthermore, it can treat every type of data unlike k-Means that is limited to numerical data and allow one to visualize the partition as it is constructed. In the near future, we will try to develop a method to automate the parameters’ setting. We will also use Visual AntClust as a base for an interactive clustering and data visualization tool.

338

Nicolas Labroche et al.

Table 2. Mean number of clusters (# clusters) and mean clustering error (ClusteringError) and their standard deviation ([std]) for each data set and each method computed over 50 runs K-Means Data sets mean [std] Art1 3.98 [0.14] 2.00 [0.00] Art2 3.84 [0.37] Art3 2.00 [0.00] Art4 8.10 [0.75] Art5 4.00 [0.00] Art6 Iris 2.96 [0.20] Glass 6.88 [0.32] P ima 2.00 [0.00] Soybean 3.96 [0.20] T hyroid 3.00 [0.00]

# clusters Clustering Error AntClust VAntClust K-Means AntClust VAntClust mean [std] mean [std] mean [std] mean [std] mean [std] 4.70 [0.95] 4.66 [1.04] 0.11 [0.00] 0.22 [0.03] 0.17 [0.07] 2.30 [0.51] 3.76 [2.36] 0.04 [0.00] 0.07 [0.02] 0.14 [0.06] 2.72 [0.88] 3.54 [2.11] 0.22 [0.02] 0.15 [0.02] 0.23 [0.07] 4.18 [0.83] 2.16 [0.37] 0.00 [0.00] 0.23 [0.05] 0.03 [0.04] 6.74 [1.66] 4.5 [1.69] 0.09 [0.02] 0.26 [0.02] 0.29 [0.12] 4.06 [0.24] 3.98 [0.14] 0.01 [0.04] 0.05 [0.01] 0.00 [0.02] 2.82 [0.75] 2.28 [0.49] 0.14 [0.03] 0.22 [0.01] 0.19 [0.06] 5.90 [1.23] 6.2 [1.16] 0.32 [0.01] 0.36 [0.02] 0.35 [0.03] 10.66 [2.33] 19.38 [7.68] 0.44 [0.00] 0.46 [0.01] 0.49 [0.01] 4.16 [0.55] 4.00 [0.00] 0.09 [0.08] 0.07 [0.04] 0.00 [0.00] 4.62 [0.90] 11.66 [3.44] 0.18 [0.00] 0.16 [0.03] 0.36 [0.07]

References [1] A. Colorni, M. Dorigo, and V. Maniezzo, “Distributed optimization by ant colonies,” in Proceedings of the First European Conference on Artificial Life (F. Varela and P. Bourgine, eds.), pp. 134–142, MIT Press, Cambridge, 1991. 332 [2] E. Bonabeau, M. Dorigo, and G. Theraulaz, From natural to artificial swarm intelligence. New York: Oxford University Press, 1999. 332 [3] Y. Chiou and L. W. Lan, “Genetic clustering algorithms,” European journal of Operational Research, no. 135, pp. 413–427, 2001. 332 [4] L. Y. Tseng and S. B. Yang, “A genetic approach to the automatic clustering problem,” Pattern Recognition, no. 34, pp. 415–424, 2001. 332 [5] P. Kuntz and D. Snyers, “Emergent colonization and graph partitioning,” in Cliff et al. [11], pp. 494–500. 332 [6] E. Lumer and B. Faieta, “Diversity and adaptation in populations of clustering ants,” in Cliff et al. [11], pp. 501–508. 332 [7] N. Monmarch´e, M. Slimane, and G. Venturini, “On improving clustering in numerical databases with artificial ants,” in Lecture Notes in Artificial Intelligence (D. Floreano, J. Nicoud, and F. Mondala, eds.), (Swiss Federal Institute of Technology, Lausanne, Switzerland), pp. 626–635, 1999. 332 [8] N. Labroche, N. Monmarch´e, and G. Venturini, “A new clustering algorithm based on the chemical recognition system of ants,” in Proc. of 15th European Conference on Artificial Intelligence (ECAI 2002), Lyon FRANCE, pp. 345–349, 2002. 332, 336 [9] N. Carlin and B. H¨ olldobler, “The kin recognition system of carpenter ants(camponotus spp.). i. hierarchical cues in small colonies,” Behav Ecol Sociobiol, vol. 19, pp. 123–134, 1986. 333 [10] B. H¨ olldobler and E. Wilson, The Ants. Springer Verlag, Berlin, Germany, 1990. 333

Visual Clustering with Artificial Ants Colonies

339

[11] D. Cliff, P. Husbands, J. Meyer, and S. W., eds., Third International Conference on Simulation of Adaptive Behavior: From Animals to Animats 3, MIT Press, Cambridge, Massachusetts, 1994. 338

Maximizing Benefit of Classifications Using Feature Intervals ˙ Nazlı Ikizler and H. Altay G¨ uvenir Bilkent University Department of Computer Engineering 06533, Ankara Turkey {inazli,guvenir}@cs.bilkent.edu.tr

Abstract. There is a great need for classification methods that can properly handle asymmetric cost and benefit constraints of classifications. In this study, we aim to emphasize the importance of classification benefits by means of a new classification algorithm, Benefit-Maximizing classifier with Feature Intervals (BMFI) that uses feature projection based knowledge representation. Empirical results show that BMFI has promising performance compared to recent cost-sensitive algorithms in terms of the benefit gained.

1

Introduction

Classical machine learning applications try to reduce the quantity of the errors and usually ignore the quality of errors. However, in real-world applications, the nature of the error is very crucial. Further, the benefit of correct classifications may not be the same for all cases. Cost-sensitive classification research addresses this imperfection and evaluates the effects of predictions rather than simply measuring the predictive accuracy. By incorporating cost(and benefit) knowledge to the process of classification, the effectiveness of the algorithms in real-world situations can be evaluated more rationally. In this study, we concentrate on costs of misclassifications and try to minimize those costs, by maximizing the total benefit gained during the process of classification. Within this framework, we propose a new cost-sensitive classification technique, called Benefit-Maximizing classifier with Feature Intervals (BMFI for short),that uses the predictive power of feature projection method previously proposed in [6]. In BMFI, voting procedure has been changed to impose the cost-sensitivity property. Generalization techniques are implemented to avoid overfitting and to eliminate redundancy. BMFI has been tested over several benchmark datasets and a number of real-world datasets that we have compiled. The rest of the paper is organized as follows: In Section 2, benefit maximization problem is addressed. Section 3 gives the algorithmic descriptions of BMFI algorithm along with the details of feature intervals concept, voting method and generalizations. Experimental evaluation of BMFI is presented in Section 4. Finally, Section 5 reviews the results and presents future research directions on the subject. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 339–345, 2003. c Springer-Verlag Berlin Heidelberg 2003


340

2

˙ Nazlı Ikizler and H. Altay G¨ uvenir

Benefit Maximization Problem

Recent research in machine learning has used the terminology of costs when dealing with misclassifications. However, those studies mostly lack the information that correct classifications may have different interpretations. Besides implying no cost, accurate labeling of instances may entail indisputable gains. Elkan points out the importance of these gains [3]. He states that doing accounting in terms of benefits is commonly preferable because there is a natural baseline from which all benefits can be measured, and thus, it is much easier to avoid mistakes. Benefit concept is more appropriate to real world situations, since net flow of gain is more accurately denoted by benefits attained. If a prediction is profitable from the decision agent’s point of view, its benefit is said to be positive. Otherwise, it is negative, which is the same as cost of wrong decision. To incorporate this natural knowledge of benefits to cost-sensitive learning, we have used benefit matrices. B=[bij ] is a n × m benefit matrix of domain D if n equals to the number of prediction labels, m equals to the number of possible class labels in D and bij ’s are such that
≥ 0 if i = j bij = . (1) < bii otherwise Here, bij represents the benefit of classifying an instance of true class j as class i. The structure of the benefit matrix is similar to that of the cost matrix, with the extension that entries can either have positive or negative values. In addition, diagonal elements should be non-negative values, ensuring that correct classifications can never have negative benefits. Given a benefit matrix B, the optimal prediction for an instance x is the class i that maximizes expected benefit (EB ), that is  P (j|x) × bij . (2) EB(x, i) = j

where P (j|x) is the probability that x has true class j. The total expected benefit of the classifier model M over the whole test data is    EBM = arg max EB(x, i) = arg max P (j|x) × bij . (3) x

i∈C

x

i∈C

j

where C is the set of possible class labels in the domain.

3

Benefit Maximization with Feature Intervals

As shown in [6], feature projections based classification is a fast and accurate method, and the rules it learns are easy for humans to verify. For this reason, we have chosen to extend its predictive power to involve benefit knowledge. In a particular classification problem, given the training dataset which consists of p features, an instance x can be thought as a point in a p-dimensional space with an associated class label xc . It is represented as a vector of nominal or

Maximizing Benefit of Classifications Using Feature Intervals

341

train(T rainingSet, Benef itM atrix) begin for each feature f sort(f , T rainingSet) i list ← make point intervals(f ,T rainingSet) for each interval i in i list votei (c) ← voting method (i,f ,Benef itM atrix) if f is linear i list ← generalize(i list,Benef itM atrix) end.

Fig. 1. Training phase of BMFI linear feature values together with its associated class, i.e., <x1 , x2 , .., xp , xc >. Here, xf represents the value of the f th feature of the instance x. If we consider each feature separately, and take x’s projection onto each feature dimension, then we can represent x by the combination of its feature projections. Training process of BMFI algorithm is given in Fig. 1. In the beginning, for each feature f , all training instances are sorted with respect to their value for f . This sort operation is identical to forming projections of the training instances for each feature f . A point interval is constructed for each projection. Initially, lower and upper bounds of the interval are equal to the f value of the corresponding training instance. If the f value of a training instance is unknown, it is simply ignored. If there are several point intervals with the same f value, they are combined into a single point interval by adding the class counts. At the end of point interval construction, vote for each class label is determined by using one of the two voting methods. The first one is the voting method of CFI algorithm [5], called VM1 in our context. VM1 can be formulated as follows: V M 1(c, I) =

Nc . classCount(c)

(4)

where Nc is the number of instances that belong to class c in interval I and classCount(c) is the total number of instances of class c in the entire training set. This voting method favors the prediction of minority class in proportion to its occurrence in the interval. The second voting method, called VM2, is basically founded on optimal prediction approximation given by (2) and makes direct use of the benefit matrix. VM2 casts votes to class c in interval I as  bck × P (k|I) . (5) V M 2(c, I) = k∈C

P (k|I) is the estimated probability that an instance falling to interval I will have the true class k, and is calculated as P (k|I) =

Nk . classCount(k)

(6)

342

˙ Nazlı Ikizler and H. Altay G¨ uvenir generalize(interval list) begin I ← first interval in interval list while I is not empty do I
← interval after I I” ← interval after I
if merge condition(I, I
, I”) is true merge I
(and/or) I” into I else I ← I
end.

Fig. 2. Generalization of intervals step in BMFI

After the initial assignment of votes, for linear features, intervals are generalized to form range intervals in order to eliminate redundancy and avoid overfitting. The generalization process is illustrated in Fig. 2. Here, merge condition() is a comparison function that evaluates relative properties of each interval and returns true if sufficient level of similarity between neighboring intervals is reached. Besides adding more prediction power to the algorithm, proper generalization reduces the number of intervals, and by this way, decreases the classification time. In this work, we have experimented with three interval generalization methods. The first one, called SF (same frequent) joins two consecutive intervals if the most frequently occurring class of both are the same. The second method, SB (same beneficial) joins two consecutive intervals if they have the same beneficial  class. A class c  is the beneficial class of an interval i iff for ∀j ∈ C and j = c, x∈i B(x, c) ≥ x∈i B(x, j) . If the beneficial classes of two consecutive intervals are the same, then it can be more profitable to unite them into a single interval. The third method, HC (high confidence) combines three consecutive intervals into a single one, when the middle interval has less confidence on its votes than the other two. The confidence of an interval is measured as the difference between votes of the most beneficial class and second beneficial class. Table 1. List of evaluated cost-sensitive algorithms Name

Description

MetaNB MetaJ48 C1NB C2NB C1J48 C2J48 C1VFI C2VFI

MetaCost on Naive Bayes MetaCost on J4.8 CostSensitiveClassifier with CostSensitiveClassifier with CostSensitiveClassifier with CostSensitiveClassifier with CostSensitiveClassifier with CostSensitiveClassifier with

reweighting on Naive Bayes direct minimization on Naive Bayes reweighting on J4.8 direct minimization on J4.8 reweighting on VFI direct minimization on VFI

Maximizing Benefit of Classifications Using Feature Intervals

343

classify(q) begin for each class c vc ← 0 for each feature f if qf is known I ← search interval(f, qf ) for each class c vc ← vc + interval vote(c, I) prediction ← argmaxc(vc ) end.

Fig. 3. Classification step in BMFI

The classification process of the BMFI algorithm is given in Fig. 3. The choice of voting method to be used depends on the characteristics of the domain. Based on our empirical results, we propose to use VM1 voting together with SF, SB and HC techniques when the correct classification of the minority class is more beneficial than the other classes. On the contrary, when the benefit matrix is not correlated with the distribution, VM2 can be employed together with SB and HC to boost up the benefit performance. Experimental results presented in Sect. 4 are achieved by using this general rule-of-thumb.

4

Experimental Results

For evaluation purposes, we have used benchmark datasets from UCI ML Repository [1]. These data sets do not have predefined benefit matrices, so we formed their benefit matrices in the following manner. In binary datasets, one class is assumed to be more important to predict correctly than the other by a constant benefit ratio, b. We have tested our algorithm by using five different b values that are 2, 5, 10, 20, 50. Note that when b is equal to 1, the problem reduces to the classical classification problem. Further, we have compiled four new datasets. Their benefit matrices have been defined by experts of each domain. For more information about the datasets and benefit matrices the reader is referred to [7]. We have compared BMFI with MetaCost [2] and CostSensitiveClassifier of Weka [4] on well-known base classifiers which are Naive Bayesian Classifier, C4.5 decision tree learner and VFI [6]. Table 1 lists these algorithms with their base classifiers (J4.8 is Weka’s implementation of C4.5 in Java). MetaCost is a wrapper algorithm that takes a base classifier and makes it sensitive to costs of classification [2]. It operates with a bagging logic beneath and learns multiple classifiers on multiple bootstrap replicates of the training set. MetaCost has become a benchmark for comparing cost-sensitive algorithms. In addition to MetaCost, we have compared our algorithm with two cost sensitive classifiers provided in Weka. The first method uses reweighting of training

344

˙ Nazlı Ikizler and H. Altay G¨ uvenir

Table 2. Comparative evaluation of BMFI with wrapper cost-sensitive algorithms. The entries are benefit per instance values. Best results are shown in bold domain

MetaNB MetaJ48 C1NB C2NB C1J48 C2J48 C1VFI C2VFI BMFI

breast-cancer pima-diabetes ionosphere liver disorders sonar bank-loans bankruptcy dermatology lesion

4.0 2.8 5.7 5.3 3.3 -0.8 7.8 7.5 8.7

4.0 3.0 6.1 5.2 4.5 -0.9 7.7 7.5 8.9

3.8 2.8 6.5 5.3 4.6 -0.4 7.5 7.2 7.8

4.0 2.7 6.0 5.4 4.0 -0.6 7.4 7.5 9.0

3.9 2.9 6.5 5.4 4.6 0.1 7.5 7.2 7.8

3.7 2.5 5.7 4.4 3.3 -0.5 7.3 7.3 7.7

3.7 -1.5 6.4 4.3 0.0 -1.2 7.7 6.9 6.4

2.8 2.8 6.1 5.3 4.0 -2.8 7.8 5.6 4.0

3.9 (VM1) 2.7 (VM1) 6.5 (VM2) 5.4 (VM2) 4.9 (VM2) -0.1 (VM1) 7.9 (VM1) 7.4 (VM2) 9.0 (VM1)

instances and the second method makes direct cost-minimization based on probability distributions [8]. We call these two classifiers C1 and C2, respectively. Experimental results are presented in Table 2. In this table, results of binary datasets are benefit per instance values for b=10. All results are recorded by using 10-fold cross validation. As the results demonstrate, BMFI algorithm is very successful in most of the domains and remarkably comparable to other algorithms in all of the domains. In ionosphere, liver, sonar, bankruptcy and lesion domains, BMFI attains the maximum benefit per instance value. In the remaining datasets its performance is very high and comparable to other algorithms. We have observed that benefit achieved is highly dependent on the nature of the domain, i.e., benefit matrix information, distribution of classes, etc, as expected. In addition, it is worthwhile to note that BMFI outperforms cost-sensitive versions of its base classifier VFI (C1VFI and C2VFI). This observation suggests that using benefit knowledge inside the algorithm itself is more effective than wrapping a meta-stage around to transform it into a cost-sensitive classifier. In binary datasets, we observed that the success of BMFI increases as the benefit ratio increases. This is an important highlight of BMFI and is mostly due to its high sensitivity to benefit of classifications. This aspect of BMFI has been illustrated with the results of pima-diabetes dataset given in Table 3.

Table 3. Benefit per instance values of pima-diabetes dataset with different benefit ratios. Best results are shown in bold b MetaNB MetaJ48 C1NB C2NB C1J48 C2J48 C1VFI C2VFI BMFI 2 5 10 20 50

0.5 1.2 2.8 5.8 16.6

0.6 1.2 2.8 5.8 16.2

0.7 1.5 3.0 6.2 16.2

0.6 1.3 2.7 6.1 16.6

0.6 1.2 2.9 6.1 16.3

0.6 1.2 2.5 5.6 14.7

0.0 -0.5 -1.5 -3.3 -9.0

0.0 1.1 2.8 6.3 16.7

0.5 1.2 2.7 6.3 16.8

Maximizing Benefit of Classifications Using Feature Intervals

5

345

Conclusions and Future Work

In this study, we have focused on the problem of making predictions when the outcomes have different benefits associated with them. We have implemented a new algorithm, namely BMFI that uses the predictive power of feature intervals concept in maximizing the total benefit of classifications. We make direct use of benefit matrix information provided to the algorithm in tuning the prediction so that the resultant benefit gain is maximized. BMFI has been compared to MetaCost and two other cost-sensitive classification algorithms provided in Weka. These generic algorithms are wrapped over NBC, C4.5 and VFI. The results show that BMFI is very effective in maximizing the benefit per instance values. It is more successful in domains where the prediction of a certain class is particularly important. Empirical results we obtained also show that using benefit information directly in the algorithm itself is more effective than using a meta-stage around the base classifier. In benefit maximization problem, we have observed that individual characteristics of the datasets influence results significantly, due to the extreme correlation between cost-sensitivity and class distributions. As future work, feature-dependent domains can be explored in depth and feature-dependency aspect of BMFI can be improved. Benefit maximization can be extended to include the feature costs. Feature selection mechanisms that are sensitive to individual costs of features can be utilized.

References [1] Blake C. L. and Merz C. J.: UCI repository of machine learning databases. University of California, Irvine, Department of Information and Computer Sciences (1998) [http://www.ics.uci.edu/˜mlearn/MLRepository.html] 343 [2] Domingos P.: Metacost: A general method for making classifiers cost-sensitive. In: Proceedings of the International Conference on Knowledge Discovery and Data Mining, San Diego, CA (1999) 155-64 343 [3] Elkan C.: The Foundations of Cost-Sensitive Learning. In: Proceedings of the Seventeenth International Joint Conference on Articial Intelligence (2001) 340 [4] Frank E. et al.: Weka 3 - Data Mining with Open Source Machine Learning Software in Java. The University of Waikato (2000) [http://www.cs.waikato.ac.nz/˜ml/weka] 343 [5] G¨ uvenir H. A.: Detection of abnormal ECG recordings using feature intervals. In: Proceedings of the Tenth Turkish Symposium on Artificial Intelligence and Neural Networks (2001) 265-274 341 ˙ [6] G¨ uvenir H. A, Demir¨ oz G., and Ilter N.: Learning differential diagnosis of erythemato-squamous diseases using voting feature intervals. Artificial Intelligence in Medicine, Vol. 13(3) (1998) 147-165 339, 340, 343 ˙ [7] Ikizler N.: Benefit Maximizing Classification Using Feature Intervals. Technical Report BU-CE-0208, Bilkent University (2002) 343 [8] Ting K. M.: An instance weighting method to induce cost-sensitive trees. IEEE Transactions on Knowledge and Data Engineering, Vol. 14(3) (2002) 659-665 344

Parameter Estimation for Bayesian Classification of Multispectral Data Refaat M Mohamed and Aly A Farag Computer Vision and Image Processing Laboratory University of Louisville, Louisville, KY 40292, USA {refaat,farag}@ cvip.uofl.edu www.cvip.uofl.edu

Abstract. In this paper, we present two algorithms for estimating the parameters of a Bayes classifier for remote sensing multispectral data. The first algorithm uses the Support Vector Machines (SVM) as a multi-dimensional density estimator. This algorithm is a supervised one in the sense that it needs in advance, the specification of the number of classes and a training sample for each class. The second algorithm employs the Expectation Maximization (EM) algorithm, in an unsupervised way, for estimating the number of classes and the parameters of each class in the data set. Performance comparison of the presented algorithms shows that the SVM- based classifier outperforms those based on Gaussian-based and Parzen window algorithms. We also show that the EM based classifier provides comparable results to Gaussian-based and Parzen window-based while it is an unsupervised algorithm. Keywords: Bayes classification, density estimation, support vector machines (SVM), expectation maximization (EM), multispectral data.

1

Introduction

Bayes classifier constitutes the basic setup for a large category of classifiers. To implement the bayes classifier, there are two main parameters to be addressed [1]. The first parameter is the estimation of the class conditional probability for each class, and the second one is the prior probability of each class. In this paper, we address both of these parameters. Support Vector Machines (SVM) approach was developed to solve the classification problem, but recently have been extended to regression problems [2]. SVM have been shown to perform well for density estimation where the probability distribution function of the feature vector x can be inferred from a random sample Ð. SVM represent Ð by a few number of support vectors and the associated kernels [3]. This paper employs the SVM as a density estimator for multi-dimensional feature spaces, and uses this estimate in a Bayes classifier. SVM as a density estimator work V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 346-355, 2003.  Springer-Verlag Berlin Heidelberg 2003

Parameter Estimation for Bayesian Classification of Multispectral Data

347

in a supervised setup; the number of classes and a design data sample from each class need to be available. There are a number of unsupervised algorithms for parameter estimation for Bayes classifier, e.g. the well known k-means algorithms [4]. The k-means algorithm requires the knowledge of the number of classes. In this paper, we propose a fully unsupervised approach for estimating the Bayes classifier parameters. The Expectation Maximization (EM) algorithm is a general method of finding the maximum-likelihood estimate (MLE) of the parameters of an underlying distribution from a given data set, when the data is incomplete or has missing values. The proposed approach in this paper incorporates the EM algorithm with some generalizations to estimate the number of classes, parameter of each class as well as the prior probability of each class. The two algorithms introduced in this paper are the following: Algorithm 1: SVM as a density estimator for Bayes Classifier; and Algorithm 2: EM algorithm for Estimating number of classes, class conditional probability parameters, as well as the prior probabilities for a Bayesian classification of multispectral data classification. We describe the details of the algorithms and implementation for classification of seven-dimensional multispectral Landsat data in a Bayes classifier setup. We evaluate the performance of the two algorithms with respect to the following classical approaches: Gaussian-based and Parzen window-based classifiers. The proposed algorithms show high performance against these algorithms.

2

Support Vector Machines for Density Estimation

Support Vector Machines (SVM) have been developed by Vapnik [5] and are gaining popularity due to many attractive features and promising empirical performance. The formulation embodies the Structural Risk Minimization (SRM) principle, which has been shown to be superior to traditional Empirical Risk Minimization (ERM) principle employed in conventional learning algorithms (e.g. neural networks). SRM minimizes an upper bound on the generalization error, as opposed to ERM, which minimizes the error on the training data. It is this difference, which makes SVM more attractive in statistical learning applications. In this section, we present a brief outlines of the SVM approach for density estimation. 2.1

The Density Estimation Problem

The probability density function p(x) of the random vector x is a nonnegative quantity which is defined as: x F (x) = ∫ p (x ′) dx ′ −∞

(1)

where F (x) is the cumulative distribution function. Hence, in order to estimate the probability density, we need to solve the integral equation:

348

Refaat M Mohamed and Aly A Farag

x ∫ p (x ′, α ) dx ′ = F (x) −∞

(2)

on a given set of densities p( x, α ) , where the integration in (2) is a vector integration, and α is the set of parameters to be determined. The estimation problem in (2) can be regarded as solving the linear operator equation: A p ( x) = F ( x)

(3)

where the operator A is a one-to-one mapping for the elements p(x) of the Hilbert space E1 into elements F (x) of the Hilbert space E 2 . 2.2

Support Vector Machines

In support vector machines, we look for a solution of the density estimation problem in the form: N

p ( x) =

∑β

k

K ( x, x k )

(4)

k =1

where N is the size of the training sample, K (x, x k ) is the kernel that defines a

Reproducing Kernel Hilbert Space (RKHS), and β k ' s are the parameters to be estimated using the SVM method. The density estimation problem in (3) is known to be an ill-posed problem, see [6]. One solution for the ill-posed problems is to introduce a semi-continuous, and positive functional (Λ ( p(x)) ≤ c ; c > 0 ) in a RKHS. Also, we define p(x) as a trade-off between Λ ( p(x)) and

Ap (x) − FN (x) . An

example for such method is that by Phillips [6]: min p ( x)

Λ ( p (x))

(5)

Ap(x) − FN (x) < ε N , ε N > 0, ε N → 0

such that:

= εk max FN (x) − ∫−x ∞ p(t ) dt x = xk k

or simply,

(6)

where FN (x) is an estimate for F (x) which is characterized by an estimate error ε k for each instant x k , see [6]. Using the properties of RKHS, (5) rewritten as:  N β k K (x, x k ), Λ( p( x)) = ( p(x), p(x)) H =    k =1

∑

N

=

N

∑ β ∑ β (K (x, x k

k =1

j

j =1

N

∑β j =1

j

 K ( x, x j )   H N

k

), K (x, x j )

N

)H = ∑∑ β k β j K (x k , x j ) k =1 j =1

(7)

Parameter Estimation for Bayesian Classification of Multispectral Data

349

Therefore, to solve the estimation problem in (3) we minimize the functional: N

W ( β ) = Λ( p(x), p(x)) =

N

∑∑ β

k β j K (x k , x j )

(8)

k =1 j =1

subject to the constraints (6), and : N

βk ≥0,

∑β

j

=1

(9)

j =1

where 1 ≤ k ≤ N . The constraints in (9) are imposed to obtain the solution in the form of a mixture of densities. The only remaining issue for the SVM description is the choice of the functional form for the kernel. To obtain a solution in the form of a mixture of densities, we choose a nonnegative kernel which satisfies the following conditions:  x − xk   x − xk  ; a(γ ) K  K γ (x, x k ) = a (γ ) K   γ   γ and K (0) = 1

∫

  dx = 1 ; 

(10)

Following is a listing of the implementation steps for SVM as a density estimator. Algorithm 1. SVM density estimator Step 1: Step 2: Step 4: Step 5: Step 6: Step 7:

Get the random sample Ð from the density function to be estimated. Use Ð to construct FN (x) and ε k ' s . Select a kernel function that satisfies (10). Set up the objective function (8) for the optimization algorithm. Set up the constraints (6) and (9). Apply the optimization algorithm. The nonzero parameters ( β k ' s ) and their corresponding data vectors are considered as the support vectors. Step 8: Calculate the density function value from (4) corresponding to a feature vector x .

3

Unsupervised Density Estimation Using the EM Algorithm

The EM algorithm had been proposed as an iterative optimization algorithm to find the maximum likelihood estimates (e.g., [7][8]). In this paper, we use the EM algorithm to estimate the parameters for a mixture of densities. For an observed data set X of size N , the probabilistic model is defined as: M

p (x | Θ) =

∑α

j

p j (x | θ j )

j =1

where,

M : number of different components “classes” in the mixture,

(11)

350

Refaat M Mohamed and Aly A Farag M

α j : weight of component i in the mixture;

∑α

j

=1 ,

j =1

θ j : parameters indexing the density of component i in the mixture, Θ = (α j ' s , θ j ' s ) : parameters that indexing the mixture density.

The likelihood for the mixture density can be written as: N

l (Θ | X) =

∏ p(x

| Θ)

i

(12)

i =1

The optimization problem can be significantly simplified if we assume that the data set X is incomplete and there is an unobserved data set w = {wi }iN=1 , each component of which is associated with a component of X . The values of w determine which density in the mixture (i.e. class) generated the particular data instant. Thus, we have wi = k , for wi ∈{1, 2, ..., M } , if the sample i of X is generated by mixture component k . If w is known, (12) can be written as: N

∑ log(P(x

log(l (Θ | X, w ) ) = log(P ( X, w | Θ) ) =

i

| wi ) P (w ) )

i =1

N

=

∑ log(α

wi

p wi (x i | θ wi )

i =1

(13)

)

The parameter set Θ = (α j ' s , θ j ' s ) can be obtained if the components of w are known. In the case where w is unknown, we still can proceed by assuming w to be a random vector. In the reminder of this section, we will develop an expression for the distribution of the unobserved data w . We start by guessing initial values Θ c of the mixture density parameters, i.e. we assume Θ c = (α cj , θ cj ; j = 1...M ) . Using Bayes rule, the conditional probability density for the value wi ”class of site i ” is written as: p( wi | x i , Θ c ) =

α wc i p wi (x i | θ cwi ) p (x i | Θ)

=

α wc i p wi (x i | θ cwi )

∑

M

α kc p k (x i | θ ck )

(14)

k =1

Then, the joint density for w can be written as: p(w | X, Θ c ) =

N

∏ p(w

i

i =1

| xi , Θc )

(15)

Parameter Estimation for Bayesian Classification of Multispectral Data

3.1

351

Maximizing the Conditional Expectation

The EM algorithm first finds the expected value (Q(Θ, Θ c )) of the log-likelihood of the complete-data (the observed data X and the unobserved data w ) with respect to the unknown data w . The pieces of information which known here are the observed data set X and the current parameter estimates Θ c . Thus, we can write: Q (Θ, Θ c ) = E [( log p ( X, w | Θ) ) | X, Θ c ]

(16)

where Θ are the parameters that we optimize them to maximize the conditional expectation Q . In (16), X and Θ c are constants, Θ is a regular variable that we try to adjust, and w is a random variable that governed by the density function defined in (15). Thus, using (13) and (15), conditional expectation in (16) can be rewritten as: Q(Θ, Θ c ) =

∫ log p(X, w | Θ) p(w | X, Θ

c

) dw =

∑ log (l ( Θ | X, w)) p(w | X, Θ

c

)

w∈w

w∈w

(17)

With some mathematical simplification for (17) the conditional expectation can be rewritten as: Q (Θ, Θ c ) =

M

N

∑∑ log (α

j

p j (x i | θ j )) p ( j | x i , Θ c )

j =1 i =1

M

=

N

∑∑ log(α

j ) p(

c

j | xi , Θ ) +

j =1 i =1

M

(18)

N

∑∑ log( p

j (x i

c

| θ j )) p ( j | x i , Θ )

j =1 i =1

In (18), the term containing α j and the term containing θ j are independent. Hence, those terms can be maximized independently, which significantly simplifies the optimization problem. 3.2

Parameters Estimation

In this section we will present the implementation steps to estimate the different parameters of the density function underlying the distribution of the classes in a data set. The density is assumed to be a mixture of densities. Thus, the parameters to be estimated are, number of components in the mixture, the mixing coefficients and the parameters of each density component As stated before, the first term of the conditional expectation in (18) can be maximized independently to obtain the mixing coefficients, α j ' s . Maximizing the first term in (18) results in the value of a mixing coefficient as:

αj=

1 N

N

∑ p( j | x , Θ i

c

)

(19)

i =1

We stated before, that we use the EM algorithm under the assumption that we set a parametric form for the density of each class. Theoretically, the algorithm can be

352

Refaat M Mohamed and Aly A Farag

applied to any form for the class densities. For computational purpose, we assume the class densities to be in the form of multinomial Gaussian. Under this assumption, the parameters will be the mean vector and the covariance matrix, i.e. θ j = (µ j , Σ j ) . Under the normality assumption, the second term of (18) becomes: M

N

∑∑log(p (x | θ ))p( j | x , Θ ) = j

i

j

i

c

j =1 i=1

M

N

1   1 T −1 c  − log(| Σ j |) − (xi − µ j ) Σ j (xi − µ j )p( j | xi , Θ ) 2 2  j =1 i =1 

∑∑

(20)

Maximizing (20) with respect to µ, and Σ will give: N

µj =

∑

N

x i p( j | x i , Θ c )

i =1 N

∑

∑ p( j | x , Θ i

and Σ j = p( j | x i , Θ c )

i =1

c

)(x i − µ j )(x i − µ j ) T

i =1

N

∑

(21) p( j | x i , Θ c )

i =1

The essence of the proposed algorithm is that we can automatically estimate the number of classes M in the scene. The optimum value of M is that value at which Q is maximum. This can be done simply by assuming a number of classes, M and then optimize (16) to obtain the other parameters and at the same time we calculate the final optimized value of Q . Then, we change M and redo the optimization process. This iterative procedure is continued until a maximum value of Q is obtained. The value of M at which Q is maximum is the optimum number of classes defined in the data set. Algorithm 2. Step 1: Step 2: Step 3: Step 4: Step 5: Step 6:

Set a value for M . Define an initial estimate for Θ . Compute the value of Q from (18) using these parameters. Compute the new values of Θ from (19), and (21). Compute the new value of Q . Repeat 4 and 5 to get acceptable small difference between subsequent values of Q . Step 7: Increase M by one and iterate from step 2 until Q is maximum.

4

Experimental Results

In this section we present the classification results of using the above two algorithms as density estimators in the design of a Bayesian classifier. For the SVM approach, a 7-D Gaussian-like kernel function has been used. The experiments were carried out

Parameter Estimation for Bayesian Classification of Multispectral Data

353

using real data acquired from the Landsat Thematic Mapper (TM) for two different data sets as will be discussed below. The proposed two algorithms are evaluated against the Gaussian-based and Parzen window-based algorithms. Details of these algorithms are elsewhere (e.g. [1]). 4.1

Golden Gate Bay Area

The first part of the experiments is carried out using real multispectral data (7-bands) for the Golden Gate Bay area of the city of San Francisco, California, USA. Five classes are defined on this image: Trees, Streets (light urban), Water, Buildings (dense urban) and Earth. The available ground truth data for this data set includes about 1000 points per class, which is approximately 1% of the image scene. The classification results, Table 1 and Fig.1 and, show that the SVM-based classifier outperforms Gaussian- based and Parzen window-based algorithms, which reflects that the SMV works well as a density estimator in multi-dimensional. For the second algorithm, Fig. 2 shows the evolution of the conditional expectations with the number of components assumed in the area. The figure shows that the maximum value of Q occurs at M = 5 , which is the nominal value. The estimated parameters using the proposed algorithm are used in a Bayes classifier. Table 1 shows that the unsupervised classifier provides comparable results which reflects its applicability for the problem.

Fig. 1. Classified Images for the Golden Gate Area. From upper left to lower right, original, Gaussian-based, Parzen Window-based, SVM-based, and EM-based Bayes classifier

4.2

Agricultural Area

The first part of this experiment was carried out using real multispectral data for an agricultural area, Fig 3. It is known that this area contains: Background, Corn, Soybean, Wheat, Oats, Alfalfa, Clover, Hay/Grassland, and Unknown. The ground

354

Refaat M Mohamed and Aly A Farag

truth data for that area is available. Again, the results in Fig 1 and Table 2 emphasize the same concluded notes in the above Golden Gate Bay area data set.

Fig. 2. Evolution of Conditional Expectation with the number of classes, left Golden Gate, right Agricultural area (Actual value is Q*104)

Fig. 3. Classified Images for the Agricultural Area. From upper left to lower right, original, Gaussian-based, Parzen Window-based, SVM-based, and EM-based Bayes classifier

Parameter Estimation for Bayesian Classification of Multispectral Data

5

355

Conclusion and Future Work

This paper presented two algorithms for estimating the parameters for a Bayes classifier of multispectral data classification. The first one is a supervised algorithm which uses the SVM as an estimator for the class conditional probabilities. The second one is an unspervised and it assumes the density distribution of a class as a mixture of different densities. The algorithm is used to estimate the number of components of this mixture (number of classes), the parameters of each component and the proportion of each component in the mixture (which is regarded as the prior probability of the class). The results show that the SVM has the capabilities to simulate well density functions in high dimensions. Also, the results illustrated that the unsupervised algorithm performs well with the data sets used in the experiments. For future work, we study methods for enhancing the SVM performance and to automatically choose its parameters. For the unsupervised algorithm, we will relax the normality assumption of the densities. Also, we will adjust the algorithm to estimate the distribution of the observed data set assuming nonparametric forms for the densities of the classes in the data.

References [1] [2] [3] [4] [5] [6] [7] [8]

A. Farag, R. M. Mohamed and H. Mahdi, “Experiments in Image Classification and Data Fusion,” Proceedings of 5th International Conference on Information Fusion, Annapolis, MD, Vol. 1, pp. 299-308, July 2002. V. Vapnik, S. Golowich and A. Smola, “Support Vector Method for Function Approximation, Regression Estimation, and Signal Processing,” Neural Information Processing Systems, Vol. 9, MIT Press, Cambridge, MA, 1997. Vapnik and S. Mukherjee, “Support Vector Method for Multivariate Density Estimation,” Neural Information Processing Systems 1999, Vol. 12, MIT Press 2000. R. O. Duda and et al., Pattern Classification, John Wiley & Sons, 2nd edition, 2001. V. N. Vapnik, The Nature of Statistical Learning Theory, Springer, 2nd Edition, 2000. Aly Farag, and Refaat Mohamed, “Classification of Multispectral Data Using Support Vector Machines Approach for Density Estimation,” Proceedings of the 7th INES 2003, Assiut, Egypt, March4-6, 2003. D. Phillips, “A Technique for the Numerical Solution of Certain Integral Equations of the First Kind,” Journal of Association Computing Machinery, 9:84-97, 1962. Greg A. Rempala and Richard A. Derrig, “ Modeling Hidden Exposures in Claim Severity via the EM Algorithm,” TR-MATH, Mathematical department, University of Louisville, December 2002.

Goal Programming Approaches to Support Vector Machines Hirotaka Nakayama1, Yeboon Yun2 , Takeshi Asada3 , and Min Yoon4 1

Konan University, Kobe 658-8501, Japan Kagawa University, Kagawa 761-0396, Japan 3 Osaka University, Osaka 565-0871, Japan Yonsei University, Seoul 120-749, Republic of Korea 2

4

Abstract. Support vector machines (SVMs) are gaining much popularity as effective methods in machine learning. In pattern classification problems with two class sets, their basic idea is to find a maximal margin separating hyperplane which gives the greatest separation between the classes in a high dimensional feature space. However, the idea of maximal margin separation is not quite new: in 1960’s the multi-surface method (MSM) was suggested by Mangasarian. In 1980’s, linear classifiers using goal programming were developed extensively. This paper considers SVMs from a viewpoint of goal programming, and proposes a new method based on the total margin instead of the shortest distance between learning data and separating hyperplane.

1

Introduction

Recently, Support Vector Machines (SVMs, in short) are gaining much popularity for machine learning. One of main features of SVMs is that they are kernel based linear classifiers with maximal margin on the feature space . The idea of maximal margin in linear classifiers is intuitive, and its reasoning in connection with perceptrons was given in early 1960’s (e.g., Novikoff [6]). The maximal margin is effectively applied for discrimination analysis using mathematical programming, e.g., MSM (Multi-Surface Method) by Mangasarian [4]. Later, linear classifiers with maximal margin were formulated as linear goal programming, and extensively studied through 1980’s to the beginning of 1990’s. The pioneering work was given by Freed-Glover [2], and a good survey can be seen in Erenguc-Koehler et al. [1]. This paper discusses the goal programming approach to SVMs, and shows a close relationship between the so-called ν-SVM and our suggested total margin method.

2

Goal Programming Approach to Linear Discrimination Analysis

Let X be a space of conditional attributes. For binary classification problems, the value of +1 or −1 is assigned to each data xi according to its class A or V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 356–363, 2003. c Springer-Verlag Berlin Heidelberg 2003


Goal Programming Approaches to Support Vector Machines

357

B. The aim of machine learning is to predict which class newly observed data belong to by virtue of a certain discrimination function f (x) constructed on the basis of the given data set (xi , yi ) (i = 1, . . . , l), where yi = +1 or −1. Namely, f (x)
0 ⇒ x ∈ A f (x) < 0 ⇒ x ∈ B When the discrimination function is linear, namely if f (x) = wT x + b, this is referred to as a linear classifier. In this case, the discrimination boundary is given by a hyperplane defined by wT x + b = 0. In 1981, Freed-Glover suggested to get a hyperplane separating two classes with as few misclassified data as possible by using goal programming [2]. Let ξi denote the exterior deviation which is a deviation from the hyperplane for a point xi improperly classified. Similarly, let ηi denote the interior deviation which is a deviation from the hyperplane for a point xi properly classified. Some of main objectives in this approach are as follows: i) Minimize the maximum exterior deviation (decrease errors as much as possible) ii) Maximize the minimum interior deviation iii) Maximize the weighted sum of interior deviation iv) Minimize the weighted sum of exterior deviation Of course, we can combine some of them. Note that the objective ii) corresponds to maximizing the shortest distance between the learning data and the hyperplane. Although many models have been suggested, the one considering iii) and iv) above may be given by the following linear goal programming: Letting yi = +1 for i ∈ IA and yi = −1 for i ∈ IB , (GP)

minimize

l


(hi ξi − ki ηi )

i=1

subject to

yi (xTi w + b) = ηi − ξi , ξi , ηi
0, i = 1, . . . , l.

Here, hi and ki are positive constants. In order for ξi and ηi to have the meaning of the exterior deviation and the interior deviation respectively, the condition ξi ηi = 0 for every i = 1, . . . , l must hold. Lemma 1. If hi > ki for i = 1, . . . , l, then we have ξi ηi = 0 for every i = 1, . . . , l at the solution to (GP). Proof. Easy due to lemma 7.3.1 of [7]. It should be noted that the above formulation may yield some unacceptable solutions such as w = 0 and unbounded solution.

358

Hirotaka Nakayama et al.

Example Let z1 = (−1, 1), z2 = (0, 2) ∈ A and z3 = (1, −1), z4 = (0, −2) ∈ B. Constraint functions of SVM are given by z1 z2 z3 z4

: : : :

w1 (−1) + w2 (1) + b = η1 − ξ1 w1 (0) + w2 (2) + b = η2 − ξ2 w1 (1) + w2 (−1) + b = η3 − ξ3 w1 (0) + w2 (−2) + b = η4 − ξ4

Here, it is clear that ξ = 0 at the optimal solution. The constraints include ηi added in the right hand side. Note that the feasible region  in this formulation moves to the north-west by increasing ηi . Maximizing ηi yields unbounded optimal solution unless any further constraint in w are added. In the goal programming approach to linear classifiers, therefore, some appropriate normality condition must be imposed on w in order to provide a bounded optimal solution. One of such normality conditions is ||w|| = 1. If the classification problem is linearly separable, then using the normalization ||w|| = 1, the separating hyperplane H : wT x+b = 0 with maximal margin can be given by (GP1 )

maximize

η

subject to

yi (xTi w + b)
η, i = 1, . . . , l, ||w|| = 1.

However, this normality condition makes the problem to be of nonlinear optimization. Note that the following SVM formulation with the objective function minimizing ||w|| can avoid this unboundedness handily. Instead of maximizing the minimum interior deviation in (GP1 ) stated above, we use the following equivalent formulation with the normalization wT z+b = ±1 at points with the minimum interior deviation: (SVM)

minimize subject to

||w||   yi w T z i + b
1, i = 1, . . . , l,

where yi is +1 or −1 depending on the class of z i . Several kinds of norm are possible. When ||w||2 is used, the problem is reduced to quadratic programming, while the problem with ||w||1 or ||w||∞ is reduced to linear programming (see, e.g., Mangasarian [5]). For the above example, we have the following condition in the SVM formulation: z1 : w1 (−1) + w2 (1) + b
1 z2 : w1 (0) + w2 (2) + b
1 z3 : w1 (1) + w2 (−1) + b  −1 z4 : w1 (0) + w2 (−2) + b  −1 Since it is clear that the optimal hyperplane has b = 0, the constraint functions for z3 and z4 are identical to those for z1 and z2 . The feasible region in

Goal Programming Approaches to Support Vector Machines

359

(w1 , w2 )-plane is given by w2
w1 + 1 and w2
1/2. Minimizing the objective function of SVM yields the optimal solution (w1 , w2 ) = (−1/2, 1/2) for the QP formulation. Similarly, we have a solution among the line segment {w2
w1 + 1} ∩ {−1/2  w1  0} depending on the initial solution for the LP formulation. On the other hand, Glover [3] shows the following necessary and sufficient condition for avoiding unacceptable solutions:

xi + lB xi )T w = 1, (1) (−lA i∈IB

i∈IA

where lA and lB denote the number of data for the category A and B, respectively. Geometrically, the normalization (1) means that the distance between two hyperplanes passing through centers of data for lA and lB is scaled by lA lB . Taking into account that ηi /||w|| represents the margin of correctly classified data xi from the hyperplane wT x + b = 0, larger value of ηi and smaller value of ||w|| are more desirable in order to maximize the margin. On the other hand, since ξi /||w|| stands for the margin of misclassified data, the value of ξi should be minimized. The methods considering all of ξi /||w|| and ηi /||w|| are referred to as the total margin methods. Now, we have the following formulation for getting a linear classifier with maximal total margin: (GP2 )

minimize

||w|| +

l


(hi ξi − ki ηi )

i=1

subject to

  yi xTi w + b = ηi − ξi , i = 1, . . . , l, ξi , ηi
0, i = 1, . . . , l.

If we maximize the smallest margin for correctly classified data instead of the sum of all margins, the formulation is reduced to the well known ν-SVM by setting ηi
ρ, hi = 1/l, and ki = ν: l

(ν−SVM)

minimize subjectto

1
ξi l i=1   yi xTi w + b
ρ − ξi , i = 1, . . . , l,

||w|| − νρ +

ξi , ηi
0, i = 1, . . . , l. In the following, we use a simplified formulation for (GP2 ) as follows: (GP3 )

minimize subject to

l l

1 ||w||22 + C1 ξi − C2 ηi 2 i=1 i=1   yi xTi w + b
ηi − ξi , i = 1, . . . , l,

ξi , ηi
0, i = 1, . . . , l.

360

Hirotaka Nakayama et al.

Note that equality constraints in (GP2 ) are changed into inequality constraints (GP3 ) which bring the equivalent solution. When the data set is not linearly separable, we can consider linear classifiers on the feature space mapped from the original data space by some nonlinear map Φ in a similar fashion to SVMs. Considering the dual problem to (GP3 ), we have a simple formulation by using a kernel K(xi , xj ) with the property K(xi , xj ) = Φ(xi ), Φ(xj ) as follows: l 1
(GP3 −SVM) minimize yi yj αi αj K (xi , xj ) 2 i,j=1 subject to

l


αi yi = 0,

i=1

C2  αi  C1 , i = 1, . . . , l. Let α be the optimal solution to the problem (GP3 -SVM). Then, ∗

w=

l


yi α∗i xi .

i=1

  n+ +n− l

  1 ∗ b = yi αi K xi , xj , (n+ − n− ) − n+ + n− j=1 i=1 ∗

where n+ is the number of xj with C2 < α∗j < C1 and yj = +1 and n− is the number of xj with C2 < α∗j < C1 and yj = −1. Likewise, similar formulations following i) and/or iv) of the main objectives of goal programming are possible: l
1 ||w||22 + C1 σ − C2 minimize ηi (GP4 ) 2 i=1   subject to yi xTi w + b
ηi − σ, i = 1, . . . , l, (GP5 )

minimize subject to

σ, ηi
0, i = 1, . . . , l. 1 ||w||2 + C1 σ − C2 ρ 2 2  yi xTi w + b
ρ − σ, i = 1, . . . , l, σ, ρ
0.

GP4 and GP5 can be reformulated using kernels in a similar fashion to GP3 SVM. l 1
minimize yi yj αi αj K (xi , xj ) (GP4 −SVM) 2 i,j=1 subject to

l
i=1 l


αi yi = 0, αi  C1 ,

i

C2  αi , i = 1, . . . , l.

Goal Programming Approaches to Support Vector Machines

361

Table 1. Classification rate of various SVMs (Liver–90% training data) SVM(hard) SVM(soft) ν-SVM GP3 SVM GP4 SVM GP5 SVM

GP

AVE

–

67.57

68.38

71.11

63.53

59.08

69.12

STDV

–

8.74

7.10

8.59

8.10

6.79

6.55

TIME

–

221

109.96

192.14

1115

93.83

2.29

Table 2. Classification rate of various SVMs (Liver–70% training data) SVM(hard) SVM(soft) ν-SVM GP3 SVM GP4 SVM GP5 SVM

GP

AVE

66.51

65.32

67.35

68.33

65.05

55.48

68.16

STDV

2.43

4.83

4.2

4.68

6.39

5.28

2.94

TIME

20.2

161.81

28.57

61.33

109.83

19.98

1.23

Table 3. Classification rate of various SVMs (Cleveland–90% training data) SVM(hard) SVM(soft) ν-SVM GP3 SVM GP4 SVM GP5 SVM

GP

AVE

74.67

80.0

80.59

79.89

71.67

72.81

80.67

STDV

6.86

4.88

4.88

5.85

5.63

5.63

5.33

TIME

17.57

35.06

22.4

36.04

6.86

6.86

2.25

(GP5 −SVM)

minimize

subject to

l 1
yi yj αi αj K (xi , xj ) 2 i,j=1 l


αi yi = 0,

i=1

C2 

l


αi  C1 ,

i

0  αi , i = 1, . . . , l.

3

Numerical Example

We compare several SVMs through well known bench mark problems: 1) BUPA liver disorders(345 instances with 7 attributes), 2) Cleveland heart-disease (303 instances with 14 attributes)(http://www.ics.uci.edu/ mlearn/MLSummary.html).

362

Hirotaka Nakayama et al.

Table 4. Classification rate of various SVMs (Cleveland–70% training data) SVM(hard) SVM(soft) ν-SVM GP3 SVM GP4 SVM GP5 SVM

GP

AVE

75.6

81.07

81.97

81.03

79.01

75.06

83.19

STDV

6.27

3.29

3.95

2.88

2.58

6.08

2.78

TIME

6.6

12.64

9.18

14.41

12.51

5.63

1.16

We made 10 trials with randomly selected training data of 90% and 70% from the original data set and the test data of rest 10% and 20%, respectively. Tables 1-4 show the results. We used the computer with CPU of Athron +1800. As can be seen in Table 1, SVM (hard margin) cannot produce any solution for the 90% training data of BUPA liver disorders problem. This seems because the problem is not linearly separable in a strict sense even in the feature space. On the other hand, however, the linear classifier of GP provides a good result. Therefore, we may conclude that the problem is nearly linearly separable, but not in a strict sense. The data around the boundary may be affected by noise, and this causes that nonlinear decision boundaries can not necessarily yield good classification ability.

4

Concluding Remarks

SVM was discussed from a viewpoint of Goal Programming. Since the model (GP) is a linear programming, the computation time is extremely short. However, GP sometimes produces undesirable solutions. It is seen through our experiments that the stated normalization can not get rid of this difficulty completely. SVMs with LP formulation are of course possible. We shall report numerical experiments with LP-based SMVs in another opportunity.

References [1] Erenguc, S. S. and Koehler, G. J. (1990) Survey of Mathematical Programming Models and Experimental Results for Linear Discriminant Analysis, Managerial and Decision Economics, 11, 215-225 356 [2] Freed, N. and Glover, F. (1981) Simple but Powerful Goal Programming Models for Discriminant Problems, European J. of Operational Research, 7, 44-60 356, 357 [3] Glover, F. (1990) Improved Lineaar Programming Models for Discriminant Analysis, Decision Sciences, 21, 771-785 359 [4] Mangasarian, O. L. (1968): Multisurface Method of Pattern Separation, IEEE Transact. on Information Theory, IT-14, 801-807 356 [5] Mangasarian, O. L. (2000) Generalized Support Vector Machines, in Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Sh¨ olkopf, and D. Schuurmans (eds.) MIT Press, Cambridge, pp.135-146 358

Goal Programming Approaches to Support Vector Machines

363

[6] Novikoff, A. B. (1962): On Convergence Proofs on Perceptrons, Symposium on the Mathematical Theory of Automata, 12, 615-622 356 [7] Sawaragi, Y., Nakayama, H. and Tanino, T. (1994) Theory of Multiobjective Optimization, Academic Press 357 [8] Sch¨ olkopf, B. and Smola, A. S. (1998) New Support Vector Algorithms, NeuroCOLT2 Technical report Series NC2-TR-1998-031

Asymmetric Triangular Fuzzy Sets for Classification Models J. F. Baldwin and Sachin B. Karale* A.I. Group, Department of Engineering Mathematics, University of Bristol Bristol BS8 1TR, UK {Jim.Baldwin,S.B.Karale}@bristol.ac.uk

Abstract. Decision trees have already proved to be important in solving classification problems in various fields of application in the real world. The ID3 algorithm by Quinlan is one of the well-known methods to form a classification tree. Baldwin introduced probabilistic fuzzy decision trees in which fuzzy partitions were used to discretize continuous feature universes. Here, we have introduced a way of fuzzy partitioning in which we can have asymmetric triangular fuzzy sets for mass assignments approach to fuzzy logic. In this paper we have shown with example that the use of asymmetric and unevenly spaced triangular fuzzy sets will reduce the number of fuzzy sets and will also increase the efficiency of probabilistic fuzzy decision tree. Keywords. Decision trees, fuzzy decision trees, fuzzy sets, mass assignments, ID3 algorithm

1

Introduction

ID3 is a classic decision tree generation algorithm for classification problems. This was later extended to C4.5 to allow continuous variables with the use of crisp partitioning [1]. If attribute A is partitioned in two sets, one having values of A> α and the other A ≤ α , for some parameter α , then small changes in attribute value can result in inappropriate changes to the assigned class. So most of the time the ID3 algorithm becomes handicapped in the presence of continuous attributes. There are many more partitioning techniques available, to deal with continuous attributes [2,3]. Fuzzy ID3 extends ID3 by allowing the values of attributes for fuzzy sets. This allows discretization of continuous variables using fuzzy partitions. The use of fuzzy sets to partition the universe of continuous attributes gives much more significant results [2]. Baldwin's mass assignment theory is used to translate attribute values to probability distribution over the fuzzy partitions. Fuzzy decision rules are less sensitive to small changes in attribute values near partition boundaries and thus helps to obtain considerably good results from ID3 algorithm. *

Author to whom correspondence should be addressed.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 364-370, 2003.  Springer-Verlag Berlin Heidelberg 2003

Asymmetric Triangular Fuzzy Sets for Classification Models

365

A fuzzy partition in the context of this paper is a set of triangular fuzzy sets such that for any attribute value the sum of membership of the fuzzy sets is 1. The use of symmetric, evenly spaced triangular fuzzy sets generates a decision tree, which normally gives good classification on training sets but for test data sets it may tend to give erroneous results, depending on the data set used as training set. This can be avoided by using asymmetric, unevenly spaced triangular fuzzy sets. This is discussed in the following sections and supported with examples.

2

Symmetric and Asymmetric Triangular Fuzzy Partitioning

The need of partition universe has introduced a new problem into decision tree induction. The use of asymmetric, unevenly spaced triangular fuzzy sets can help to improve the results compared to those obtained by symmetric, evenly spaced triangular fuzzy sets. Asymmetric fuzzy sets will give improved results or the similar results depending on circumstances. 2.1

Symmetric, Evenly Spaced Triangular Fuzzy Sets

Symmetric, evenly spaced triangular fuzzy partitioning is very much popular and has been used for various applications in number of research areas. If fuzzy sets {f1,…,fn}forms fuzzy partition then we have n

∑ prob( f ) =1 i

(1)

i =1

where prob( f ) is the membership function of ‘ f ', with respect to a given distribution over the universe [4]. While forming the fuzzy partitions the total universe is evenly divided into number of symmetric triangular partitions, decided by the user, which generally looks like the fuzzy partitions as shown in Fig.1.

Fig. 1. Symmetric, evenly spaced triangular fuzzy sets

Depending on the training data set available this may result into forming one or more fuzzy partitions, which have no data point associated with them. In this case if there is a data point in a test data set which lies in this fuzzy partition then obviously

366

J. F. Baldwin and Sachin B. Karale

the model gives equal probability towards each class for that data point and thus there is the possibility of erroneous results. 2.2

Asymmetric, Unevenly Spaced Triangular Fuzzy Sets

Same as in symmetric fuzzy sets, in this case also, we have fuzzy sets {f1,…,fn} forms fuzzy partition and probability distribution given by formula (1). But the fuzzy partitions formed in this method will depend on the training data set available. The training data set is divided so that each fuzzy partition will have almost same number of data points associated with it. This will end up in obtaining asymmetric fuzzy sets, which will look like fuzzy sets shown in Fig. 2.

Fig. 2. Asymmetric, unevenly spaced triangular fuzzy sets

This will minimise the possibility of developing rules, which were giving equal probability, prob( f i ) towards each class for data points in test data set and will help to let us know the class of the data point to which it belongs. In turn it will increase the average accuracy of the decision model. If the training data is having data points clustered at few points of the universe of the attribute instead of scattered all over the universe or if there are gaps in the universe of the attribute without any data point, in those cases this modification will also help to minimize the number of fuzzy sets required and/or will let us know the maximum fuzzy sets possible (i.e. in such cases the algorithm will provide the exact number of fuzzy sets required).

3

Defuzzification

After having formed a decision tree, we can obtain probabilities for each data point towards each class of the target attribute. Now, it is another important task to defuzzify these probabilities and obtain a class of the data point to which it belongs to. Here, we have used the value at the maximum membership of the fuzzy set (i.e. the point of the fuzzy set at which the data points show 100% dependency towards that fuzzy set) for defuzzification. This can be easily justified with an example as below;

Asymmetric Triangular Fuzzy Sets for Classification Models

367

Let us assume we have an attribute with universe from 0 to 80 and five fuzzy sets on this attribute as shown in Fig. 2. If we have a data point having value 30 on this attribute then after fuzzification we get membership of 70% on the second fuzzy set and 30% membership on the third fuzzy set as shown in Fig. 3.

Fig. 3. Example diagram for defuzzification

If we defuzzify this point using the values at the maximum membership of the fuzzy sets, then we will get: • •

Second fuzzy set 15 (value at maximum membership) × 0.7 = 10.50, and Third fuzzy set 65 (value at maximum membership) × 0.3 = 19.50

i.e. we get back the total 10.50 + 19.50 = 30.00, which is exactly as the value at the initial point before fuzzification. Thus the final output value is given by n

V=

∑α

v

i i

(2)

i =1

where ‘ n ' is the number of fuzzy sets each associated with probability ‘ α i ' and ‘ vi ' is the value at maximum membership of each fuzzy set. Whereas the use of any other value for defuzzification, like, if we use the mean value of the fuzzy set then the value obtained after defuzzification will not be same to the value before fuzzification.

4

Results and Discussion

4.1

Example 1

Consider an example having six attributes, one of which is a categorical, noncontinuous attribute with four classes and five are non-categorical, continuous attributes. Universes for all continuous attributes are shown in Table 1.

368

J. F. Baldwin and Sachin B. Karale Table 1. Universes for continuous attributes

Non-categorical attribute 1 2 3 4 5

Universe 10 - 100 0 - 550 3 - 15 0 - 800 0 – 1000

A training data set of 500 data points is generated such that the first and the fourth non-categorical attributes have a gap on their universe, where no data point is available. For this example the gap is adjusted so that there is one fuzzy set in symmetrical fuzzy partitioning on each of the first and the fourth attributes without any data point belong to them. For example, in Fig. 4, symmetric triangular partitioning for attribute one is shown, where no data point associated with third fuzzy set.

Fig. 4. Symmetric fuzzy partitions on the first attribute

Fig. 5. Asymmetric fuzzy partitions on the first attribute

On use of asymmetric fuzzy sets, the partitioning is such that each fuzzy partition has almost same number of data points, as shown in Fig. 5. Results obtained for both symmetric and asymmetric approaches are shown in Table 2a. Test data set of 500 data points is generated which contains the data points scattered all over the universe of the attributes instead of clustered. We can see from Table 2a that, on use of asymmetric fuzzy sets the percentage accuracy is significantly higher than that on the use of symmetric fuzzy sets for both training and test data sets. In this problem, the use of four symmetric fuzzy sets avoid forming empty fuzzy partitions and gives more accuracy than the accuracy obtained by the use of five fuzzy sets. But still the accuracy cannot be improved up to that obtained by asymmetric fuzzy sets.

Asymmetric Triangular Fuzzy Sets for Classification Models

369

Table 2a. Comparison for use of symmetric and asymmetric fuzzy sets

Fuzzy Sets

4 5 6 7 4.2

Symmetric Triangular Fuzzy Sets %Accuracy %Accuracy on Training on Test Data Set Data Set 54.56 67.87 55.45 47.51 55.91 51.83 65.17 36.87

Asymmetric Triangular Fuzzy Sets %Accuracy %Accuracy on Training on Test Data Set Data Set 83.60 74.20 85.37 71.75 88.84 75.90 91.35 77.12

Example 2

In this example also the training data set of 500 data points with five non categorical attributes with universes as shown in Table 1 and one categorical attribute with four classes is used. This time the training data set is generated so that the gap in the first example is reduced to 50% on third fuzzy partition for the first attribute, as shown in Fig. 6. Even on a reduction in the gap, we can see from Table 2b that the results obtained by the use of asymmetric fuzzy sets are better than those obtained by the use of symmetric fuzzy sets. The test data set used in this case is same as that used in Example 1.Fig. 7 shows the fuzzy partitioning on use of asymmetric fuzzy sets with reduced gap.

Fig. 6. Symmetric fuzzy partitions on the first attribute with reduced gap

Fig. 7. Asymmetric fuzzy partitions on the first attribute with reduced gap

370

J. F. Baldwin and Sachin B. Karale

Table 2b. Comparison for use of symmetric and asymmetric fuzzy sets with reduced gap

Fuzzy Sets

4 5 6 7

5

Symmetric Triangular Fuzzy Sets %Accuracy %Accuracy on Training on Test Data Set Data Set 63.93 69.68 57.25 66.58 77.09 81.93 62.66 69.08

Asymmetric Triangular Fuzzy Sets %Accuracy %Accuracy on Training on Test Data Set Data Set 83.10 82.77 84.64 86.11 87.03 83.86 86.92 84.13

Conclusions

The new attempt made to use Fuzzy ID3 algorithm generate a decision tree which can give same or improved results compared to the use of symmetric fuzzy sets in Fuzzy ID3 algorithm for the real-world classification problems. The use of asymmetric, unevenly spaced triangular fuzzy sets makes Fuzzy ID3 more efficient and increases the reliability of the algorithm for variety of data. This approach of fuzzy partitioning for mass assignment ID3 can improve the results either by reducing the number of fuzzy sets or by increasing the accuracy, depending on the problem.

References [1] [2] [3] [4]

Quinlan, J.R.: C4.5: Programs for Machine Learning. San Mateo, Morgan Kaufmann Publishers (1993) Baldwin, J.F., Lawry, J., Martin, T. P.: A Mass Assignment Based ID3 Algorithm for Decision Tree Induction. International Journal of Intelligent Systems, 12 (1997) 523-552 Fayyad, U.M., Irani, K.B.: On the handling of continuous-valued attributes in decision tree generation. Mach. Learn., 8 (1992) 87-102 Baldwin, J.F., Martin, T.P., Pilsworth, B.W.: FRIL- Fuzzy and Evidential Reasoning in A.I. Research Studies Press, Wiley, New York (1995)

Machine Learning to Detect Intrusion Strategies Steve Moyle and John Heasman Oxford University Computing Laboratory Wolfson Building, Parks Road, Oxford, OX1 3QD, England

Abstract. Intrusion detection is the identification of potential breaches in computer security policy. The objective of an attacker is often to gain access to a system that they are not authorized to use. The attacker achieves this by exploiting a (known) software vulnerability by sending the system a particular input. Current intrusion detection systems examine input for syntactic signatures of known intrusions. This work demonstrates that logic programming is a suitable formalism for specifying the semantics of attacks. Logic programs can then be used as a means of detecting attacks in previously unseen inputs. Furthermore the machine learning approach provided by Inductive Logic Programming can be used to induce detection clauses from examples of attacks. Experiments of learning ten different attack strategies to exploit one particular vulnerability demonstrate that accurate detection rules can be generated from very few attack examples.

1

Introduction

The importance of detecting intrusions in computer based systems continues to grow. In early 2003 the Slammer worm crippled the Internet taking only 10 minutes to spread across the world. It contained a simple, fast scanner to find vulnerable machines in a small worm with a total size of only 376 bytes. Fortunately the bug did not contain a malicious payload. Intrusion detection is the automatic identification of potential breaches in computer security policy. The objective of an attacker is often to gain access to a system that they are not authorized to use. The attacker achieves this by exploiting a (known) software vulnerability by sending the system a particular input. In this work the buffer over-flow vulnerability is studied, and its technical basis described (in section 1). This is followed by the description of a semantic intrusion detection framework in section 2. Section 3 describes an experiment to use Inductive Logic Programming techniques to automatically produce intrusion detection rules from examples of intrusions. These results are discussed in the final section of the paper. 1.1

The Buffer Overflow Attack

Buffer overflow attacks are a type of security vulnerability relating to incomplete input validation. The circumstances in which a buffer overflow can occur are well V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 371–378, 2003. c Springer-Verlag Berlin Heidelberg 2003


372

Steve Moyle and John Heasman

understood and wholly avoidable if the software is developed according to one of many secure programming checklists [3, ch. 23].1 From a technical perspective the concepts behind exploiting a buffer overflow vulnerability are simple but the implementation details are subtle. They are discussed thoroughly from the view of the attacker in the seminal paper “Smashing the Stack for Fun and Profit” [1]. Sufficient background and implementation details are reproduced here so that the reader may understand how such attacks can be detected. Buffer overflow attacks exploit a lack of bounds checking on a particular user-supplied field within a computer software service that performs some transaction. The attacker supplies an overly large input transaction to the service. In the course of processing the transaction, the service copies the field into a buffer of a fixed size.2 As a result, memory located immediately after the buffer is overwritten with the remainder of the attacker’s input. The memory that is overwritten may have held other data structures, the contents of registers, or, assuming the system is executing a procedure, the return address from which program execution should continue once the procedure completes. It is the return address that is of most interest to the attacker. The function that copies data from one location to another without regard for the data size is referred to in the sequel as simply the dangerous function. Before the service executes the procedure containing the function call that leads to the overflow, the return address is pushed onto the stack so that program execution can continue from the correct place once the procedure is complete. Information about the processor’s registers is then pushed onto the stack so that the original state can be restored. Space for variables local to the procedure is then allocated. The attacker aims to overwrite the return address thus controlling the flow of the program. A malicious transaction input of a buffer overflow attack consists of several components as shown in Fig. 1. The dangerous function call takes the attacker’s input and overwrites the intended buffer in the following order: 1) the space allocated for the procedure’s local variables and the register save area, and 2) the return address. The purpose of the sequence of idle instructions at the beginning of the exploit code stream is to idle the processor. Idle instructions are used to reduce the uncertainty of the attacker having to guess the correct return address which must be hard coded in the attack stream. By including a sequence of idle instructions at the start of the exploit code, the return address need only be set to an address within this sequence which greatly improves the chances of an exploit succeeding. This sequence of idle instructions is often referred to as a NOP sledge. NOP is the opcode that represents a no operation instruction within the CPU. The attacker aims to set the return address to a location within the idle sequence so that his exploit code is executed correctly. He supplies multiple copies of his desired return address to compensate for the uncertainty of where 1 2

This does not demean the target of this work – although preventable – buffer overflow vulnerabilities are still prolific in computer software! The size of the buffer is smaller than that necessary to store the entire input provided by the attacker.

Machine Learning to Detect Intrusion Strategies

373

Arbitrary Sequence of idle Exploit code Multiple copies of data instructions return address

Fig. 1. attack

Components of a malicious transaction containing a buffer overflow

the actual return address is located. Once the procedure has completed, values from the (overwritten) register save area are popped off the stack back into system registers and program execution continues from the new return address – now executing the attacker’s code. For the attacker to correctly build the malicious field within the transaction, he must know some basic properties of the target system, including the following: – The processor architecture of the target host as machine code instructions have different values in different processors. – The operating system the target is running as each operating system has a different kernel and different methods of changing between user and kernel mode. – The stack start address for each process which is fixed for each operating system/kernel. – The approximate location of the stack pointer. This is a guess at how many bytes have been pushed onto the stack prior to the attack. This information together with the stack start address allow the attacker to correctly calculate a return address that points back into the idle sequence at the beginning of the exploit code. The work described in the sequel is limited to the discussion of buffer overflow exploits for the Intel x86 family of processors running the Linux operating system. These are only pragmatic constraints. The work is easily extended to different processor architectures, operating systems, and some vulnerabilities. 1.2

An IDS for Recognizing Buffer Overflow Attacks

An attacker must provide a stream of bytes, that, when executed after being stored in an overflowed buffer environment, will gain him control of the CPU, which will in-turn, perform some task for him (say execute a shell on a Unix system). When crafting a string of bytes the attacker is faced with many alternatives, given the constraints that he must work within (e.g. the byte stream must be approximately the size of the vulnerable buffer). These alternatives are both at the strategic and implementation levels. At the strategic level, the attacker must decide in what order the tasks will be performed by the attack code. For example, will the address of the command to the shell be calculated and stored before the string that represents the shell type to be called? Having decided on the components that make up the strategy, at the implementation level the attacker has numerous alternative encodings to choose from. Which register, for example, should the null byte be stored in? It

374

Steve Moyle and John Heasman

is clear that for any given attack strategy, there are vast numbers of alternative attack byte code streams that will, when executed, enact the strategy. This also means that simple syntax checking intrusion detection systems can easily be thwarted even by attackers using the same strategy that the syntactic IDS has been programmed to detect. A semantic Intrusion Detection System (IDS) was developed as a Prolog program in [4], which was further used for the basis of this work. The IDS program includes a dis-assembler that encodes the semantics of Intel x86 class processor byte codes. The IDS includes a byte stream generator that utilizes the same dis-assembler components. This byte stream generator produces a stream of bytes by producing a randomized implementation of the chosen intrusion strategy. This stream of bytes is then compiled into a dummy executable which simulates the process of the byte stream being called. This simply emulates the situation of the buffer having already been over-flowed, and not the process of over flowing it per se. Each attack byte stream is verified in its ability to open a shell (and hence its success for an attacker).

2

Inductive Logic Programming

Inductive Logic Programming (ILP) [6] is a field of machine learning involved in learning theories in the language of Prolog programs. ILP has demonstrated that such a powerful description language (First Order Predicate Calculus) can be applied to a large range of domains including natural language induction [2], problems in bio-informatics [7], and robot programming [5]. This work is closet to that of natural language programming in that it is concerned with inducing context dependent parsers from examples of “sentences”. The general setting for ILP is as follows. Consider an existing partial theory (e.g. a Prolog program) B (known as Background knowledge), from which one is not able to derive (or predict) the truth of an externally observed fact. Typically these facts are known as examples3 , E. This situation is often presented as B
|= E + . The objective of the ILP system is to produce an extra Prolog program H known as a theory, that, when combined with the background knowledge enables all the examples to be predicted. It is this new theory that is the knowledge discovered by the ILP system. B ∪ H |= E In the context of the semantic intrusion detection system, the basic assembler of the CPU architecture and the general semantic parser program, BIDS make up the background knowledge. The examples are the byte streams known to produce a successful attack. Consider the following single example of e+ 1 , where the attack is represented as a list of bytes in Prolog. 3

In general, the examples in ILP can be considered as either positive E + examples, or negative E − examples such that E = E + ∪ E − and E + ∩ E − = ∅.

Machine Learning to Detect Intrusion Strategies

e+ 1 =

375

attack([0x99,0x90,0x90,0xf8,0x9b,0x9b,0xeb,0x1c,0x5b,0x89,0x5b,0x08,0xba,0xff,0xff,

0xff,0xff,0x42,0x88,0x53,0x07,0x89,0x53,0x0c,0x31,0xc0,0xb0,0x0b,0x8d,0x4b,0x08,0x8d, 0x53,0x0c,0xcd,0x80,0xe8,0xdf,0xff,0xff,0xff,0x2f,0x62,0x69,0x6e,0x2f,0x73,0x68]).

Such an example is a single instance of the SSFFP buffer overflow exploit. It can be described (and parsed) by the following high level strategy (written in Prolog). shown below (which covers semantic variations in the exploit code taken from “Smashing the Stack for Fun and Profit” [1]). attack(A):- consume_idles(A,B), consume_load(B,C,Addr,Len), consume_place_addr(C,D,Addr,Len), consume_null_long(D,E,Null,Addr), consume_null_operation(E,F,Len,Null,Addr), consume_set_sys_call(F,G,Null,0x0b,Addr), consume_params(G,H,Addr,Null,Len), consume_system_call(H,[]).

Consider for example, the predicate consume idles(A,B) which transforms a sequence of bytes A into a suffix sequence of bytes B by removing the contiguous idle instructions from the front of the sequence. The above attack predicate is clearly much more powerful than a syntactic detection system in that it is capable detecting the whole class of byte streams for that particular strategy. However, it does not detect other strategies of attacks that have different structural properties. A simple solution is to add more rules that cover variations in the exploit code structure but this requires substantial human effort. If the expert is faced with new real world examples he believes can be expressed using the background information that is already encoded, he might ask “is there a way of automating this process?”. This is the motivation for the following section, which demonstrates that inducing updates to the semantic rule set is possible.

3

An Experiment to Induce Semantic Protection Rules

This section describes controlled experiments to (re-)learn semantic intrusion detection rules using the ILP system ALEPH [9]. The motivation for the experiment can be informally stated as testing the hypothesis that ILP techniques can recover reasonably accurate rules describing attack strategies. The basic method was to use correct and complete Prolog programs to randomly generate sequences of bytes that successfully exploit a buffer overflow4. Such byte sequences are considered as positive examples, E + , of attacks. The byte sequence generating predicates were then removed from the Prolog program leaving a common background program to be used as background knowledge, BIDS , to ALEPH. This background program was then used, along with positive examples of attack byte sequences, to induce rules similar to the byte sequence generating predicates. 3.1

Results

The learning curves for the recovery of each of the ten intrusion strategies are presented in Figure 2. Here, accuracy is measured as the proportion of test set 4

All such generated sequences, when supplied as part of a transaction with a vulnerable service, enabled the attacker to open a shell.

376

Steve Moyle and John Heasman ×  ♦

100


× +  ♦ 


× +  ♦ 


× +  ♦ 

+ 
× ♦ 

80 Average Accuracy (%) on hold-out × sets 60

+ 


+ ♦ 


40  3

9 27 Cardinality of training set (logarithmic scale)

81

Fig. 2. Learning curves for the recovery of ten different intrusion strategies from examples. Each line on the graph represents a different intrusion attack strategy (i.e. of the form attack(A) :- consume. . . ). Each point on the graph represents the average accuracy of 10 randomly chosen training and test sets

examples which are detected by the recovered rule. The proportion of the test set examples which are not detected by the recovered rule is the intrusion detection false negative rate. It can be seen from Figure 2 that, on average, for the attack strategies studied, relatively few examples of attacks are required to recover rules that detect a high proportion of attacks. In fact, for all of the ten studied attack strategies, only ten positive attack examples were required to recover a rules that were 100% accurate on the test set. This finding is discussed in the following section.

4

Discussion and Conclusions

The graphs in Figure 2 demonstrate that it is possible to induce semantic detection rules for buffer overflows attacks that are 100% accurate. Furthermore, these results can be obtained using, on average, a low number of examples for the induction. The need for few examples can be attributed to the relatively low size of the hypothesis space, which was a result of the high input and output connectedness of the background predicates. The average estimated size of the hypothesis space for the recovery of these attack strategies is 797 clauses and is comparable to the classic ILP test-bed problem, the Trains Srinivasan provides estimates of the hypothesis space for the trains problem and other common ILP data sets in [8].

Machine Learning to Detect Intrusion Strategies

377

Even the rules with lower accuracies can be understood by a domain expert and have some intrusion detection capabilities. This result demonstrates that ILP was well suited to the intrusion problem studied. It may well be suited to real world intrusion detection problems, provided the background theory is sufficient. These experiments indicate that the background information was highly accurate and relevant. The importance of relevancy has been studied by Srinivasan and King [10]. This application domain is very “crisp” and well defined. In a particular context, a byte op-code always performs a particular function. For example when the byte 0x9B is executed on an Intel x86 processor it always triggers the FWAIT functions to be performed. Such well defined, and cleanly specified domain is well suited to being represented in a logic program and susceptible to ILP. Contrast this with a biological domain where the activity of a particular system varies continuously with respect to many inputs – for example the concentration of certain chemicals. The work described has shown that it is possible to encode the operational semantics of attack transactions in a logical framework, which can then be used as background knowledge to an ILP system. Sophisticated attack strategies can then be represented in such a framework. In studying the particular application of a buffer overflow attack, it has been shown that ILP techniques can be used to learn rules that detect – with low false negative rates – attack strategies from relative few examples of attacks. Furthermore, only positive examples of attacks are necessary for the induction of attack strategy rules.

References [1] ”Aleph One”. Smashing The Stack For Fun And Profit. Phrack 49, 1996. 372, 375 [2] J. Cussens. Part-of-speech tagging using Progol. In Inductive Logic Programming: Proceedings of the 7th International Workshop (ILP-97), p. 93-108. Prague, 1997. Springer. 374 [3] S. Garfinkel, G. Spafford. Practical UNIX and Internet Security. Sebastopol, O’Reilly and Associates, 1996. 372 [4] S. A. Moyle and J. Heasman. Applying ILP to the Learning of intrusion strategies. Technical Report PRG-RR-02-03, Oxford University Computing Laboratory, University of Oxford, 2002. 374 [5] S. Moyle. Using Theory Completion to learn a Robot Navigation Control Program. In S. Matwin, editor, Proceedings of the 12th International Workshop on Inductive Logic Programming, 2002. 374 [6] S. Muggleton. Inverse Entailment and Progol. New Generation Computing, 13(3 and 4):245–286, 1995. 374 [7] S. H. Muggleton, C. H. Bryant, A. Srinivasan, A. Whittaker, S. Topp, and C. Rawlings. Are grammatical representations useful for learning from biological sequence data? – a case study. Computational Biology, 8(5):493-522, 2001. 374 [8] A. Srinivasan. A study of two probabilistic methods for searching large spaces with ILP. Technical Report PRG-TR-16-00, Oxford University Computing Laboratory, University of Oxford, 2000. 376

378

Steve Moyle and John Heasman

[9] A. Srinivasan. The Aleph Manual. http://web.comlab.ox.ac.uk/oucl/research/ ~areas/machlearn/Aleph/, 2001. 375 [10] A. Srinivasan, R. D. King. An empirical study of the use of relevance information in Inductive Logic Programming. Technical Report PRG-RR-01-19, Oxford University Computing Laboratory, University of Oxford, 2001. 377

On the Benchmarking of Multiobjective Optimization Algorithm Mario K¨ oppen Fraunhofer IPK Dept. Security and Inspection Technologies, Pascalstr. 8-9, 10587 Berlin, Germany [email protected]

Abstract. The ”No Free Lunch” (NFL) theorems state that in average each algorithm has the same performance, when no a priori knowledge of single-objective cost function f is assumed. This paper extends the NFL theorems to the case of multi-objective optimization. Further it is shown that even in cases of a priori knowledge, when the performance measure is related to the set of extrema points sampled so far, the NFL theorems still hold. However, a procedure for obtaining function-dependent algorithm performance can be constructed, the so-called tournament performance, which is able to gain different performance measures for different multiobjective algorithms.

1

Introduction

The ”No Free Lunch” (NFL) theorems state the equal average performance of any optimization algorithm, when measured against the set of all possible cost functions and if no domain knowledge of the cost function is assumed [3]. Usually, the NFL theorem is considered in a context of design of algorithms, especially it became well-known in the scope of evolutionary computation. However, the NFL theorem has also some other facettes, one of which is the major concern of this paper. So, the NFL theorem can also be seen as stating the impossibility to obtain a concise mathematical definition of algorithm performance. In this context, this paper considers multi-objective optimization and how the NFL theorems apply in this field. After recalling some basic definitions of multi-objective optimization in section 2, esp. the concept of Pareto front, the standard NFL theorem is proven for the multi-objective case in section 3. Then, the proof is extended to the case where sampling of extrema is also involved in the performance measure in section 4, proving that there is no gain in using such a measure. Only the case that two algorithms are compared directly give rise to so-called tournament performance and a heuristic procedure to measure algorithm performance. This will be presented in section 5.

2

Basic Definitions

In multi-objective optimization, optimization goal is given by more than one objective to be extreme [1]. Formally, given a domain as subset of Rn , there V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 379–385, 2003. c Springer-Verlag Berlin Heidelberg 2003


380

Mario K¨ oppen

are assigned m functions f1 (x1 , . . . , xn ), . . . , fm (x1 , . . . , xn ). Usually, there is not a single optimum but rather the so-called Pareto set of non-dominated solutions: For two vectors a and b it is said that a (Pareto-)dominates b, when each component of a is less or equal to the corresponding component of b, and at least one component is smaller: a >D b ←→ ∀i(ai ≤ bi ) ∧ ∃k(ak < bk ).

(1)

Note that in a similar manner Pareto dominance can be related to ”>”-relation. The subset of all vectors of a set M of vectors, which are not dominated by any other vector of M is the Pareto set (also Pareto front) P F . The Pareto set for univariate data (single objective) contains just the maximum of the data. The task of multi-objective optimization algorithm is to sample points of the Pareto front. A second instantiation (often called decision maker) is needed to further select from the Pareto front.

3

NFL-Theorem for Multi-objective Optimization Algorithms

A slight modification extends the proof of the single-objective NFL theorems given in [2] to the multi-objective case. Be X a finite set and Y a set of k finite domains Yi with i = 1, . . . , k. Then we consider the set of all sets of k cost functions fi : X → Yi with i = 1, . . . , k, or f : X → Y for simplicity. Let m be a non-negative integer < |X|. Define dm as a set {(dxm (i), dym (i) = = dxm (j). (f (dxm (i)))}, i = 1, . . . , m where dxm (i) ∈ X ∀ i and ∀ i, j, dxm (i)  Now consider a deterministic search algorithm a which assign to every possible dm an element of X \ dxm (see fig. 1): dxm+1 (m + 1) = a[dm ]  ∈ {dxm }.

(2)

Fig. 1. A deterministic algorithm derives next sampling point dxm+1 (m + 1) from the outcome of foregoing sampling dm

On the Benchmarking of Multiobjective Optimization Algorithm

381

Define Y (f, m, a) to be the sequence of m Y values produced by m successive applications of the algorithm a to f . Let δ(., .) be the Kronecker delta function that equals 1 if its arguments are identical, 0 otherwise. Then the following holds: Lemma 1. For any algorithm a and any dym ,


δ(dym , Y (f, m, a)) =

k 

|Yi ||X|−m .

i=1

f

Proof. Consider all cost functions f+ for which δ(dym , Y (f+ , m, a)) takes the value 1, 2 asf. of the sequence dym : i) f+ (a(∅)) = dym (1) ii) f+ (a[dm (1)]) = dym (2) iii) f+ (a[dm (1), dm (2)]) = dym (3) ... where dm (j) ≡ (dxm (j), dym (j)). So the value of f+ is fixed for exactly m distinct elements from X. For the remaining |X| − m elements from  X, the corresponding value  of f+ can be assigned freely. Hence, out of the i |Yi ||X| separate f , exactly i |Yi ||X|−m will result in a summand of 1 and all others will be 0. Then, we can continue with the proof of NFL theorem in multi-objective case. Take any performance measure c(.), mapping sets dym to real numbers. Theorem 1. For any two deterministic algorithms a and b, any performance value K ∈ R, and any c(.),

δ(K, c(Y (f, m, a))) = δ(K, c(Y (f, m, b))). f

f

dym

Proof. Since more than one may give the same value of the performance measure K, for each K the l.h.s. is expanded over all those possibilities:
δ(K, c(Y (f, m, a))) = f
= δ(K, c(dym ))δ(dym , Y (f, m, a)) (3) m f,dy m ∈Y

=




m dy m ∈Y

=


m dy m ∈Y

=

k  i=1

δ(K, c(dym ))




δ(dym , Y (f, m, a))

f

δ(K, c(dym ))

|Yi ||X|−m

k 

|Yi ||X|−m (by Lemma 1)

i=1


m dy m ∈Y

δ(K, c(dym ))

The last expression does not depend on a but only on the definition of c(.).

(4)

382

4

Mario K¨ oppen

Benchmarking Measures

The formal proof of the NFL theorems assumes no a priori knowledge of the function f . This can be easily seen in the proof of Theorem 1, when the expansion over f is made (line 3 of the proof): it is implicitely assumed that the performance measure c(.) does not depend on f . There are performance measures depending on f , for which Theorem 1 does not hold and that can be easily constructed (as e.g. derived from the requirement to scan (x, y) pairs in a given order). This is a reasonable assumption for evaluating an algorithm a. Domain knowledge of f could result in algorithm a somehow designed in a manner to show increased performance on some benchmark problems. However, common procedure to evaluate algorithms is to apply them onto a set of so-called ”benchmark problems.” This also holds in the multi-objective case. From a benchmark function f , usually analytic properties (esp. the extrema points) are given in advance. In [1], an extensive suite of such benchmark problems is proposed, in order to gain understanding of abilities of multi-objective optimization algorithms. So, for each benchmark problem, a description of the Pareto front of the problem is provided. The task given to a multi-objective optimization algorithm is to sample as many points from the Pareto front as possible. To name it here again: clearly, such a performance measure is related to a priori of f itself. NFL theorems given with Theorem 1 do not cover this case. However, in the following, it will be shown that NFL theorems even apply in such a case. It is based on the following lemma: Lemma 2. For any algorithm a it holds  {a ◦ f } |Y = Y |X| f

A given algorithm a applied to any f gives a sequence of values from Y . The union of all those sequences will be the set of all possible sequences of |X| elements chosen from Y . Or, in other words: each algorithm, applied to all possible f will give a permutation of the set of all possible sequences, with each sequence appearing exactly once. Proof. Assume that for two functions f1 and f2 algorithm a will give the same sequence of y-values (y1 , y2 , . . . , y|X| ). This also means that the two corresponding x-margins are permutations of X. Via induction we show that then follows f1 = f2 . Verification. Since we are considering deterministic algorithms, the choice of the first element x1 is fixed for an algorithm a (all further choices for x values are functions of the foregoing samplings). So, both f1 and f2 map x1 to y1 . Step. Assume f1 (xi ) = f2 (xi ) for i = 1, . . . , k (and k < |X|). Then according to eq. 2, algorithm a will compute the same dxk+1 (k +1) since this computation only depends on the sequence dm that is equal for f1 and f2 by proposition. Since the y-margins are also equal in the position (k + 1), for both f1 and f2 xk+1 = dxk+1 (k + 1) is mapped onto yk+1 .

On the Benchmarking of Multiobjective Optimization Algorithm

383

Table 1. Performance measure Pareto sampling after two steps in the example case Y = {0, 1}2 and |X| = 3 y1 y2 00 00 00 00 00 00 00 00 00 01 00 01 00 01 00 01 00 10 00 10 00 10 00 10 00 11 00 11 00 11 00 11 Sum

y3 00 01 10 11 00 01 10 11 00 01 10 11 00 01 10 11

P F c(2) y1 y2 y3 P F c(2) y1 y2 y3 P F c(2) y1 y2 y3 P F c(2) 00 1 01 00 00 00 1 10 00 00 00 1 11 00 00 00 1 00 1 01 00 01 00 1 10 00 01 00 1 11 00 01 00 1 00 1 01 00 10 00 1 10 00 10 00 1 11 00 10 00 1 00 1 01 00 11 00 1 10 00 11 00 1 11 00 11 00 1 00 1 01 01 00 00 0 10 01 00 00 0 11 01 00 00 0 00 1 01 01 01 01 1 10 01 01 01, 10 2 11 01 01 01 1 00 1 01 01 10 01, 10 1 10 01 10 01, 10 2 11 01 10 01, 10 1 00 1 01 01 11 01 1 10 01 11 01, 10 2 11 01 11 01 1 00 1 01 10 00 00 0 10 10 00 00 0 11 10 00 00 0 00 1 01 10 01 01, 10 2 10 10 01 01, 10 1 11 10 01 01, 10 1 00 1 01 10 10 01, 10 2 10 10 10 10 1 11 10 10 10 1 00 1 01 10 11 01, 10 2 10 10 11 10 1 11 10 11 10 1 00 1 01 11 00 00 0 10 11 00 00 0 11 11 00 00 0 00 1 01 11 01 01 1 10 11 01 01, 10 1 11 11 01 01 0 00 1 01 11 10 01, 10 1 10 11 10 10 1 11 11 10 10 0 00 1 01 11 11 01 1 10 11 11 10 1 11 11 11 11 1 16 16 16 11 Average Performance 59/64 ∼ 0.92

This completes the proof. It has to be noted that not each permutation of Y |X| can be accessed by an algorithm (what can be easily seen from the fact that there are much more permutations than possible algorithm specifications). Following this lemma, all performance calculations that are independent of the sorting of the elements of Y |X| will give the same average performance, independent on a. Sampling of Pareto front elements after m algorithm steps is an example for such a measure. For illustration, table 1 gives these compuations for the simple case Y = {0, 1} × {0, 1} and X = {a, b, c}. Table 1 displays all possible functions f : X → Y and the corresponding Pareto set P F . The column c(2) shows the number of Pareto set elements that have already been sampled after two steps. The computation of the average performance does not depend on the order in which the functions are listed, thus each algorithm a will have the same average performance cav (2) = 59/64 . A remark on the single-objective case: in the foregoing discussion, multiobjectivity of f was not referenced explicitely. Hence, the discussion holds also for the ”single-objective” version, in which an algorithm is judged by its ability to find extrema points within a fixed number of steps. The NFL theorems also appy to this case.

5

Tournament Performance

Among the selection of function-dependent performance measures, one should be pointed out in the rest of this paper. For obtaining ”position-dependence” of the measure on a single function f , the value obtained by applying a base algorithm A is taken. Algorithm a now runs competively against A. In such

384

Mario K¨ oppen

Table 2. Tournament performance of algorithm a after two steps f (a) f (b) 01 00 01 00 01 00 01 00 01 01 01 01 01 01 01 01 01 10 01 10 01 10 01 10 01 11 01 11 01 11 01 11 Total

f (c) 00 01 10 11 00 01 10 11 00 01 10 11 00 01 10 11

P F (dy 2 (A, f )) 00 00 00 00 01 01 01 01 01, 10 01, 10 01, 10 01, 10 01 01 01 01

P F (dy 2 (a, f )) 00 01 01, 10 01 00 01 01, 10 01 00 01 01, 10 01 00 01 01, 10 01

y P F (dy 2 (A, f )) \ P F (d2 (a, f )) 00 00 00 00 01 01 01 01, 10 01, 10 01, 10 01 01 01

|.| 0 0 0 0 -1 0 0 0 -2 0 0 0 -1 0 0 0 -4

a case, the NFL theorem does not hold. For seeing this, it is sufficient to provide a counterexample. Before, we define the difference of two Pareto sets P Fa and P Fb as the set P Fa \ P Fb , in which in P Fa all elements are removed, which are dominated by any element of P Fb . Be Y = {0, 1}2 and X = {a, b, c}, as in the foregoing example. Now algorithm A is as follows: Algorithm A: Take the dxm in the following order: a, b, c. And be Algorithm a: Take a as first choice. If f (a) = {0, 1} then select c as next point, otherwise b. Table 2 shows the essential part of all possible functions f , in which algorithms A and a behave different. For ”measuring” the performance of a at step m, we compute the size of the Pareto set difference |P F (dym (A, f )) \ P F (dym (a, f ))| .

(5)

and take the average over all possible f as average performance. For functions that do not start with f (a) = {0, 1}, both algorithms are identical, so in these cases  = 0. For functions mapping x-value a onto {0, 1}, we see  = −4. Now taking other algorithms: Algorithm b: Take a as first choice. If f (a) = {1, 0} then select c as next point, otherwise b. Algorithm c: Take a as first choice. If f (a) = {1, 1} then select c as next point, otherwise b.

On the Benchmarking of Multiobjective Optimization Algorithm

385

For b and c, a similar computation gives  = −4 and  = −5 respectively. In this sense ”strongest” algorithm (i.e. in comparison to A) is to sample in the order a, c, b with a performance  = −13. It should be noted that this performance measure is also applicable to the single-objective case. However, more studies on this measure have to be performed. Based on this, a heuristic procedure to measure performance of multi-objective optimization algorithm a might look like: 1. Let algorithm a run for k evaluations of cost function f and take the set M1 of non-dominated points from Y obtained by the algorithm. 2. Select k random domain values of X and compute the Pareto set M2 of the corresponding Y values. 3. Compute the set M3 of elements of M2 that are not dominated by any element of M1 . The relation of |M1 | to |M3 | gives a measure how algorithm a performs against random search.

References [1] Carlos A. Coello Coello, David A. Van Veldhuizen, Gary B. Lamont. Evolutionary Algorithms for Solving Multi-Objective Problems. Kluwer Academic Publishers, 2002. 379, 382 [2] Mario K¨ oppen, David H. Wolpert and William G. Macready, “Remarks on a recent paper on the ”no free lunch” theorems,” IEEE Transactions on Evolutionary Computation, vol. 5, no. 3, pp. 295–296, 2001. 380 [3] David H. Wolpert and William G. Macready, “No free lunch theorems for optimization,” IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, 1997. 379

Multicategory Incremental Proximal Support Vector Classifiers Amund Tveit and Magnus Lie Hetland Department of Computer and Information Science Norwegian University of Science and Technology, N-7491 Trondheim, Norway {amundt,mlh}@idi.ntnu.no

Abstract. Support Vector Machines (SVMs) are an efficient data mining approach for classification, clustering and time series analysis. In recent years, a tremendous growth in the amount of data gathered has changed the focus of SVM classifier algorithms from providing accurate results to enabling incremental (and decremental) learning with new data (or unlearning old data) without the need for computationally costly retraining with the old data. In this paper we propose an efficient algorithm for multicategory classification with the incremental proximal SVM introduced by Fung and Mangasarian.

1

Introduction

Support Vector Machines (SVMs) are an efficient data mining approach for classification, clustering and time series analysis [1, 2, 3]. In recent years, a tremendous growth in the amount of data gathered (for example, in e-commerce and intrusion detection systems) has changed the focus of SVM classifier algorithms from providing accurate results to enabling incremental (and decremental) learning with new data (or unlearning old data) without the need for computationally costly retraining with the old data. Fung and Mangasarian [4] introduced the Incremental and Decremental Linear Proximal Support Vector Machine (PSVM) for binary classification and showed that it could be trained extremely efficiently, with one billion examples (500 increments of two million examples) in two hours and twenty-six minutes on relatively low-end hardware (400 MHz Pentium II). In this paper we propose an efficient algorithm based on memoization, in order to support Multicategory Classification for the Incremental PSVM.

2

Background Theory

The standard binary SVM classification problem with soft margin (allowing some errors) is shown visually in Fig. 1(a) and as a constrained quadratic programming problem in (1). Intuitively, the problem is to maximize the margin between the solid planes and at the same time permit as few errors as possible, errors being positive class points on the negative side (of the solid line) or vice versa. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 386–392, 2003. c Springer-Verlag Berlin Heidelberg 2003


Multicategory Incremental Proximal Support Vector Classifiers

(a) Standard SVM classifier

387

(b) Proximal SVM classifier

Fig. 1. SVM and PSVM

min

(w,γ,y)∈Rn+1+m

{ve y + 12 w w} (1)

s.t. D(Aw − eγ) + y ≥ e y≥0 A ∈ Rm×n , D ∈ {−1, +1}m×1 , e = 1m×1

Fung and Mangasarian [5] replaced the inequality constraint in (1) with an equality constraint. This changed the binary classification problem, because the points in Fig. 1(b) are no longer bounded by the planes, but are clustered around them. By solving the equation for y and inserting the result into the expression to be minimized, one gets the following unconstrained optimization problem: min

(w,γ)∈Rn+1+m

Setting ∇f =




f (w, γ) = ν2 D(Aw − eγ) − e2 + 12 (w w + γ 2 )

∂f ∂f ∂w , ∂γ



(2)

= 0 one gets:

−1      −1   I w A De A A + νI −A e   + E = E E De =   γ −e De −e A ν1 + m ν       B 

X

(3)

A−1

E = [A − e], E ∈ Rm×(n+1) Fung and Mangasarian [6] later showed that (3) can be rewritten to handle increments (E i , di ) and decrements (E d , dd ), as shown in (4). This decremental approach is based on time windows.

388

Amund Tveit and Magnus Lie Hetland

 X =  =

w γ



I + E  E + (E i ) E i − (E d ) E d ν

−1



E  d + (E i ) di − (E d ) dd



,

(4)

where d = De .

3

Incremental Proximal SVM for Multiple Classes

In the multicategorical classification case, the (incremental) class label vector di consists of mi numeric labels in the range {0, . . . , c − 1}, where c is the number of classes, as shown in (5).   w0 . . . wc−1 (5) X = = A−1 B γ0 . . . γc−1 3.1

The Naive Approach

In order to apply the proximal SVM classifier in a “one-against-the-rest” manner, the class labels must be transformed into vectors with +1 for the positive class and −1 for the rest of the classes, that is, Θ(cmi ) operations in total, and later Θ(cmi n) for calculating (E i ) d for each class. The latter (column) vectors are collected in a matrix B ∈ R(n+1)c . Because the training features represented 2 by E i are the same for all the classes, it is enough to calculate A ∈ R(n+1) once, giving Θ(mi (n + 1)2 + (n + 1)2 ) operations for calculating (E i ) E i and adding it to νI + E  E. The specifics are shown in shown in Algorithm 1. Theorem 1. The running time complexity of Algorithm 1 is Θ(cminc n). Proof. The conditional statement in lines 3–7 takes Θ(1) time and is performed minc times (inner loop, lines 2–8) per iteration of classId (outer loop, line 1–10). Calculation of the matrix-vector product B[classId, ] in line 9 takes Θ((n + 1)minc ) per iteration of classId. This gives a total running time of Θ(c · (minc + minc (n + 1))) = Θ(cminc n) .



Multicategory Incremental Proximal Support Vector Classifiers

389

Algorithm 1 calcB Naive(Einc , dinc ) Require: Einc ∈ Rminc x(n+1) , dinc ∈ {0, . . . , c − 1}minc and n, minc ∈ N Ensure: B ∈ R(n+1)xc 1: for all classId in {0, . . . , c − 1} do 2: for all idx in {0, . . . , minc − 1} do 3: if dinc [idx ] = classId then 4: dclassId [idx, ] = +1 5: else 6: dclassId [idx, ] = −1 7: end if 8: end for
dclassId 9: B[classId, ] = Einc 10: end for 11: return B

3.2

The Memoization Approach

The dclassId vectors, c in all, (in line 3 of Algorithm 1) are likely to be unbalanced, that is, have many more −1 values than +1 values. However, if there are more than two classes present in the increment di , the vectors will at least share one index position where the value is −1. With several classes present in the increment di , the matrix-vector products (E i ) dclassId actually perform duplicate calculations each time there exists two or more dclassId vectors that have −1 values in the same position. The basic idea for the memoization approach (Algorithm 3) is to only calculate the +1 positions for each vector dclassId by first creating a vector F = −[E.i j ] 

(a vector with the negated sum of E’s columns, equivalent to multiplying E i with a vector filled with −1) and then to calculate the dclassId vectors using F and only switching the −1 to a +1 by adding the row vector of E twice if the row in dclassId is equal to +1. In order to do this efficiently, an index of di for each class ID has to be created (Algorithm 2). Algorithm 2 buildClassMap(dinc ) Require: dinc ∈ {0, . . . , c − 1}minc and minc ∈ N 1: classMap = array of length c containing empty lists 2: for all idx = 0 to minc − 1 do 3: append idx to classMap[dinc [idx, ]] 4: end for 5: return classMap

Theorem 2. The running time complexity of Algorithm 2 is Θ(minc ).

390

Amund Tveit and Magnus Lie Hetland

Proof. Appending idx to a the tail of a linked list takes Θ(1) time, lookup of classMap[dinc [idx, ]] in the directly addressable arrays classMap and dinc also takes Θ(1) time, giving a total for line 3 of Θ(1) time per iteration of idx. idx is iterated minc times, giving a total of Θ(minc ) time.





Algorithm 3 calcB Memo(Einc , dinc , Einc Einc ) Require: Einc ∈ Rminc ×(n+1) , dinc ∈ {0, . . . , c − 1}minc and n, minc ∈ N Ensure: B ∈ R(n+1)xc , F ∈ R(n+1) 1: classMap = buildClassMap(dinc ) 2: for all classId in {0, . . . , c − 1} do  3: B[classId, ] = Einc Einc [n] 4: for all idx in classMap[classId, ] do  5: B[idx, classId, ] = B[idx, classId, ] + 2 · n j=0 Einc [idx , j] 6: end for 7: end for 8: return B

Theorem 3. The running time complexity of Algorithm 3 is Θ(n(c + minc ). Proof. Calculation of classMap (line 1) takes Θ(minc ) time (from Theorem 2). Line 3 takes Θ(n + 1) time per iteration of classId, giving a total of Θ(c(n + 1)). Because classMap provides a complete indexing (| c−1 u=0 classMap[u]| = minc ) of the class labels in d inc , and because there are no repeated occurrences of idx for c−1 different classId s ( u=0 classMap[u] = ∅), line 5 will run a total of minc times. This gives a total running time of Θ(minc + (n + 1)minc + c(n + 1) + minc ) = Θ(n(c + minc )) .

Corollary 1. Algorithms 1 and 3 calculate the same B if provided with the same input.

4

Empirical Results

In order to test and compare the computational performance of the incremental multicategory proximal SVMs with the naive and lazy algorithms, we have used three main types of data: 1. Forest cover type, 580012 training examples, 7 classes and 54 features (from UCI KDD Archive [7])

50

391

● ●

5

●

30

Seconds

●

Seconds

●

Naive Lazy 20

10

●

Naive Lazy

●

40

15

Multicategory Incremental Proximal Support Vector Classifiers

●

●

10

● ●

● ● ●

●●

0e+00

0

0

● ●

1e+05

2e+05

3e+05

4e+05

5e+05

● ●

● ●

200

Number of Examples

(a) Runtime vs examples (cover type)

●

400

●

●

●

●

600

800

1000

Number of Classes

(b) Runtime vs classes (synthetic)

Fig. 2. Computational performance: training time

2. Synthetic datasets with a large number of classes (up to 1000 classes) and 30 features 3. Synthetic dataset with a large number of examples (10 million), 10 features and 10 classes The results for the first two data sets are shown in Fig. 4; the average time from tenfold cross-validation is used. For the third data set, the average classifier training times were 18.62 s and 30.64 s with the lazy and naive algorithm, respectively (training time for 9 million examples, testing on 1 million). The incremental multicategory proximal SVM was been implemented in C++ using the CLapack and ATLAS libraries. The tests were run on an Athlon 1.53 GHz PC with 1 GB RAM running Red Hat Linux 2.4.18.

Acknowledgements We would like to thank Professor Mihhail Matskin and Professor Arne Halaas. This work is supported by the Norwegian Research Council in the framework of the Distributed Information Technology Systems (DITS) program, and the ElComAg project.

5

Conclusion and Future Work

We have introduced the multiclass incremental proximal SVM and shown a computational improvement for training the multiclass incremental proximal SVM, which works particularly well for classification problems with a large number of

392

Amund Tveit and Magnus Lie Hetland

classes. Another contribution is the implementation of the system (available on request). Future work includes applying the algorithm to demanding incremental classification problems, for example, web page prediction based on analysis of click streams or automatic text categorization. Algorithmic improvements that need to be done include (1) develop balancing mechanisms (in order to give hints for pivot elements to the applied linear system solver for reduction of numeric errors), (2) add support for decay coefficients for efficient decremental unlearning, (3) investigate the appropriateness of parallelized incremental proximal SVMs, (4) strengthen implementation with support for tuning set, kernels as well as one-against-one classifiers.

References [1] Burbidge, R., Buxton, B. F.: An introduction to support vector machines for data mining. In Sheppee, M., ed.: Keynote Papers, Young OR12, University of Nottingham, Operational Research Society, Operational Research Society (2001) 3–15 386 [2] Huang, J., Shao, X., Wechsler, H.: Face pose discrimination using support vector machines (svm). In: Proceedings of 14th Int’l Conf. on Pattern Recognition (ICPR’98), IEEE (1998) 154–156 386 [3] Muller, K. R., Smola, A. J., Ratsch, G., Scholkopf, B., Kohlmorgen, J., Vapnik, V.: Predicting time series with support vector machines. In: ICANN. (1997) 999–1004 386 [4] Fung, G., Mangasarian, O. L.: Incremental support vector machine classification. In Grossman, R., Mannila, H., Motwani, R., eds.: Proceedings of the Second SIAM International Conference on Data Mining, SIAM (2002) 247–260 386 [5] Fung, G., Mangasarian, O. L.: Multicategory Proximal Support Vector Classifiers. Submitted to Machine Learning Journal (2001) 387 [6] Schwefel, H. P., Wegener, I., Weinert, K., eds.: 8. Natural Computing. In: Advances in Computational Intelligence: Theory and Practice. Springer-Verlag (2002) 387 [7] Hettich, S., Bay, S. D.: The UCI KDD archive. http://kdd.ics.uci.edu (1999) 390

Arc Consistency for Dynamic CSPs Malek Mouhoub Department of Computer Science University of Regina, 3737 Waskana Parkway, Regina SK, Canada, S4S 0A2 [email protected]

Abstract. Constraint Satisfaction problems (CSPs) are a fundamental concept used in many real world applications such as interpreting a visual image, laying out a silicon chip, frequency assignment, scheduling, planning and molecular biology. A main challenge when designing a CSPbased system is the ability to deal with constraints in a dynamic and evolutive environment. We talk then about on line CSP-based systems capable of reacting, in an efficient way, to any new external information during the constraint resolution process. We propose in this paper a new algorithm capable of dealing with dynamic constraints at the arc consistency level of the resolution process. More precisely, we present a new dynamic arc consistency algorithm that has a better compromise between time and space than those algorithms proposed in the literature, in addition to the simplicity of its implementation. Experimental tests on randomly generated CSPs demonstrate the efficiency of our algorithm to deal with large size problems in a dynamic environment. Keywords: Constraint Satisfaction, Search, Dynamic Arc Consistency

1

Introduction

Constraint Satisfaction problems (CSPs) [1, 2] are a fundamental concept used in many real world applications such as interpreting a visual image, laying out a silicon chip, frequency assignment, scheduling, planning and molecular biology. This motivates the scientific community from artificial intelligence, operations research and discrete mathematics to develop different techniques to tackle problems of this kind. These techniques become more popular after they were incorporated into constraint programming languages [3]. A main challenge when designing a CSP-based system is the ability to deal with constraints in a dynamic and evolutive environment. We talk then about on line CSP-based systems capable of reacting, in an efficient way, to any new external information during the constraint resolution process. A CSP involves a list of variables defined on finite domains of values and a list of relations restricting the values that the variables can take. If the relations are binary we talk about binary CSP. Solving a CSP consists of finding an assignment of values to each variable such that all relations (or constraints) are satisfied. A CSP is known to be an NP-Hard problem. Indeed, looking for V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 393–400, 2003. c Springer-Verlag Berlin Heidelberg 2003


394

Malek Mouhoub

a possible solution to a CSP requires a backtrack search algorithm of exponential complexity in time1 . To overcome this difficulty in practice, local consistency techniques are used in a pre-processing phase to reduce the size of the search space before the backtrack search procedure. A k-consistency algorithm removes all inconsistencies involving all subsets of k variables belonging to N . The kconsistency problem is polynomial in time, O(N k ), where N is the number of variables. A k-consistency algorithm does not solve the constraint satisfaction problem, but simplifies it. Due to the incompleteness of constraint propagation, in the general case, search is necessary to solve a CSP problem, even to check if a single solution exists. When k = 2 we talk about arc consistency. An arc consistency algorithm transforms the network of constraints into an equivalent and simpler one by removing, from the domain of each variable, some values that cannot belong to any global solution. We propose in this paper a new technique capable of dealing with dynamic constraints at the arc consistency level. More precisely, we present a new dynamic arc consistency algorithm that has a better compromise between time and space than those algorithms proposed in the literature [4, 5, 6], in addition to the simplicity of its implementation. In order to evaluate the performance in time and memory space costs of the algorithm we propose, experimental tests on randomly generated CSPs have been performed. The results demonstrate the efficiency of our methods to deal with large size dynamic CSPs. The rest of the paper is organized as follows. In the next section we will present the dynamic arc consistency algorithms proposed in the literature. Our dynamic arc consistency algorithm is then presented in section 3. Theoretical comparison of our algorithm and those proposed in the literature is covered in this section. The experimental part of our work is presented in section 5. Finally, concluding remarks and possible perspectives are listed in section 6.

2

Dynamic Arc-Consistency Algorithms

2.1

Arc Consistency Algorithms

The key AC algorithm was developed by Mackworth[1] called AC-3 over twenty years ago and remains one of the easiest to implement and understand today. There have been many attempts to best its worst case time complexity of O(ed3 ) and though in theory these other algorithms (namely AC-4[7], AC-6[8] and AC7[9]) have better worst case time complexities, they are harder to implement. In fact, the AC-4 algorithm fares worse on average time complexity than the AC-3 algorithm[10]. It was not only until recently when Zhang and Yap[11]2 proposed 1

2

Note that some CSP problems can be solved in polynomial time. For example, if the constraint graph corresponding to the CSP has no loops, then the CSP can be solved in O(nd2 ) where n is the number of variables of the problem and d the domain size of the different variables. Another arc consistency algorithm (called AC-2001) based on the same idea as AC3.1 was proposed by Bessi`ere and R´egin [12]. We have chosen AC-3.1 for the simplicity of its implementation.

Arc Consistency for Dynamic CSPs

395

an improvement directly derived from the AC-3 algorithm into their algorithm AC-3.1. The worst case time complexity of AC-3 is bounded by O(ed3 ) [13] where e is the number of constraints and d is the domain size of the variables. In fact this complexity depends mainly on the way the arc consistency is enforced for each arc of the constraint graph. Indeed, if anytime a given arc (i, j) is revised, a support for each value from the domain of i is searched from scratch in the domain of j, then the worst case time complexity of AC-3 is O(ed3 ). Instead of a search from scratch, Zhang and Yap [11] proposed a new view that allows the search to resume from the point where it stopped in the previous revision of (i, j). By doing so the worst case time complexity of AC-3 is achieved in O(ed2 ). 2.2

Dynamic Arc Consistency Algorithms

The arc-consistency algorithms we have seen in the previous section can easily be adapted to update the variable domains incrementally when adding a new constraint. This simply consists of performing the arc consistency between the variables sharing the new constraint and propagate the change to the rest of the constraint network. However, the way the arc consistency algorithm has to proceed with constraint relaxation is more complex. Indeed, when a constraint is retracted the algorithm should be able to put back those values removed because of the relaxed constraint and propagate this change to the entire graph. Thus, traditional arc consistency algorithms have to be modified so that it will be able to find those values which need to be restored anytime a constraint is relaxed. Bessi`ere has proposed DnAC-4[4] which is an adaptation of AC-4[7] to deal with constraint relaxations. This algorithm stores a justification for each deleted value. These justifications are then used to determine the set of values that have been removed because of the relaxed constraint and so can process relaxations incrementally. DnAC-4 inherits the bad time and space complexity of AC-4. Indeed, comparing to AC-3 for example, AC-4 has a bad average time complexity[10]. The worst-case space complexity of DnAC-4 is O(ed2 + nd) (e, d and n are respectively the number of constraints, the domain size of the variables and the number of variables). To work out the drawback of AC-4 while keeping an optimal worst case complexity, Bessi`ere has proposed AC-6[8]. Debruyne has then proposed DnAC-6 adapting the idea of AC-6 for dynamic CSPs by using justifications similar to those of DnAC-4[5]. While keeping an optimal worst case time complexity (O(ed2 )), DnAC-6 has a lower space requirements (O(ed + nd)) than DnAC-4. To solve the problem of space complexity, Neveu and Berlandier proposed AC|DC[6]. AC|DC is based on AC-3 and does not require data structures for storing justifications. Thus, it has a very good space complexity (O(e + nd)) but is less efficient in time than DnAC-4. Indeed, with its O(ed3 ) worst case time complexity, it is not the algorithm of choice for large dynamic CSPs. Our goal here is to develop an algorithm that has a better compromise between running time and memory space than the above three algorithms. More precisely, our ambition is to have an algorithm with the O(ed2 ) worst case time

396

Malek Mouhoub

complexity of DnAC-6 but without the need of using complex and space expensive data structures to store the justifications. We have then decided to adapt the new algorithm proposed by Zhang and Yap[11] in order to deal with constraint relaxations. The details of the new dynamic arc consistency algorithm we propose that we call AC-3.1|DC, are presented in the next section. The basic idea that we took was to integrate the AC-3.1 into the AC|DC algorithm since that algorithm was based on AC-3. The problem with the AC|DC algorithm was that it relied solely on the AC-3 algorithm and did not keep support lists like DnAC4 or DnAC6 causing the restriction and relaxation of a constraint to be fairly time consuming. This is also the reason for its worst case time complexity of O(ed3 ). If AC-3.1 was integrated into the AC|DC algorithm, then by theory the worst case time complexity for a constraint restriction should be O(ed2 ). In addition to this, the worst case space complexity remains the same as the original AC|DC algorithm of O(e + nd). The more interesting question is whether this algorithm’s time complexity can remain the same during retractions. Following the same idea of AC|DC, the way our AC3.1|DC algorithm deals with relaxations is as follows (see pseudo-code of the algorithm in figure 3). For any retracted constraint (k, m) between the variables k and m, we perform the following three phases : 1. An estimation (over-estimation) of the set of values that have been removed because of the constraint (k, m) is first determined by looking for the values removed from the domains of k and m that have no support on (k, m). Indeed, those values already suppressed from the domain of k (resp m) and which do have a support on (k, m), do not need to be put back since they have been suppressed because of another constraint. This phase is handled by the procedure Propose. The over-estimated values are put in the array propagate list[k] (resp propagate list[m]). 2. The above set is then propagated to the other variables. In this phase, for each value (k, a) (resp (m, b)) added to the domain of k (resp m) we will look for those values removed from the domain of the variables adjacent to k (resp m) supported by (k, a) (resp (m, b)). These values will then be propagated to the adjacent variables. The array propagate list is used to contain the list of values to be propagated for each variable. After we propagate the values in propagate list[i] of a given variable i, these values are removed from the array propagate list and added to the array restore list in order to be added later to the domain of the variable i. This way we avoid propagating the values more than once. 3. Finally a filtering procedure (the function Filter) based on AC-3.1 is then performed to remove from the estimated set the values which are not arcconsistent with respect to the relaxed problem.

3

AC-3.1|DC

The worst case time complexity of the first phase is O(d2 ). AC-3.1 is applied in the third phase and thus the complexity is O(ed2 ). Since the values

Arc Consistency for Dynamic CSPs

397

in propagate list are propagated only once, then the complexity of the second phase is also O(ed2 ). Thus the overall complexity of the relaxation is O(ed2 ). In terms of space complexity, the arrays propagate list and restore list require O(nd). AC-3.1 requires an array storing the resume point for each variable value (in order to have O(ed2 ) time complexity). The space required by this array is O(nd) as well. If we add to this the O(e + nd) space requirement of the

Function Relax(k,m) 1. propagate list ← nil 2. Remove (k, m) from the set of constraints 3. Propose(k,m,propagate list) 4. Propose(m,k,propagate list) 5. restore list ← nil 6. Propagate(k,m,propagate list,restore list) 7. Filter(restore list) 8. for all i ∈ V do 9. domaini ← domaini ∪restore list[i] Function Propose(i,j,propagate list) 1. for all value a ∈ dom[i] − D[i] do 2. support ← false 3. for all b ∈ D[j] do 4. if ((i a),(j b)) is satisfied by (i,j) then 5. support ← true 6. exit 7. if support ← false then 8. propagate list[i] ← propagate list[i] ∪ {a} Function Propagate(k,m,propagate list,restore list) 1. L ← {k,m} 2. while L = nil do 3. i ← pop(L) 4. for all j such that (i,j) ∈ the set of constraints 5. S ← nil 6. for all b ∈ dom[j] − (d[j]∪restore list[j]∪propagate list[j]) do 7. for all a ∈ propagate list[i] do 8. if ((i a),(j b)) is satisfied by (i,j) then 9. S ← S ∪ {b} 10. exit 11. if S = nil do 12. L ← L ∪ {j} 13. propagate list[j] ← propagate list[j] ∪ S 14. restore list[i] ← restore list[i] ∪ propagate list[i] 15. propagate list[i] ← nil

Fig. 1. Pseudo code of the dynamic arc consistency algorithm

398

Malek Mouhoub

Table 1. Comparison in terms of time and memory costs of the four algorithms DnAC-4 DnAC-6 AC|DC AC-3.1|DC Space complexity O(ed2 + nd) O(ed + nd) O(e + nd) O(e + nd) Time complexity O(ed2 ) O(ed2 ) O(ed3 ) O(ed2 )

traditional AC-3 algorithm, the overall space requirement is O(e + nd) as well. Comparing to the three dynamic arc consistency algorithms we mentioned in the previous section, ours has a better compromise, in theory, between time and space costs as illustrated by table 1.

4

Experimentation

Theoretical comparison of the four dynamic arc consistency algorithms shows that AC3.1|DC has a better compromise between time and space costs than the other three algorithms. In order to see if the same conclusion can be said in practice we have performed comparative tests on dynamic CSPs randomly generated as we will show in the following subsection. The criteria used to compare the algorithms are the running time needed and the memory space required by each algorithm to achieve the arc consistency . The experiments are performed on a Sun Sparc 10 and all procedures are coded in C/C++. Given n the number of variables and d the domain size, each CSP instance is randomly obtained by genconstraints are then picked randomly erating n sets of d natural numbers. n(n−1) 2 from a set of arithmetic relations {=,  =, <, ≤, >, ≥, . . .}. The generated CSPs are characterized by their tightness, which can be measured, as shown in [14], as the fraction of all possible pairs of values from the domain of two variables that are not allowed by the constraint. Figure 2 shows the performance in time performed by each arc consistency algorithm to achieve the arc consistency in a dynamic environment, as follows. Starting from a CSP having n = 100 variables, d = 50

7

6

DnAC-4

Time (sec)

5

4

3

2 AC|DC

AC3.1|DC 1 DnAC-6 0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Tightness

Fig. 2. Comparative tests of the dynamic arc-consistency algorithms

Arc Consistency for Dynamic CSPs

399

Table 2. Comparative results in terms of memory cost n 500 500 500 500 500 500 500 500 500 500 500 d 100 90 80 70 60 50 40 30 20 10 5 DnAC6 343MB 336MB 278MB 243MB 203MB 160MB 130MB 98MB 67MB 37MB 23MB AC|DC3.1 55MB 51MB 45MB 42MB 36MB 32MB 26MB 16MB 16MB 12MB 10MB

and 0 constraints, restrictions are done by adding the relations from the random ) is obtained. AfterCSP until a complete graph (number of constraints= n(n−1) 2 wards, relaxations are performed until the graph is 50% constrained (number of ). These tests are performed on various degree of tightness constraints= n(n−1) 4 to determine if one type of problem, (over-constrained, middle-constrained or under-constrained) favored any of the algorithms. As we can easily see, the results provided by AC-3.1|DC fares better than that of AC|DC and DnAC-4 in all cases. Also AC-3.1|DC algorithm is comparable if not better than DnAC6 (that has the best running time of the three dynamic arc consistency algorithms) as can be seen in figure 2. Table 2 shows the comparative results of DnAC-6 and AC3.1|DC in terms of memory space. The tests are performed on randomly generated CSPs in the same way as for the previous ones. As we can easily see, AC3.1|DC requires much less memory space than DnAC-6 especially for large problems with large domain size.

5

Conclusion and Future Work

In this paper we have presented a new algorithm for maintaining the arc consistency of a CSP in a dynamic environment. Theoretical and experimental comparison of our algorithm with those proposed in the literature demonstrate that our algorithm has a better compromise between time and memory costs. In the near future we are looking to integrating our dynamic arc consistency algorithm during the backtrack search phase in order to handle the addition and relaxation of constraints. For instance, if a value from a variable domain is deleted during the backtrack search, would it be worthwhile to use a DAC algorithm to determine its effect or would it be more costly than just continuing on with the backtrack search.

References [1] A. K. Mackworth. Consistency in networks of relations. Artificial Intelligence, 8:99–118, 1977. 393, 394 [2] R. M. Haralick and G. L. Elliott. Increasing tree search efficiency for Constraint Satisfaction Problems. Artificial Intelligence, 14:263–313, 1980. 393 [3] P Van Hentenryck. Constraint Satisfaction in Logic Programming. The MIT Press, 1989. 393

400

Malek Mouhoub

[4] C. Bessi`ere. Arc-consistency in dynamic constraint satisfaction problems. In AAAI’91, pages 221–226, Anaheim, CA, 1991. 394, 395 [5] R. Debruyne. Les algorithmes d’arc-consistance dans les csp dynamiques. Revue d’Intelligence Artificielle, 9:239–267, 1995. 394, 395 [6] B. Neuveu and P. Berlandier. Maintaining arc consistency through constraint retraction. In ICTAI’94, pages 426–431, 1994. 394, 395 [7] R. Mohr and T. Henderson. Arc and path consistency revisited. Artificial Intelligence, 28:225–233, 1986. 394, 395 [8] C. Bessi`ere. Arc-consistency and arc-consistency again. Artificial Intelligence, 65:179–190, 1994. 394, 395 [9] C. Bessi`ere, E. Freuder, and J. C. Regin. Using inference to reduce arc consistency computation. In IJCAI’95, pages 592–598, Montr´eal, Canada, 1995. 394 [10] R. J. Wallace. Why AC-3 is almost always better than AC-4 for establishing arc consistency in CSPs. In IJCAI’93, pages 239–245, Chambery, France, 1993. 394, 395 [11] Yuanlin Zhang and Roland H. C. Yap. Making ac-3 an optimal algorithm. In Seventeenth International Joint Conference on Artificial Intelligence (IJCAI’01), pages 316–321, Seattle, WA, 2001. 394, 395, 396 [12] C. Bessi`ere and J. C. R´egin. Refining the basic constraint propagation algorithm. In Seventeenth International Joint Conference on Artificial Intelligence (IJCAI’01), pages 309–315, Seattle, WA, 2001. 394 [13] A. K. Mackworth and E. Freuder. The complexity of some polynomial networkconsistency algorithms for constraint satisfaction problems. Artificial Intelligence, 25:65–74, 1985. 395 [14] D. Sabin and E. C. Freuder. Contradicting conventional wisdom in constraint satisfaction. In Proc. 11th ECAI, pages 125–129, Amsterdam, Holland, 1994. 398

Determination of Decision Boundaries for Online Signature Verification Masahiro Tanaka1 , Yumi Ishino2 , Hironori Shimada2 , Takashi Inoue2 , and Andrzej Bargiela3 1

Department of Information Science and Systems Engineering Konan University, Kobe 658-8501, Japan m [email protected] http://cj.is.konan-u.ac.jp/~tanaka-e/ 2 R & D Center Glory Ltd., Himeji 670-8567, Japan {y ishino,shimada,inoue}@dev.glory.co.jp 3 Department of Computing and Mathematics The Nottingham Trent University, Burton St, Nottingham NG1 4BU, UK [email protected]

Abstract. We are developing methods for online (or dynamical) signature verification. Our method is first to move the test signature to the sample signature so that the DP matching can be done well, and then compare the pen information along the matched points of the signatures. However, it is not easy to determine how to use the several elements of discrepancy between them. In this paper, we propose a verification method based on the discrimination by RBF network.

1

Introduction

We are developing methods for online signature verification. The data is multidimensional time series obtained by the tablet and an electronic pen ([1, 4, 3]). The information available in our system includes x and y coordinates, the pressure of the pen, the azimuth and the altitude of the pen. Hence the data can be seen as a sequence of vectors. Our method is first to modify the test signature to the template signature so that the DP matching can be done well, and then compare the pen information along the matched points of the signatures. One of the most intuitive methods for verification or recognition of handwritten characters or signatures is to extract the corresponding parts of two drawings and compare them next. For this objective, we use only the coordinates of the drawings and neglect other elements. We developed this method as a preprocessing tool before the main procedure of verification method [5, 6, 7]. However, it is not easy to determine how to use the several elements of differences between them. In this paper, we propose a verification method based on the discrimination by neural network. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 401–407, 2003. c Springer-Verlag Berlin Heidelberg 2003


402

2

Masahiro Tanaka et al.

Data Categories

Suppose q-dimensional time-series vector z ∈ Rq is available from the tablet with a fixed interval, whose elements include x and y coordinates, pressure, azimuth and the altitude of the pen. We will use k as the discrete time index. For a signature verification problem, the following three kinds of data category should be prepared. – Template. Template signature is the one registered by the true person. We could use more than two template signatures, but we use one template here for the simplicity of the treatment. Let S ∈ Rq×ns represent it. – Training signatures. We need both true and forgeries. It is possible to make a model with only genuine signatures and establish probability density function model, but the ability of discrimination is low. It is better to use both positive and negative cases. The more training signatures, the higher the accuracy. Let T1 , T2 , · · ·, Tm be the genuine signatures. Let Fij , j = 0, · · · , m(i); i = 1, · · · , n be the forgeries, where i is the ID of the person and j is the file number. – Test signatures. We also need both genuine and forgeries for checking the model of verification. If we need to authenticate n signers, we need to prepare n-sets of the above.

3

Matching Method

By considering the diversity of genuine signatures as the partial drift of the speed during signing procedure, it is appropriate to apply DP matching. Here we will use only x and y which are elements of z for matching of two vectors. One ¯ = [p(1), · · · , p(ns )] ∈ sequence is the x, y elements of S. We will denote it as S 2×ns . Another sequence to compare with it is a 2-dimensional sequence which R is denoted here as Q = [q(1), · · · , q(nq )]. For this problem, DP matching did not work sufficiently well. However, we considered this to be particularly important for verification. So, we proposed the matching method in [5], which is to apply DP matching and the coordinate modification iteratively. Next we will explain the summary of DP matching and coordinate modification procedures. 3.1

DP Matching

DP matching used here is the method to derive the corresponding points that give the minimal cost for two vector sequences, and the following recursive form is well known:    D(i − 1, j) + d(i, j)  D(i, j) = min D(i − 1, j − 1) + 2d(i, j) (1)   D(i, j − 1) + d(i, j)

Determination of Decision Boundaries for Online Signature Verification

403

where d(i, j) is the distance between p(i) and q(j), which is usually Euclidean distance. 3.2

Coordinate Modification

This is a procedure for moving points of the test signature to the sample signature. The following is the model: x (k) = a(k)x(k) + b(k) 

y (k) = a(k)y(k) + c(k)

(2) (3)

where k denotes the time index, x(k) and y(k) are the elements of p(k). x (k) and y  (k) are the transformed values which are taken as the elements of q(k), and a(k), b(k), c(k) are time-variant transformation parameters. Now we define the parameter vector θ as θ(k) = [a(k) b(k) c(k)] If they are allowed to change slowly and independently, the model can be written as the following: θ(k + 1) = θ(k) + w(k) (4) where w(k) is a zero-mean random vector with covariance matrix Q = diag(q1 , q2 , q3 ). If the diagonal elements q1 , q2 , q3 are small, it means the elements of θ(k) are allowed to change slowly. Now we fix the template data p(k). Then it is expected that the test data transformed by (2)-(3) is close to the template data p(k). We can write this assertion by the following equation obtained from equations (2), (3) z(k) = H(k)θ(k) + v(k) where



x (k) z(k) =  y (k)

is the test data, and

 H(k) =

(5)



x(k) 1 0 y(k) 0 1

 (6)

includes the elements of the model data where v(k) is a 2-dimensional random vector independent from w, its mean is zero and covariance R. This absorbs the unmodelled portion of the observation data. Based on equations (4) and (5), we can estimate the time-variant parameter θ(k) by using the linear estimation theory. If we must estimate them on-line, we would use Kalman filter or fixed-lag smoother. However besides this, the data must be matched by using DP matching, where we must use the data in an offline way. So, we must work in an off-line manner, and “fixed-interval smoother” yields the best result for off-line processing. Hence we will use fixed-interval smoother.

404

4

Masahiro Tanaka et al.

Verification Method

We have constructed the signature verification method based on the matching of signatures. Here we will describe the method in detail. 4.1

Normalisation of Training Data

We have experimentally experienced that data normalisation is very useful before matching. Let p be the original 2-D signal, and let R be the covariance matrix of p. By the orthogonalisation, we have RV = V Λ, where V is the orthonormal matrix and Λ is the diagonal matrix. Since V is orthonormal, we have V −1 = V  and V  RV = λ. ¯ be the mean of p. Also define ¯r = V  p ¯ and r = V  p − ¯r. Then we Now, let p have ¯ )(p − p ¯ ) ]V = Λ E[rr ] = V  E[(p − p If we want to have the data whose standard deviation of the horizontal axis is 1000, we can normalise it by 1000 r = √ (V  p − ¯r) λ1 thus we have E[rr ] =

4.2



106 0

λ2 λ1

0 × 106



Matching Data

Here we apply the iterative procedure of DP matching and coordinate modification of the data as described in section 3. 4.3

Extracting Feature Vectors

After the matching of two signatures s and ti , we can use the square of the distance between each elements as d(i) = [d1 (i) d2 (i) · · · d7 (i)], i = 1, · · · , m or m(j)

(7)

The above criteria are common to the genuine and forgeries. Each of the elements in (7) are the mean square difference between the template signature and the training signature. Each of the criteria have the following meaning. Suppose (i, j) is the index after matching for one of the vectors. Then we trace back the original position (i , j  ) and calculate the velocity by  vel = (x(i − 1) − x(i ))2 + (y(j  − 1) − y(j  ))2 (8)    2   2 + (x(i ) − x(i + 1)) + (y(j ) − y(j + 1))

Determination of Decision Boundaries for Online Signature Verification

405

Table 1. Elements of Criteria element number meaning 1 length of the data 2 modified coordinate x after matching 3 modified coordinate y after matching 4 pressure of matched points 5 angle of matched points 6 direction of matched points 7 velocity of matched points

This trace back is necessary because the matched index does not necessarily proceed smoothly, and will not show the actual velocity of the signature. The meaning of other elements will be straightforward, so we will omit the detail explanation. 4.4

Model Building by RBF Network

Thus we have the verification model. A training vector d(i) is 7-dimensional, and the output o(i) is 1 (for genuine) or 0 (for forgery). We have m cases for o(i) = 1 and m(1) + · · · + m(n) cases for o(i) = 0. However, the scale of the criteria vary a lot, thus we need further normalisation. Hence we normalised the criteria by 1. Find the maximum maxj among the j-th elements in the training vectors, and 2. Divide all the elements of j-th elements by maxj . This maxj is again used in the verification. Let us denote the training data by d(k), k = 1, · · · , N . By using all the samples as the kernel of the function, we have the model f (x) =

N k=1

x − dk 2 θk exp − σ2

(9)

for a general x ∈ R7 . This can be rewritten as Y = Mθ

(10)

where Y ∈ RN , M ∈ RN ×N , and θ ∈ RN . By using the training data dk , k = 1, · · · , N , we have the n × N matrix M whose rank is N . So, we have a unique parameter θ. It is clear that equation (9) produces the exact values for the training data dk , k = 1, · · · , N . Various kinds of neural networks can adapt to nonlinear complicated boundary problems. However, we applied RBF network for a special reason. It has

406

Masahiro Tanaka et al.

a universal approximation property like multi-layer perceptron. However, RBF network has another good property for pattern recognition like this. Due to the functional form, it intrinsically produces nearly zero if the input vector is not similar to any of the training data. Other recogniser like multi-layer perceptron does not have this property. 4.5

Verification

We can use the model for verifying whether the data is genuine or forgery based on the model. When we have the data, we have to process it as follows. 1. Normalise the original data 2. Normalize the feature vector 3. Input the feature vector to the recogniser.

5

Experimental Results

Fig. 1 shows the ranges of each criterion. The sub-figures are numbered in the order from the top-left, top-right, and to the second row. Sub-figure 1 shows the values of d1 (i), i = 1, · · · , 9. The same for the sub-figure 2 with values d2 (i), i = 1, · · · , 9, and so on. The first var corresponds to the genuine signature. By using each criterion alone, it is almost impossible to divide the space clearly. However, by using the RBF network, we have obtained a good separation result as shown in Fig. 2. The first 15 cases are the genuine signatures and all the others are forgeries. If we put the threshold at 0.4, the errors of the first kind is 3 out of 15 and the errors of the second kind is 3 out of 135. This is fairly a good result.

1500

6

1000

4

500 0 3

x 10

4

2

0 x 10

4

2

4

6

8

10

0 4

0 x 10

5

2

4

6

8

10

0 x 10

4

2

4

6

8

10

2

4

6

8

10

2 2 1 0 3

0 x 10

5

2

4

6

8

10

3

2

2

1 0 4

0

1

0 x 10

4

2

4

6

8

10

2

4

6

8

10

0

0

2

0

0

Fig. 1. Ranges of the criteria

Determination of Decision Boundaries for Online Signature Verification

407

1.2

1

0.8

output

0.6

0.4

0.2

0

-0.2

0

50

100

150

sample #

Fig. 2. Output of RBF network for test data

6

Conclusions

The performance of the algorithm will be upgraded if we apply the matching procedure more times. We will have to apply the algorithm for other persons’ data as the template. Further comparison will be our future work.

References [1] L. L. Lee, T. Berger and E. Aviczer, “Reliable On-line Human Signature Verification Systems,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 18, pp. 643-647, 1996. 401 [2] M. L. Minsky and S. A. Papert, Perceptrons – Expanded Edition, MIT, 1969. [3] V. S. Nalwa, “Automatic On-line Signature Verification,” Proceedings of the IEEE, Vol. 85, pp. 215-239, 1997. 401 [4] R. Plamondon and S. N. Srihari, “On-line and Off-line Handwriting Recognition: A Comprehensive Survey,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 22, pp. 63-84, 2000. 401 [5] M. Tanaka, Y. Ishino, H. Shimada and T. Inoue, “DP Matching Using Kalman Filter as Pre-processing in On-line Signature Verification,” 8th International Workshop on Frontiers in Handwriting Recognition,, pp. 502–507, Niagara-onthe-Lake, Ontario, Canada, August 6-8, 2002. 401, 402 [6] M. Tanaka, Y. Ishino, H. Shimada and T. Inoue, Dynamical Scaling in Online Hand-written Characters’ Matching, 9th International Conference on Neural Information Processing, Singapore, November 19-22, 5 pages (CD ROM), 2002. 401 [7] M. Tanaka, Y. Ishino, H. Shimada, T. Inoue and A. Bargiela, “Analysis of Iterative Procedure of Matching Two drawings by DP Matching and Estimation of Time-Variant Transformation Parameters,” The 34th International Symposium on Stochastic Systems Theory and Its Applications, accepted. 401

On the Accuracy of Rotation Invariant Wavelet-Based Moments Applied to Recognize Traditional Thai Musical Instruments Sittisak Rodtook and Stanislav Makhanov Information Technology Program, Sirindron International Institute of Technology, Thamasat University, Pathumthani 12121, Thailand {sittisak,makhanov}@siit.tu.ac.th

Abstract. Rotation invariant moments constitute an important technique applicable to a versatile number of applications associated with pattern recognition. However, although the moment descriptors are invariant with regard to spatial transformations, in practice the spatial transformation themselves, affect the invariance. This phenomenon jeopardizes the quality of pattern recognition. Therefore, this paper presents an experimental analysis of the accuracy and the efficiency of discrimination under the impact of the rotation. We evaluate experimentally the behavior of the noise induced by the rotation versus the most popular basis functions based on wavelets. As an example, we consider a particular but interesting case of the Thai traditional musical instruments. Finally, We present a semi heuristic pre computing technique to construct a set of descriptors suitable for discrimination under the impact of the spatial transformation.

1

Introduction

It has been very well documented that performance of pattern recognition may critically depend on whether the employed descriptors are invariant with respect to spatial transformations. A popular class of the invariant shape descriptors is based on the moment techniques, first, introduced by Hu[1] . However, a dramatic increase in complexity when increasing the order makes the Hu’s moments impractical. Shortly after the Hu’s paper a variety of invariant moments based techniques designed to recognize moving objects was proposed and analyzed [2][6]. The major developments of the rotational invariant moment-based methods are the orthogonal Zernike, orthogonal Fourier-Mellin and complex moments. Finally, Shen [3] introduced the wavelet moments and showed that multi-resolution accomplished by the wavelets made it possible to construct adaptable moment descriptors best suited for a particular set of objects. Sensitivity of the moment descriptors to the image noise has been repeatedly mentioned in the Literature. An interesting consequence of this is that the moment descriptors are invariant only when they are computed from the ideal analog images. Even in the absence of a noise induced by physical devices, there V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 408–414, 2003. c Springer-Verlag Berlin Heidelberg 2003


On the Accuracy of Rotation Invariant Wavelet-Based Moments

409

always exists a noise due to the spatial transformations. Therefore, although the moment descriptors are invariant with regard to the spatial transformations, in practice the spatial transformation themselves affect the invariance. This phenomenon could seriously jeopardize the quality of pattern recognition. Therefore, this paper considers accuracy and discriminative properties of the wavelet-based moment descriptors under the impact of rotation. We perform the experiments as applied to an interesting case of Thai musical instruments. The impact of the rotation transforms is profound due to the elongated shape of the instruments. We analyze the range of the errors and the accuracy. Finally, we present a combination of a variance-based procedure with a semiheuristic pre-computing technique to construct the appropriate wavelet-based descriptors.

2

Rotationally Invariant Wavelet-Based Moment

T ype A general rotation invariant moment MOrder with respect to moment function T ype Forder (r, θ) defined in the polar coordinates (with the origin at the centroid of the object) is given by


2π
1 T ype MOrder

T ype f (r, θ)FOrder (r, θ)rdrdθ

= 0

0

T ype T ype T ype We assume that Forder (r, θ) = RΩ (r)Gα (θ) , where RΩ (r) denotes the radial moment function and Gα (θ) the angular moment function. Note that the angular function defined by Gα (θ) = ei α θ provides rotational since    invariance   T ype   T ype  it leads to the circular Fourier transform. Therefore, MOrder  = M ROrder  [3] T ype where M ROrder be a moment of the rotated object image and φ be the angle of rotation. In words, rotation of the object affects the phase but not the magnitude. In order to compute the rotation invariant moments of a given image, we first calculate the centroid of the object image. Next, we define the polar coordinates and map the object onto the parametric unit circle. Finally, we represent the integral by


2π
1 T ype MOrder

= 0

where Sα (r) =

2π 


1 T ype f (r, θ)FOrder (r, θ)rdrdθ

0

T ype RΩ (r)Sα (r)rdr.

= 0

f (r, θ)Gα (θ)dθ. The continuous image function is obtained by

0

the standard bilinear interpolation applied to the discrete image. 1 T ype T ype | is rotation invariant but RΩ (r) |Sα (r)| rdr and Finally, not only |MOrder  1  T ype  RΩ (r)Sα (r)r  dr are rotation invariant as well. 0

0

410

Sittisak Rodtook and Stanislav Makhanov

It is plain that from the viewpoint of functional analysis, each object is represented by an infinite and unique set of the rotational invariants. In other words, T ype if the set RΩ (r) constitutes a basis in L2 [0, 1] then Sα (r) can be represented with a prescribed accuracy. However, in practice, we always have a finite set of moment descriptors affected by noise. Wavelets being well localized in time and frequency are an efficient tool to construct appropriate moment descriptors. Consider a wavelet-based radius function given by Rm,n (r) = √1m ψ( r−n m ) where ψ(r) is the mother wavelet, m the dilation parameter (the scale index), n the shifting parameter. Unser et al show the usefulness of biorthogonal spline wavelets transform for texture analysis [7]. In this paper we apply the cubic B-spline to construct the appropriate moment descriptors. In this case of a B-spline the mother wavelet is given by (2r − 1)2 4 σw cos(2πf0 (2r − 1)) exp(− 2 ). ψ(r) =  2σw (n + 1) 2π(k + 1) 2 where k = 3, a = 0.7, f0 = 0.41 and σw = 0.56. The basis functions are given by m

ψm,n (r) = 2 2 ψ(2m r − 0.5n).

3

Errors of the Moment Descriptors Applied to Thai Musical Instruments

We analyze the accuracy of the wavelet-based moments as applied to the Thai musical instrument images in presence of the geometric errors induced by the rotational transforms and the subsequent binarization. We also consider an effect of the rotation. Photographs of the instruments are rotated by 360◦ with the increment 5◦ (see Fig. 1) by the Adobe Photoshop. In order to eliminate accumulation of errors due to multiple re-sampling, each rotation has been performed by rotating the original photograph corresponding to 0◦ . Fig. 2 shows a typical impact of rotation in the case of the spline wavelet moment. We evaluate the accuracy by measuring the standard deviation of the normalized spline moment descriptor. The error varies from 0.0 to 27.81 % with the maximum error produced by the ”SUENG” rotated by 50◦ for |M0,1,5 |. The wavelets make it possible to control not only the spatial position of the basis function but the frequency range as well. However, without an appropriate adaptation, the rotation may drastically affect a wavelet descriptor. We analyze the accuracy of spline wavelet descriptors |Mm,n,α | where α is the angular order. The comparison of the accuracy versus the position and the main frequency of wavelet basis function shows that the maximum error is only 7.21% for a 130◦ rotated ”SAW OU” with |M1,2,5 |. Note that the extrema of ψ1,2 (r) and extrema of |S5 (r)| almost coincide which results in a large numerical value of the moment. Furthermore, Fig. 3 indicates the best wavelet for particular frequencies. Fig. 4 shows why wavelets ψ1,4 (r),ψ2,1 (r) ,ψ2,4 (r) and ψ3,1 (r) produce poor results. The noise either has been substantially magnified at various positions

On the Accuracy of Rotation Invariant Wavelet-Based Moments

(a)

(b)

(d)

(e)

411

(c)

(f)

Fig. 1. Gray-scale photos and Silhouettes of the Thai musical instruments (a). ”SUENG”, Lute. (b). ”SAW SAM SAI”, Fiddle. (c). ”PEE CHAWA”, Oboe. (d). ”PEE NOKE”, Pipe. (e). ”SAW DUANG”, Fiddle. (f). ”SAW OU”, Fiddle

Fig. 2. Impact of rotation of |S5 (r)| by ψ2,1 (r), ψ2,4 or has been ”washed out” along with the peak itself by ψ1,4 (r)or ψ3,1 (r) . In other words, although ψ1,4 (r) and ψ3,1 (r) eliminate the noise they ”wash out” information about the object as well. Such functions are easily detected by the energy threshold (see for instance[9]). Unfortunately, the rotation noise in the frequency domain often appears at low frequencies which also characterize the signal. The rotated object produces a shifted of Sα (r), in other words, the noise often ”replicates” Sα (r). That is why it is difficult to construct a conventional filter in this case.

Table 1. The standard deviation calculated for the normalized spline wavelet moment magnitude corresponding to ”SAW OU” m 0 1 2 3

The standard deviation |Mm,n,5 | n=0 n=1 n=2 0.0800 0.0255 0.0221 0.0438 0.0213 0.0201 0.0450 0.0714 0.0270 0.1093 0.0526 0.1053

of normalized n=3 0.0231 0.0455 0.0230 0.1009

n=4 0.0225 0.0679 0.0681 0.1534

412

Sittisak Rodtook and Stanislav Makhanov

Fig. 3.

ψm,n (r) versus |S5 (r)|

Fig. 4. Rotation noise. (a). |S5 (r)| (Solid line: the original ”SAW OU”, Dasheddoted line: 335◦ rotated ”SAW OU”). (b)-(f). Wavelet versus |S5 (r)|. (g)-(k). Normalized |ψ(r)S5 (r)r|. (b),(g). ψ1,2 (r) (c),(h). ψ1,4 (r) (d),(i). ψ2,1 (r) (e),(j) ψ2,4 (r) (f),(k) ψ3,1 (r)

4

Pre Computing Techniques to Classify the Wavelet-Based Descriptors Applied to the Thai Musical Instruments

Our experiments reveal that it is difficult to find an appropriate moment descriptor which provides the both accurate and discriminative representation. The most accurate moment descriptors might be different for different instruments. It is not always possible to find one small set of wavelet basis functions suitable to represent all the angular moments. Note that in case of dissimilar, unsymmetrical objects, a moment descriptor suitable for discrimination can be derived with α = 0. However, if the objects have similar shapes, the zero angular order is not sufficient. Moreover, it is difficult to decide which angular order is the most representative since different angular orders magnify different frequencies of the noise. Meanwhile, we find the best discriminative moment descriptor for each good angular order and construct an appropriate vector of wavelet-based moment descriptors (such as (|Mm1 ,n1 ,q1 | , |Mm2 ,n2 ,q2 | , ... |Mmk ,nk ,qk |)). Furthermore, given the magnitudes of the wavelet-based moments we apply the variance-based classification technique (Otsu’s algorithm) introduced in [8] which invokes the least inner-class/ the largest inter-class variance ratio representing the discriminative measures of the descriptors.

On the Accuracy of Rotation Invariant Wavelet-Based Moments

413

Fig. 5. (a)-(b). |F T (S5 (r))| , (†: the original ”SAW OU”, ∞ : 335◦ rotated ”SAW OU”). (c). |F T (N )|, N : The spatial noise in frequency domain Next, we propose a concatenation of the above variance-based procedure with the pre computing techniques as follows: 1. Pre-compute an appropriate set of the angular order α = {q1 , q2 , . . . , qn } by considering the least of circular Fourier transform square error for each qi .


M  N N   ( dif fi,j (rk )2 ∆r)

1 , ∆r =  , rk = k∆r, k = 0, 1, 2, ...N  MN N         where dif fi,j (rk ) = Sql (rk )i,Orginal  − Sql (rk )i,j . N  denotes the number of points employed for numerical integration, N the number of rotations and M denotes the number instruments. 2. Select a set of wavelet basis functions ψm,n (r). The function must belong to a basis in L2 [0, 1] 3. Check the following condition: – If |Sα (r)| is large at r than at least one of the basis  functions ψ must be large at r. The condition could be replaced by {ψi > ε} = [0, 1] for εql =

i=1 j=1 k=1

i

4. 5. 6. 7.

some ε. For each angular order and for each musical instrument find the set of moment descriptors having the best normalized standard deviation. Threshold the wavelet-based moment descriptors by the energy [9]. Collect the best descriptors. Apply the variance-based techniques to the set of the descriptors for each angular order.

The techniques applied to case of the Thai musical instruments makes it possible to decrease the computational time by 10-15%. The discriminative measures in the Table. 2 demonstrate that the B-spline wavelet moment descriptors are well separated. The discriminative measure averages of the first three best discriminative descriptors |M1,2,5 |,|M1,0,5 | and |M1,3,5 | at α = 5 are 0.0726, 0.0915 and 0.1163 respectively. The energy threshold is 2.0.

5

Conclusion

Rotations produce a noise which could substantially affect the quality of descriptors and their discriminative properties. Wavelets constitute a suitable class

414

Sittisak Rodtook and Stanislav Makhanov

Table 2. The appropriate spline wavelet descriptors for discrimination and discriminative measures at α = 5 SUENG SAM SAI CHA WA NOKE DAUNG

SAMSAI |M1,0,5 | 0.0039

CHA WA |M1,0,5 | 0.0035 |M0,1,5 | 0.0149

NOKE |M1,2,5 | 0.0013 |M1,2,5 | 0.0039 |M1,2,5 | 0.0043

DUANG |M0,1,5 | 0.0031 |M1,3,5 | 0.0054 |M1,3,5 | 0.0039 |M1,0,5 | 0.0054

OU |M1,0,5 | 0.0043 |M1,3,5 | 0.0121 |M0,3,5 | 0.0064 |M1,2,5 | 0.0023 |M1,2,5 | 0.0030

of basis functions to perform recognition under the impact of rotations. The proposed algorithm based on standard deviation and energy combined with the variance-based procedure makes it possible to efficiently construct a set of descriptors suitable for discrimination under the impact of the rotation. The best discrimination properties for the set of the Thai musical instruments are displayed by {|M0,1,1 |, |M1,2,3 |, |M1,2,5 |, |M0,1,6 |, |M1,1,7 |}, appropriate α = {1, 3, 5, 6, 7}.

References [1] Hu, M. K.: Visual pattern recognition by moment invariants. IRE Trans. Inform. Theory, 8 (1962) 179–187 408 [2] Liao, S. X., Pawlak, M.: On the Accuracy of Zernike Moments for Image Analysis. IEEE Trans. On Pattern Analysis and Machine Intelligence, Vol. 20. 12 (1998) 1358–1364 408 [3] Shen, D., Ip, H. H.: Discriminative Wavelet Shape Descriptors for Recognition of 2-D patterns. Pattern Recognition, 32 (1999) 151–165 408, 409 [4] Kan, C., Srinath, M. D.: Orthogonal Fourier-Mellin moments with centroid bounding circle scaling for invariant character recognition. 1st IEEE conf. on Electro Information Technology, Chicago illinois (2000) [5] Kan, C., Srinath, M. D.: Invariant Character Recognition with Zernike and Orthogonal Fourier-Mellin Moments. Pattern Recognition, 35 (2002) 143–154 [6] Flusser, J.: On the inverse problem of rotation moment invariants. Pattern Recognition, 35 (2002) 3015–3017 408 [7] Unser, M. Aldroubi, A., Eden, M.: Family of polynomial spline wavelet trans1 form. Signal Process., 30 (1993) 141–162 410 [8] Otsu, N.: A Threshold Selection Method from Gray Level Histograms. IEEE Tran. On Systems Man. and Cybernetics, SMC-9 (1985) 377–393 412 [9] Thuillard, M.: Wavelets in Soft Computing, Series in Robotices and Intelligent Systems, 25 World Scientific London 84-85 411, 413

A Multi-agent System for Knowledge Management in Software Maintenance Aurora Vizcaino1, Jesús Favela 2, and Mario Piattini1 1

Escuela Superior de Informática, Universidad de Castilla-La Mancha {avizcain,mpiattini}@inf-cr.uclm.es 2 CICESE, Ensenada, México [email protected]

Abstract. Knowledge management has become an important topic as organisations wish to take advantage of the information that they produce and that can be brought to bear on present decisions. This work describes a system to manage the information and knowledge generated during the software maintenance process, which consumes a large part of the software lifecycle costs. The architecture of the system is formed from a set of agent communities, each community is in charge of managing a specific type of knowledge. The agents can learn from previous experience and share their knowledge with other agents, or communities. Keywords: Muti-agent systems, Knowledge Management, Software Maintenance

1

Introduction

Knowledge is a crucial resource for organizations, it allows organizations to fulfil their mission and become more competitive. For this reason, companies are currently researching techniques and methods to manage their knowledge systematically. In fact, nearly 80% of companies worldwide have some knowledge management efforts under way [5]. Organizations have different types of knowledge that are often related to each other and which must be managed in a consistent way. For instance, software engineering involves the integration of various knowledge sources that are constantly changing. The management of this knowledge and how it can be applied to software development and maintenance efforts has received little attention from the software engineering research community so far [3]. Tools and techniques are necessary to capture and process knowledge in order to facilitate subsequent development and maintenance efforts. This is particularly true for software maintenance, a knowledge intensive activity that depends on information generated during long periods of time and by large numbers of people, many of whom may no longer be in the organisation. This paper presents a multi-agent system (KM-MANTIS) in charge of managing V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 415-421, 2003.  Springer-Verlag Berlin Heidelberg 2003

416

Aurora Vizcaino et al.

the knowledge that is produced during software maintenance. The contents of this paper are organized as follows: Section 2 presents the motivation for using a Knowledge Management (KM) system to support software maintenance. Section 3 describes the implementations of KM-MANTIS. Finally, conclusions are presented in Section 4.

2

Advantages of Using a KM System in Software Maintenance

Software maintenance consumes a large part of overall lifecycle costs [2, 8]. The incapacity to change software quickly and reliably causes organizations to lose business opportunities. Thus, in recent years we have seen an important increase in research directed towards addressing these issues. On the other hand, software maintenance is a knowledge intensive activity. This knowledge comes not only from the expertise of the professionals involved in the process, but it is also intrinsic to the product being maintained, and to the reasons that motivate maintenance (new requirements, user complaints, etc.) processes, methodologies and tools used in the organization. Moreover, the diverse types of knowledge are produced at different stages of the MP. Software maintenance activities are generally undertaken by groups of people. Each person has partial information that is necessary to other members of the group. If the knowledge only exists in the software engineers and there is no system in charge of transferring the tacit knowledge (contained in the employees) to explicit knowledge (stored on paper, files, etc) when an employee abandons the organization a significant part of the intellectual capital goes with him/her. Another well-known issue that complicates the MP is the scarce documentation that exists related to a specific software system, or even if detailed documentation was produced when the original system was developed, it is seldom updated as the system evolves. For example, legacy software written by other units often has little or no documentation describing the features of the software. By using a KM system the diverse kinds of knowledge generated may be stored and shared. Moreover, new knowledge can be produced, obtaining maximum benefit from the current information. By reusing information and producing relevant knowledge the high costs associated with software maintenance could also be decreased [4]. Another advantage of KM systems is that they help employees build a shared vision, since the same codification is used and misunderstanding in staff communications may be avoided. Several studies have shown that a shared vision may hold together a loosely coupled system and promote the integration of an entire organisation [7].

3

A Multi-agent System to Manage Knowledge in Software Maintenance

The issues explained above motivated us to design a KM system to capture, manage, and disseminate knowledge in a software maintenance organisation, thus increasing

A Multi-agent System for Knowledge Management in Software Maintenance

417

the workers' expertise, the organisation's knowledge and its competitiveness while decreasing the costs associated with the software MP. KM-MANTIS is a multi-agent system where different types of agent manage the diverse types of information generated during SMP. Agents interchange data and take advantage of the information and experience acquired by other agents. In order to foster the interchange of information the system uses an open format to store data and metadata XMI (XML metadata interchange) [6]. This is an important advantage of this system, since data and metadata defined along with other tools that support XMI can also be managed by KM-MANTIS. And it also facilitates the interchange of information between agents since they all use the same information representation. 3.1

The KM-MANTIS Architecture

The system is formed of a set of agent communities which manage different types of knowledge. There are several reasons why agents are recommendable for managing knowledge. First of all, agents are proactive, this means they decide to act when they find it necessary to do so. Moreover, agents can manage both distributed and local information. This is an important feature since software maintenance information is generated by different sources and often from different places. One aspect related to the previous one is that agents may cooperate and interchange information. Thus, each agent can share its knowledge with others or ask them for advice, benefiting from the other agents' experience. Therefore, there is reuse and knowledge management in the architecture of the system itself. Another important issue is that agents can learn from their own experience. Consequently the system is expected to become more efficient with time since the agents have learnt from their previous mistakes and successes. On the other hand, each agent may utilize different reasoning techniques depending on the situation. For instance, an agent can use an ID3 algorithms to learn from previous experiences and use case-based reasoning to advise a client on how to solve a problem. The rationale for designing KM-MANTIS with several communities is that during the software MP different types of information are produced, each with its own specific features. The types of information identified were: information related to the products to be maintained; information related to the activities to be performed in order to maintain the products; and, peopleware involved during software maintenance [10]. Therefore, KM-MANTIS has three communities: a community termed the "products community", another called the “activities community”, and the last community denoted as the "peopleware community". In what follows we describe each community in more detail. Products Community: This community manages the information related to the products to be maintained. Since each product has its own features and follows a specific evolution this community has one agent per product. The agents have information about the initial requirements, changes made to the product, and about metrics that evaluate features related to the maintainability of the product, (this

418

Aurora Vizcaino et al.

information is obtained from different documents such as modification requests, see Figure 1, perfective, corrective or preventive actions performed or product measurements). Therefore, the agents monitor the product's evolution in order to have up to date information about it at each moment. Each time an agent detects that information about its product is being introduced or modified in KM-MANTIS (the agent detects this when the application identification number that it represents is introduced or displayed in the interface of KM-MANTIS) the agent analyses the new information, or comparing it to that previously held in order to detect inconsistencies, or checking the differences and storing the relevant information in order to have up-to-date information. Information relevant to each product (data) is stored in an XMI repository. The XMI repository also stores rules (knowledge) produced by the agents through induction and decision trees-based algorithms. The decision to use XMI documents based on the MOF (Meta Object Facility) standard makes it possible for agents to have access to the different levels of information and knowledge that they need to process and classify their information and the queries that they receive. Activities community: Each new change demanded implies performing one or more activities. This community, which has one agent per activity, is in charge of managing the knowledge related to the different activities including methods, techniques and resources used to perform an activity.

Fig. 1. KM-MANTIS Interface

A Multi-agent System for Knowledge Management in Software Maintenance

419

Activities agents can also obtain new knowledge from their experience or taught learning. For instance, an activity agent learns which person usually carries out a specific activity or what techniques are most often used to perform an activity. Furthermore, activities agents use case-based reasoning techniques in order to detect whether a similar change under analogous circumstances was previously requested. When this is the case the agent informs the users on how that problem was previously solved, taking advantage of the organization's experience. Peopleware Community: Three profiles of people can be clearly differentiated in MP [9]: the maintenance engineer, the customer and the user. The Peopleware community has three types of agent, one per profile detected. One agent is in charge of the information related to staff (maintainers). This is the staff agent. Another manages information related to the clients (customers) and is called the client agent. The last one is in charge of the users and is termed the user agent. The staff agent monitors the personal data of the employees, in which activities they have worked, and what product they have maintained. Of course, the agent also has current information about each member of the staff. Therefore it knows where each person is working at each moment. The agent can utilise the information that it has to generate new knowledge. For instance, the staff agent calculates statistics to estimate the performance of an employee. The client agent stores the information of each client, their requirements (including the initial requirements, if they are available) and requests. The client agent also tries to gather new knowledge. For instance, it tries to guess future requirements depending on previous requirements or it estimates the costs of changes that the client wants to make, warning him, for instance, of the high costs associated with a specific change request. The user agent is in charge of knowing the requirements of the users of each product, their background and also their complaints and comments about the products. New knowledge could also be generated from this information, for example by testing to what degree the users' characteristics influence the maintenance of the product. Each type of agent has a database containing the information that they need. In this case there is no community repository because there is no data common to the three types of agents. 3.2

Implementation Considerations

In order to manage the XML documents different middleware alternatives were studied, some being object-relational databases such as ORACLE 9, Microsoft SQL Server 2002 and Tamino which have been designed specifically for XML documents. Finally Tamino, a Software AG's product, was chosen, because KM-MANTIS needs to store a huge amount of XML documents and manage them efficiently. The fact that an object-relational database needs to translate XML to tables and vice versa considerably reduces its efficiency. On the other hand, the platform chosen for creating the multiagent system is JADE [1] which is FIPA compliant. The agent communication language is FIPA ACL. Agents interchange information in order to take advantage of the knowledge that

420

Aurora Vizcaino et al.

others have, thereby the architecture itself performs reuse of information and knowledge.

4

Conclusions

Software maintenance is one of the most important stages of the software life cycle. This process takes a lot of time, effort, and costs. It also generates a huge amount of different kinds of knowledge that must be suitably managed. This paper describes a multiagent system in charge of managing this knowledge in order to improve the MP. The system has different types of agents in order to deal with the different types of information produced during SMP. Agents generate new knowledge and take advantage of the organization's experience. In order to facilitate the management of data and metadata or knowledge a XMI repositories have been used. XMI uses the MOF standard which enables the description of information at different levels of abstraction.

Acknowledgements This work is partially supported by the TAMANSI project (grant number PBC-02001) financed by the Consejería de Ciencia y Tecnología of the Junta de Comunidades de Castilla-La Mancha.

References [1] [2] [3]

[4] [5]

[6]

Bellifemine, A., Poggi, G., and Rimassa, G. (2001). Developing multi agent systems with a FIPA-compliant agent framework. Software Practise & Experience, (2001) 31: 103-128. Bennet K.H., and Rajlich V.T.(2000). Software Maintenance and Evolution: a Roadmap, in Finkelstein, A. (Ed.), The Future of Software Engineering, ICSE 2000, June 4-11, Limerick, Ireland, pp 75-87. Henninger, S., and Schlabach, J. (2001). A Tool for Managing Software Development Knowledge, 3ª International Conf. on Product Focused Software Process Improvement. PROFES 2001, Lecture Notes in Computer Science, Kaiserslautern, Germany, pp 182-195. de Looff, L.A., Information Systems Outsourcing Decision Making: a Managerial Approach. Hershey, PA: Idea Group Publishing, 1990. Mesenzani, M., Schael, T., and Alblino, S. (2002). Multimedia platform to support knowledge proceses anywhere and anytimeIn Knowledge-Based Intelligent Information Engineering Systems & Allied Technologies (KES 2002). Damiani, E., Howlett, R.J., Jain L.C., Ichalkaranje, N. (Eds.), pg 14341435. OMG XML Metadata Interchange (XMI), v. 1.1, Nov-2000.

A Multi-agent System for Knowledge Management in Software Maintenance

[7] [8] [9] [10]

421

Orton, J.D., and Weick, K.E. (1990) Loosely coupled systems: A reconceputalization. Academy of Management Review, 15(2), pp 203-223. Pigoski, T.M. (1997). Practical Software Maintenance. Best Practices for Managing Your Investment. Ed. John Wiley & Sons, USA, 1997. Polo, M., Piattini, M., Ruiz, F. and Calero, C.: Roles in the Maintenance Process. Software Engineering Notes; vol 24, nº 4, 84-86. ACM., (1999). Vizcaíno, A., Ruíz, F., Favela, J., and Piattini, M. A Multi-Agent Architecture for Knowledge Management in Software Maintenance. In Proceedings of International Workshop on Practical Applications of Agents and Multiagent Systems (IWPAAMS'02), Salamanca, Spain 23-25 October, (2002) 39-52.

D²G²A: A Distributed Double Guided Genetic Algorithm for Max_CSPs Sadok Bouamama, Boutheina Jlifi, and Khaled Ghédira SOI²E (ex URIASIS) SOI²E/ISG/Université Tunis, B. 204, Département d'Informatique 41 rue de la liberté, 2000 cité Bouchoucha. Tunisie [email protected] [email protected] [email protected]

Abstract. Inspired by the distributed guided genetic algorithm (DGGA), D²G²A is a new multi-agent approach, which addresses Maximal Constraint Satisfaction Problems (Max-CSP). GA efficiency provides good solution quality for Max_CSPs in one hand and benefits from multi-agent principles reducing GA temporal complexity. In addition to that the approach will be enhanced by a new parameter called guidance operator. The latter allows not only diversification but also an escaping from local optima. D²G²A and DGGA are been applied to a number of randomly generated Max_CSPs. In order to show D²G²A advantages, experimental comparison is provided. As well, guidance operator is experimentally outlined in order to determine its best given value. Keywords: Maximal constraint satisfaction problems, multi-agent systems, genetic algorithms, Min-conflict-heuristic, guidance operator

1

Introduction

The CSP formalism consists of variables associated with domains and constraints involving subsets of these variables. A CSP solution is an instantiation of all variables with values from their respective domains. The instantiation must satisfy all constraints. A CSP solution, as defined above, is costly to get and does not necessarily exist within every problem. In such cases, one had better search an instantiation of all variables that satisfies the maximal number of constraints. Such problems called Maximal CSPs and referred to as Max_CSPs, make up the framework to this paper. Max_CSPs have been dealt with by complete or incomplete methods. The first ones are able to provide an optimal solution. Unfortunately, the combinatorial explosion thwarts this advantage. The second ones, such as Genetic Algorithms [4] have the same property to avoid the trap of local optima. They also sacrifice completeness for efficiency. There is another incomplete but distributed method V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 422-429, 2003.  Springer-Verlag Berlin Heidelberg 2003

D²G²A: A Distributed Double Guided Genetic Algorithm for Max_CSPs

423

known as Distributed Simulated Annealing (DSA) that has been successfully applied to Max-CSP [2] As DSA outperforms the centralized Simulated Annealing in terms of optimality and quality, the same idea is adopted for Centralized Genetic Algorithms (CGAs), which are especially known to be expensive. The result was the distributed guided genetic algorithm for Max_CSPs [3]. This paper aims to enhance the DGGA in order to escape from local optima and for better search diversification. And it is organized as follows: The next subsection presents the Distributed Double Guided Genetic Algorithm: context and motivations, the basic concepts and global dynamic. The third section details both experimental design and results. Finally, concluding remarks and possible extensions to this work are proposed.

2

Distributed Double Guided Genetic Algorithm

2.1

Context and Motivations

The DGGA cannot be considered as a classic GA. In fact, in classic GAs the mutation aims to diversify a considered population and then to avoid the population degeneration [4]. In the DGGA, mutation operator is used improperly since it is considered as a betterment operator of the considered chromosome. However, if a gene value was inexistent in the population there is no mean to obtain it by cross-over process. Thus, it is sometimes necessary to have, a random mutation in order to generate the possibly missing gene values. The DGGA approach is a local search method. The first known improvement mechanism of local search is the diversification of the search process in order to escape from local optima [6]. No doubt, the simplest mechanism to diversify the search is to consider a noise part during the process. Otherwise the search process executes a random movement with probability p and follows the normal process with a probability 1-p [5]. In figure 1, an example of local optima attraction basin is introduced; in a maximization case, S2, which is a local maximum, is better than S1. The passing through S1 from S2, is considered as a solution destruction but give more chance to the search process to reach S3, the global optimum. For all these reasons, the new proposed approach is a DGGA enhanced by a random providing operator, which will be called guidance probability Pguid. Thus the D2G2A will possess (in addition to the cross-over and mutation operators, to the generation number and to the initialpopulation size) a guidance operator. This approach will conserve the same basic principles and the same agents structure used with in the DGGA approach [3].

Fig. 1. An example of attraction basin of local optima

424

1. 2. 3. 4. 5. 6. 7. 8. 9.

Sadok Bouamama et al.

m ← getMsg (mailBox) case m of optimization-process (sub-population) : apply-behavior (sub-population) take-into-account (chromosome) : population-pool ← population-pool ∪ {chromosome} inform-new-agent (Speciesnvc) : list-acquaintances ← list-acquaintances ∪ {Speciesnvc} stop-process : stop-behavior Fig. 2. Message processing relative to Speciesnvc

2.2

Global Dynamic

The Interface agent randomly generates the initial population and then partitions it into sub-populations accordingly to their specificities. After that the former creates Species agents to which it assigns the corresponding sub-populations. Then the Interface agent asks these Species agents to perform their optimization processes (figure 2 line 3). So, before starting its optimization process, i.e. its behavior (figure 3), each Specie agent, Speciesn, initializes all templates corresponding to its chromosomes (figure 3 line 3). After that it carries out its genetic process on its initial sub-population, i.e. the sub-population that the Interface agent has associated to it at the beginning. This process, which will be detailed in the following subsection, returns a sub-population “pop” ( figure 3 line 4) that has been submitted to the crossing and mutating steps only once, i.e. corresponding to one generation. For each chromosome of pop, Speciesn computes the number of violated constraints “nvc” (figure 3 line 6). Consequently, two cases may occur. The first one corresponds to a chromosome violating the same number of constraints of its parents. In this case, the chromosome replaces one of the latter randomly chosen (figure 3 line 8). In the second case is that this number (nvc) is different from (n), i.e, the specificity of the corresponding Speciesn. Then the chromosome is sent to another Speciesnvc (figure 3 line 10) if such an agent already exists, otherwise it is sent to the Interface agent (figure 4 line 11). The latter creates a new agent having nvc as specificity and transmits the quoted chromosome to it. Whenever a new Species agent is created, the Interface agent informs all the other agents about this creation (figure 2 line 7) and then asks the new Species to perform its optimization process (figure 2 line 3). Note that message processing is given a priority. So, whenever an agent receives a message, it stops its behavior, saves the context, updates its local knowledge, and restores the context before resuming its behavior. If all the Species agents did not meet a chromosome violating zero constraints at the end of their behavior, they successively transmit one of their randomly chosen chromosomes, linked to its specificity to the Interface agent. The latter determines and displays the best chromosome namely the one which violates the minimal number of constraints.Here we describe the syntax used in the figures: sendMsg (sender, receiver,‘message'): ‘message' is sent by “sender” to “receiver” getMsg (mailBox): retrieves the first message in mailBox.

D²G²A: A Distributed Double Guided Genetic Algorithm for Max_CSPs

Apply-behavior (initial-population) 1. init-local-knowledge 2. for i := 1 to number-of-generations do 3. template-updating (initial-population) 4. pop ← genetic-process (initial-population) 5. for each chromosome in pop do 6. nvc ← compute-violated-constraints (chromosome) 7. if (nvc = n) 8. then replace-by (chromosome) 9. else if exist-ag (Speciesnvc) 10. then sendMsg (Speciesn, Speciesnvc,'take-into-account ( chromosome)') 11. else sendMsg (Speciesn, Interface, ‘create-agent (chromosome)') 12. sendMsg (Speciesn, Interface, ‘result (one-chromosome, specificity)')

Fig. 3. Behavior relative to Speciesn

genetic process 1. mating-pool ← matching (population-pool) 2. template-updating (mating-pool) 3. offspring-pool-crossed ← crossing (mating-pool) 4. offspring-pool-mutated ← mutating (offspring-pool- crossed) 5. return offspring-pool-mutated

Fig. 4. Genetic process

Crossing (mating-pool) if (mating-pool size < 2) then return mating-pool for each pair in mating-pool do if (random [0,1] < Pcross) then offspring ← cross-over (first-pair, second-pair) nvc ← compute-violated-constraints (offspring) if (nvc = 0) then sendMsg (Speciesn, Interface, ‘Stop-process (offspring)') else offspring-pool ← offspring-pool ∪ {offspring} return offspring-pool

Fig. 5. Crossing process relative to Speciesn cross-over (chromosomei1, chromosomei2) 1. for j :=1 to size (chromosomei1) do 2. sum ← templatei1,j + templatei2,j 3. if (random-integer [0, sum – 1]< templatei1,j) 4. then genei3,j ← genei2,j 5. else genei3,j ← genei1,j 6. return chromosomei3

Fig. 6. Cross-over operator

425

426

Sadok Bouamama et al.

Guided_Mutation (chromosome i) 1. min-conflict-heuristic (chromosome i) 2. return chromosomei

Fig. 7. Guided Mutation relative to chromosomei Random_Mutation (chromosome i) 1. choose randomly a genei,j 2. choose randomly a value vi in domain of genei,j 3. value(genei,j) ← vi 4. return chromosomei

Fig. 8. Random Mutation relative to chromosomei mutating (offspring-pool) 1. for each chromosome in offspring-pool do 2. if (random [0,1]< Pmut) 3. if (random [0,1]< Pguid) 4. then Guided_Mutation (chromosome i) 5. else Random_Mutation(chromosome i ) 6. nvc* ← violated_constraints_number (chromosome i) 7. if (nvc* = 0) 8. then sendMsg (Speciesn, Interface, ‘stop-process (chromosomei)' ) 9. else offspring-pool-mutated offspring-pool-mutated ∪{chromosomei} 10. return offspring-pool-mutated

Fig. 9. New mutating sub-process relative to Speciesn min-conflict-heuristic (chromosomei) 1. δi,j ← max (templatei) /*δi,j is associated to genei,j which is in turn associated to the variable vj*/ 2. nvc* ← nc /* nc is the total number of constraints*/ 3. for each value in domain of vj do 4. nvc ← compute-violated-constraint (value) 5. if (nvc < nvc*) 6. then nvc* ← nvc 7. value* ← value 8. value (genei,j) ← value* 9. update (templatei) 10. return nvc*

Fig. 10. Min-conflict-heuristic relative to chromosomei

3

Experimentation

The goal of our experimentation is to compare two distributed implementation. The first referred to as Distributed Double Guided Genetic Algorithm (D²G²A). Whereas the second implementation is known as Distributed Guided Genetic Algorithm

D²G²A: A Distributed Double Guided Genetic Algorithm for Max_CSPs

427

(DGGA). The implementations have been done with ACTALK [1], a concurrent object language implemented above the Object Oriented language SMALLTALK-80. 3.1

Experimental Design

Our experiments are performed on binary CSP-samples randomly generated. The generation is guided by classical CSP parameters: number of variables (n), domain size (d), constraint density p (a number between 0 and 100% indicating the ratio between the number of the problem effective constraints to the number of all possible constraints, i.e., a complete constraint graph) and constraint tightness q (a number between 0 and 100% indicating the ratio between the number of forbidden pairs of values (not allowed) by the constraint to the size of the domain cross product). As numerical values, we use n = 20, d = 20. Having chosen the following values 0.1, 0.3, 0.5, 0.7, 0.9 for the parameters p and q, we obtain 25 density-tightness combinations. For each combination, we randomly generate 30 examples. Therefore, we have 750 examples. Moreover and considering the random aspect of genetic algorithms, we have performed 10 experimentations per example and taken the average without considering outliers. For each combination density-tightness, we also take the average of the 30 generated examples. Regarding GA parameters, all implementations use a number of generations (NG) equal to 10, an initial-population size equal to 1000, a cross-over probability equal to 0.5, a mutation probability equal to 0.2 and a random replacement. The performance is assessed by the two following measures: -

Run time: the CPU time requested for solving a problem instance, Satisfaction: the number of satisfied constraints.

The first one shows the complexity whereas the second receals the quality. In order to have a quick and clear comparison of the relative performance of the two approaches, we compute ratios of DGGA and D²G²A performance using the Run time and the satisfaction, as follows: -

CPU-ratio = DGGA-Run-time/D²G²A-Run-time Satisfaction-ratio = D²G²A Satisfaction/DGGA Satisfaction.

Thus, DGGA performance is the numerator when measuring the CPU time ratios, and the denominator when measuring satisfaction ratio. Then, any number greater than 1 indicates superior performance by D²G²A. 3.2

Experimental Results

Figure11 shows the performance ratios from which we draw out the following results: -

-

From the CPU time point of view, D²G²A requires lesser time for the overconstrained and most strongly tight set of examples. For the remaining set of examples the CPU-time ratio is always over 1. In average, this ratio is equal to 2.014. From the satisfaction point of view, the D²G²A always finds more or same satisfaction than DGGA. It finds about 1.23 times more for the most strongly

428

Sadok Bouamama et al.

constrained and most weakly tight set of problems. The satisfaction ratios average is about 1.05 Note that the experiment do not show a clear dependency between NG and the evolution of both CPU-time and satisfaction ratios [3]. 3.3

Guidance Operator Study

Attention is next focused on the guidance probability study in order to determine its best values. To accomplish this task, we have assembled the CPU-time averages and satisfaction averages for different values of Pguid. Both figure 12 shows that the best value of Pguid is about 0.5. This value can be explained by the fact that not only random mutations are important but also guided mutations. In fact the guided mutating sub-process allows guidance and so helps the algorithm to reach optima, and the random one helps it to escape from local optima attraction basin.

4

Conclusion and Perspective

We have developed a new approach called D²G²A which is a distributed guided genetic algorithm enhanced by a new parameter called guidance probability. compared to the DGGA, our approach has been experimentally shown to be always better in terms of satisfied constraints number and CPU time.

20

400

15

300

CPU time

satisfied constraint number

Fig. 11. CPU-time ratio and Satisfaction ratio

10 5 0

200 100 0

0

0,2

0,4

0,6

guidance probability

0,8

1

0

0,2

0,4

0,6

guidance probability

Fig. 12. CPU-time and Satisfaction relative to different values of Pguid

0,8

1

D²G²A: A Distributed Double Guided Genetic Algorithm for Max_CSPs

429

The improvement is due to diversification, which increases the algorithm convergence by escaping from local optima attraction basin. And to guidance which helps the algorithm to attain optima. So that our approach gives more chance to the optimization process to visit all the search space. We have come to this conclusion thanks to the proposed mutation sub process. The latter is sometimes random, aiming to diversify the search process, and sometimes guided in order to increase the number of satisfied constraints. The guidance operator has been tested, too. Its best given value has been about 0.5. No doubt further refinement of this approach would allow its performance to be improved. Further works could be focused on applying this approach to solve other hard problems like valued CSPs and CSOPs.

References [1]

[2]

[3] [4] [5] [6]

BRIOT J.P., “actalk: a testbed for classifying and designing Actor languages in the Smalltalk-80 Environment”, Proceedings of European Conference on Object-Oriented Programming (ECOOP'89), British Computer Society Workshop Series, July 1989, Cambridge University Press. GHÉDIRA K., “A Distributed approach to Partial Constraint Satisfaction Problem”, Lecture Notes in Artificial Intelligence, Distributed software Agents and Applications, number 1069, 1996, John W. Perram and J.P Muller Edts, Springer Verlag, Berlin Heidelberg. GHÉDIRA K & JLIFI B. “ A Distributed Guided Genetic Algorithm for Max_CSPs” journal of sciences and technologies of information (RSTI), journal of artificial intelligence series (RIA), volume 16 N°3/2002. GOLDBERG D.E, Genetic algorithms in search, Optimisation, and Machine Learning, Reading, Mass, Addison-Wesley, 1989. T. SCHIEX, H. FARGIER & G. VERFAILLIE, “Valued constrained satisfaction problems: hard and easy problems, proceeding of the 14th IJCAI, Montreal, Canada august 1995. TSANG E.P.K, WANG C.J., DAVENPORT A., VOUDOURIS C., LAU T.L, “A family of stochastic methods for Constraint Satisfaction and Optimization”, University of Essex, Colchester, UK, November 1999.

Modelling Intelligent Agents for Organisational Memories Alvaro E. Arenas and Gareth Barrera-Sanabria Laboratorio de C´ omputo Especializado Universidad Aut´ onoma de Bucaramanga Calle 48 No 39 - 234, Bucaramanga, Colombia {aearenas,gbarrera}@unab.edu.co

Abstract. In this paper we study the modelling of intelligent agents to manage organisational memories. We apply the MAS-CommonKADS methodology to the development of an agent-based knowledge management system applied to project administration in a research centre. Particular emphasis is made on the development of the expertise model, where knowledge is expressed as concepts of the CommonKADS Conceptual Model Language. These concepts are used to generate annotated XML-documents, facilitating their search and retrieval. Keywords: Agent-Oriented Software Engineering; Knowledge Management; Organisational Memory; MAS-CommonKADS

1

Introduction

An organisational memory is an explicit, disembodied and persistent representation of crucial knowledge and information in an organisation. This type of memory is used to facilitate access, sharing and reuse by members of the organisation for individual or collective tasks [3]. In this paper we describe the development of a multi-agent system for managing a document-based organisational memory distributed through the Web using the MAS-CommonKADS methodology [5]. MAS-CommonKADS is a general purpose multi-agent analysis and design methodology that extends the CommonKADS [10] design method by combining techniques from object-oriented methodologies and protocol engineering. It has been successfully applied to the optimisation of industrial systems [6], the automation of travel assistants [1] and the development of e-commerce applications [2], among others. In this paper particular emphasis is made on the development of the expertise model, where knowledge is expressed as concepts of the CommonKADS Conceptual Model Language. These concepts are used to generate annotated XML documents, which are central elements of the organisational memory, thus facilitating their search and retrieval.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 430–437, 2003. c Springer-Verlag Berlin Heidelberg 2003


Modelling Intelligent Agents for Organisational Memories

2

431

Applying MAS-CommonKADS to the Development of Organisational Memories

The concepts we develop here are applicable to any organisation made of distributed teams of people who work together online. In particular, we have selected a research centre made of teams working on particular areas as our case study. A team consists of researchers who belong to particular research projects. The organisational memory is made of electronic documents distributed throughout the Intranet of that centre. The MAS-CommonKADS methodology starts with a conceptualisation phase where an elicitation task is carried out, aiming to obtain a general description of the problem by following a user-centred approach based on use cases [9]. We have identified three human actors for our system: Document-Author, End-User and Administrator. A Document-Author is a person working in a research project and who is designated for updating information about such a project. An EndUser is any person who makes requests to the system; requests could be related to a team, a project or a particular researcher. For instance, a typical request is ‘Find the deliverables of the project named ISOA’. Finally, an Administrator is the person in charge of maintaining the organisational memory. The outcome of this phase is a description of the different actors and their use cases. 2.1

The Agent Model

The agent model specifies the characteristics of an agent, and plays the role of a reference point for the other models. An agent is defined as any entity —human or software— capable of carrying out an activity. The identification of agents is based on the actors and use cases diagrams that are generated in the conceptualisation. We have identified four classes of agents: User Agent, software agent corresponding to an interface agent that helps to carry out the actions associated with human actors; it has three subclasses, one for each actor: Author Agent, End-User Agent and Administrator Agent. Search Agent, software agent responsible for searching information in the organisational memory; it has three subclasses, one for each kind of search: Search Team Agent, Search Project Agent and Search Researcher Agent. Update Agent, software agent responsible for updating information in the organisational memory; it has three subclasses, one for kind of update: Update Team Agent, Update Project Agent, Update Researcher Agent. Create Agent, software agent responsible for creating new information in the organisational memory; it has three subclasses, one for each possible creation: Create Team Agent, Create Project Agent, Create Researcher Agent. 2.2

The Task Model

The task model describes the tasks that an agent can carry out. Tasks are decomposed according to a top-down approach. We use UML activity diagrams

432

Alvaro E. Arenas and Gareth Barrera-Sanabria

Table 1. Textual Template for Organise Search Criteria Task Task Organise Search Criteria Purpose: Classify the search criteria Description: Once the criteria are received, they are classified and the information to which they will be applied is identified Input: General criteria Output: Criteria classified Precondition: Existence of criteria Ingredient: Pattern of criteria classification

to represent the activity flow of the tasks and textual templates to describe each activity. For instance, in the case of Search Agent, it carries out tasks such as request information from other agents, organise search criteria, select information and notify its status to other agents. Table 1 shows the textual template of task Organise Search Criteria. 2.3

The Organisation Model

This model aims to specify the structural relationships between human and/or software agents, and the relationships with the environment. It includes two submodels: the human organisation model and the multi-agent organisation model. The human organisation model describes the current situation of the organisation. It includes a description of different constituents of the organisation such as structure, functions, context and processes. As a way of illustration, the process constituent for our case study includes the reception of information about new teams, projects or researchers; once the information is validated, it is registered in data bases and new pages are generated in the organisation’s Intranet; such information can be updated by modifying the corresponding records; finally, the information is offered to the different users. The multi-agent organisation model describes the structural organisation of the multi-agent system, the inheritance relationship between agents and their structural constituent. The structural organisation and the inheritance relationships are derived directly from the agent model. The structural constituent of the multi-agent system specifies the geographic distribution of the agents and the organisational memory. Figure 1 presents an abstract diagram of the structural constituent of our case study. 2.4

The Coordination and Communication Models

The coordination model shows the dynamic relationships between the agents. This model begins with the identification of the conversations between agents, where use cases play again an important role. At this level, every conversation consists of just one single interaction and the possible answer, which are

Modelling Intelligent Agents for Organisational Memories

433

Fig. 1. Abstract Diagram of the Structural Constituent of the MAS

described by means of textual templates. Next, the data exchanged in each interaction are modelled by specifying speech acts and synchronisation, and collect all this information in the form of sequence diagrams. Finally, interactions are analysed in order to determine their complexity. We do not emphasize this phase due to lack of space. The communication model includes interactions between human agents and other agents. We use templates similar to those of the coordination model, but taking into consideration human factors such as facilities for understanding recommendations given by the system. 2.5

The Expertise Model

The expertise model describes the reasoning capabilities of the agents needed to carry out specific tasks and achieve goals. It consists of the development of the domain knowledge, inference knowledge and task knowledge. The domain knowledge represents the declarative knowledge of the problem, modelled as concepts, properties, expressions and relationships using the Conceptual Modelling Language (CML) [10]. In our problem, we have identified concepts such as User, Project, Team, Profile, Product and properties such as the description of the products generated by a project, the description of the profile of an user. Table 2 presents a fragment of the concept definition of our system. Such a description constitutes the general ontology of the system, it is an essential component for generating the XML-annotated documents that make the organisational memory. The inference knowledge describes the inference steps performed for solving a task. To do so, we use inference diagrams [10]. An inference diagram consists of three elements: entities, representing external sources of information that are necessary to perform the inference, which are denoted by boxes; inferences, denoted by ovals and flow of information between entities and inferences, denoted

434

Alvaro E. Arenas and Gareth Barrera-Sanabria

Table 2. Fragment of the Concept Definition of the System Concept User Description: Metaconcept that represents the type of users in the system. It does not have specific properties Concept: External User Description: Subconcept of concept User. This concept represents the external users of the system, i.e. people form other organisations who want to consult our system Concept: Employee Description: Subconcept of concept User that details the general information of the employees of the organisation Properties: Identity: String ; Min = 8 ; Max = 15 Name : String; Max = 100 Login: String; Min = 6 ; Max = 8 Type: Integer ; Max = 2 ...

by arrows. Typical processes that require inference steps include validating the privileges of a user or the search of information in the organisational memory. Figure 2 shows the inference diagram for validating the privileges of a user. The task knowledge identifies the exact knowledge required by each task and determines its most suitable solving method. To do so, each task is decomposed in simpler subtasks, from which we extract relevant knowledge to structure the organisational memory. 2.6

The Design Model

This model aims to structure the components of the system. It consists of three submodels: the agent network, the agent design and the platform submodels. The agent network design model describes the network facilities (naming services, security, encryption), knowledge facilities (ontology servers, knowledge representation translators) and the coordination facilities (protocol servers, group management facilities, resource allocation) within the target system. In our system, we do not use any facilitator agent, so that this model is not defined. The agent design model determines the most suitable architecture for each agent. To do so, we use a template for each agent type that describes the functionalities of each subsystem of such an agent type. Finally, in the platform design model, the software and hardware platform for each agent are selected and described. We have selected Java as the programming language for the implementation of the agent subsystems. The documents that make the organisational memory are represented as XML documents. We use JDOM [4], an API for accessing, manipulating, and outputting XML data from

Modelling Intelligent Agents for Organisational Memories

435

Fig. 2. Inference Diagram for Validating the Privileges of a User

Java code, to program the search and retrieval of documents. Lastly, Java Server Pages (JSP) is used for implementing the system dynamic pages.

3

Implementation

The implementation of our system consisted of three phases: creation of a document based organisational memory, development of user interfaces and construction of the multi-agent system. The organisational memory consisted of XML-annotated documents that can be updated and consulted by the users according to their privileges. In the current implementation, such documents are generated straightforwardly from the concepts and relationships of the domain knowledge. The structure of the user interfaces is derived directly from the communication model and developed with JSP. In the current version of the system, each agent type is implemented as a Java class. We are implementing a new version with the aid of the AgentBuilder tool.

4

Conclusions and Future Work

The agent-based approach to system development offers a natural means of conceptualising, designing and building distributed systems. The successful practice of this approach requires robust methodologies for agent-oriented software

436

Alvaro E. Arenas and Gareth Barrera-Sanabria

engineering. This paper applies MAS-CommonKADS, a methodology for agentbased system development, to the development of a knowledge management system for administrating a document-based organisational memory distributed throughout the Web. We have found the application of MAS-CommonKADS useful in this type of applications. First, the organisational model was crucial in the specification of the relationship between system and society, which is a central aspect in this type of applications. In our application, it was useful for defining the structure of the information within the Intranet of the organisation. Second, the knowledge submodel of the expertise model was the basis for creating the organisational memory. From such concepts we derive XML-annotated documents that make the organisational memory. Several projects have studied ontologies for both knowledge management and Web search. The OSIRIX project also proposes the use of XML-annotated documents to build an organisational memory [8]. They develop a tool that allows users to translate a corporate ontology into a document type definition, provided that the ontology is represented in the CommonKADS Conceptual Modelling Language. Although we share similar objectives, our work was focused on the methodological aspects to develop this kind of systems. Klein et al study the relation between ontologies and XML for data exchange [7]. They have devised a procedure for translating an OIL ontology into a specific XML schema. Intended future work will focus on the automation of the generation of the XML-documents, and the inclusion of domain-specific ontologies in the system.

Acknowledgements The authors are grateful to Jos´e P´erez-Alc´azar and Juan Carlos Garc´ıa-Ojeda for valuable comments and suggestions. This work was partially funded by the Colombian Research Council (Colciencias-BID).

References [1] A. E. Arenas and G. Barrera-Sanabria. Applying the MAS-CommonKADS Methodology to the Flights Reservation Problem: Integrating Coordination and Expertise. Frontiers of Artificial Intelligence and Applications Series, 80:3–12, 2002. 430 [2] A. E. Arenas, N. Casas, and D. Quintanilla. Integrating a Consumer Buying Behaviour Model into the Development of Agent-Based E-Commerce Systems. In IIWAS 2002, The Fourth International Conference on Information Integration and Web-Based Applications and Services. SCS, European-Publishing House, 2002. 430 [3] R. Dieng. Knowledge Management and the Internet. IEEE Intelligent Systems, 15(4):14–17, 2000. 430 [4] E. R. Harold. Processing XML with Java. Addison-Wesley, 2002. 434 [5] C. A. Iglesias, M. Garijo, J. Centeno-Gonzalez, and J. R. Velasco. Analysis and Design of Multiagent Systems using MAS-CommonKADS. In Agent Theories, Architectures, and Languages, Lecture Notes in Artificial Intelligence, pages 313– 327, 1997. 430

Modelling Intelligent Agents for Organisational Memories

437

[6] C. A. Iglesias, M. Garijo, M. Gonz´ alez, and J. R. Velasco. A Fuzzy-Neural Multiagent System for Optimisation of a Roll-Mill Application. In IEA/AIE, Lecture Notes in Artificial Intelligence, pages 596–605, 1998. 430 [7] M. Klein, D. Fensel, F. van Harmelen, and I. Horrocks. The Relation between Ontologies and XML Schemas. Electronic Transactions on Artificial Intelligence, 6(4), 2001. 436 [8] A. Rabarijoana, R. Dieng, O. Corby, and R. Ouaddari. Building and Searching an XML-Based Corporate Memory. IEEE Intelligent Systems, 15(3):56–63, 2000. 436 [9] J. Rumbaugh, I. Jacobson, and G. Booch. The Unified Modelling Language Reference Manual. Addison Wesley, 1999. 431 [10] G. Schreiber, H. Akkermans, A. Anjewierden, R. de Hoog, N. Shadbolt, W. Vand de Velde, and B. Wielinga. Knowledge Engineering and Management: The CommonKADS Methodology. The MIT Press, 2000. 430, 433

A Study on the Multi-agent Approach to Large Complex Systems Huaglory Tianfield School of Computing and Mathematical Sciences Glasgow Caledonian University, 70 Cowcaddens Road, Glasgow G4 0BA, UK [email protected]

Abstract. This paper explores the multi-agent approach to large complex systems. Firstly, system complexity research is briefly reviewed. Then, self-organization of multi-agent systems is analyzed. Thirdly, complexity approximation via multi-agent systems is presented. Finally, distinctive features of multi-agent approach are summarized.

1

Introduction

Large complex systems (LCS) are found in abundance. Practically all disciplines ranging from natural sciences to engineering, from management/decision/behavior sciences to social sciences have encountered various LCS. Various natural and manmade instances of LCS are, e.g., biological body systems, biological society systems, environmental and weather systems, space/universe systems, machine systems, traffic/transport systems, human organizational/societal (e.g., politic, economic, military) systems, to just mention a few. Complexity research and its applications have received very broad attention from different disciplines. Particularly, the different models employed have largely determined the approaches to LCS, as depicted in Table 1. An LCS may have hundreds or even thousands of heterogeneous components, between which are complicated interactions ranging from primitive data exchanges to natural language communications. The information flows and patterns of dynamics in an LCS are intricate. Generally LCS have three intrinsic attributes, i.e. perceptiondecision link, multiple gradation, and nesting [1-3]. As emergent behaviors dominate the system, study of behaviors of LCS is very hard. Given that System Geometry ∆∇= components + interaction between components

(1)

where ∆∇= denotes “packaging” and “de-packaging”, system complexity is much more attributable to the interactions between components than to the components themselves. The architecture study is very hard. Apparently there is certainly a hierarchy in an LCS, but it is unclear how many levels there should be and how a level can be stratified. And integration of LCS is very hard. Fundamentally there is even no effective approach. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 438-444, 2003.  Springer-Verlag Berlin Heidelberg 2003

A Study on the Multi-agent Approach to Large Complex Systems

439

Table 1. Models in complexity research Model Statistical models

Originating area Thermodynamics

Biology

Characteristics of complexity Entropy, complexity measure, dissipative structure, synergetics, selforganization Chaos, fractal geometry, bifurcation, catastrophe, hypercylce Evolution, learning, emergence

Deterministic models Biological species models Game theoretical models Artificial life, adaptive models

Non-linear science

Operations research

Game, dynamic game, co-ordination

Cybernetics, adaptation

Co-evolution, cellular automata, replicator, self-organized criticality, edge of chaos Time series dynamics, black-box quantitative dynamics, system controllability, systems observeability Hierarchy, self-organizing, adaptation, architecture of complexity, systems dynamics To be explored

Time-series dynamic models

System science and engineering

Macroscopic dynamic models

Economics and sociology, systems research

Multi-agent models

Distributed artificial intelligence

All of these make it rather difficult to investigate the basic problems of LCS, including complexity mechanism, modeling, architecture, self-organization, evolution, performance evaluation, system development, etc. Intelligent agents and multi-agents systems have been widely accepted as a type of effective coarse-granularity metaphors for perception, modeling and decision making, particularly in systems where humans are integrated [4, 5]. They will be very effective in dealing with the heterogeneous natures of the components of LCS as intelligent agents and their emergent interaction are ubiquitous modeling metaphors whether the real-world systems are hardware or software, machines or humans, low or high level.

2

Self-Organization of Multi-agent Systems

2.1

Geometry of Multi-agent Systems

Prominent advantage of agentization is that it enables a separation of social behavior (problem solving, decision making and reasoning) at social communication level from individual behavior (routine data and knowledge processing, and problem solving) at autonomy level. Due to such a separation of the two levels, every time a social communication is conducted, the corresponding agents have to be associated beforehand. Therefore, social communication level and autonomy level, along with agent-task association, predominate the geometry of a multi-agent system on the basis of its infrastructure, as depicted in Fig. 1. Infrastructure of a multi-agent system includes agents, agent platform(s), agent management, and agent communication language.

440

Huaglory Tianfield

System Geometry Infrastructure ACL Social Communication Level

Agent Platform

Agent-Task Association

Registration Autonomy level Agent

Agent

Agent

Fig. 1. Geometry of multi-agent system (ACL: agent communication language)

Agent-task association is dynamic. The dynamics can be as long as a complete round of problem solving, and can also be as short as a round of social communication, or even as a piece of social communication (exchange of one phrase). Social communication is emergent. It is purely a case-by-case process. It is nonpredictable beforehand and is only appearing in the situation. Moreover, social communication is at knowledge level. Agents themselves have sufficient abilities. The purpose of social communication is not solving problems, but initiating, activating, triggering, invoking, inducing, and prompting the abilities of individual agents. However, the essential decision making is relying upon the corresponding agents, not the social communication between agents. 2.2

Uncertainty and Self-Exploration of Multi-agent Problem Solving Process

Multi-agent problem solving process involves three aspects: •

Agent-task association: which agents are associated, which are not, and the alternation and mixture of association and non-association in problem solving; • Social communication: organizationally constrained and non-constrained social communication, and their alternation and mixture in problem solving; • Progression of problem solving: what tasks or intermediate solutions are achieved, how they are aggregated, what are the further tasks to achieve the complete solution. The autonomy of agents, the dynamic property of agent-task association, and the emergent and knowledge level properties of social communication between agents make that multi-agent problem solving process is uncertain and self-exploring. Multi-agent problem solving process is self-exploring. At the beginning, the solving process is unknown. With the progression of problem solving, the process is made clearer and clearer.

A Study on the Multi-agent Approach to Large Complex Systems

441

In redundant multi-agent systems, agents to be associated vary uncertainly from one round or even one piece of social communication to another. Overlapping between the abilities of individual agents in a society forms the basis of the uncertainty of multi-agent problem solving process. In non-redundant multi-agent systems, agents are functionally fixed under a given organizational structure, social communication is organizationally constrained, and there is no provision of competition or alternation between agents. The total problem solving is just a mechanical additive aggregation of the abilities of individual agents over an uncertain process. This appears as if the problem solving of individual agents forms a Markov chain. t

t

1 2 Total task = PS(Agent-1) + PS(Agent-2) + ··· +

tN

PS(Agent-N)

(2)

t

where PS is abbreviation of problem solving, and “ + ” denotes a mechanical aggregation of results from the problem solving of individual agents which takes place at asynchronous instants. However, the social communication in non-redundant multi-agent systems still varies in time, content and/or quality from one round or even one piece to another. So, even in this case the multi-agent problem solving process is uncertain. For instance, within an enterprise, even provided that there is an adequate organizational structure between functional departments, from one time to another, functional departments may have failures in delivery dates and qualities, depending on how cooperation and/or negotiations are made between functional departments. Therefore, in both redundant and non-redundant multi-agent systems, multi-agent problem solving process is uncertain.

3

Approximating Complexity through Flexible Links of Simplicities

3.1

Flexibility of Multi-agent Modeling of Real-World Systems

Consider ∆

A system = ((Agent - i)N)T ( ⇔ ijt )NxN (Agent - j)N)

(3)

where parentheses denote a vector or matrix, superscript T refers to transposition of a vector, ⇔ ijt denotes social communication from agent j to agent i at asynchronous instant t. It can refer to pattern, discourse, or phrase of social communication, or all of these. Due to the dynamic property of agent-task association, the number of agents to be associated, N, varies from one round, or even one piece of social communication to another. The basic idea of multi-agent based modeling is packaging various real-world LCS into multi-agent systems, as depicted in Fig. 2. Thus Equation (1) becomes Large Complex System Geometry ∆∇ = Agents + Interactions between agents

(4)

442

Huaglory Tianfield

Unified Systems Engineering Process: Architecture, Analysis, Design, Construction, Performance Evaluation, (Co-)Evolution, etc. Multi-agent Version of Systems Engineering Process

Multi-agent Systems

De-packaging

Packaging

Non-life Living Living Non-life

Real-world Version of Systems Engineering Process

Natural Various Real-World Large Complex Systems Man-made

Fig. 2. Multi-agent systems as metaphor of various real-world LCS

Then, multi-agent modeling involves specifications upon packaging, agents, agent communication language, and agent platform. This apparently requires a specification language. Multi-agent packaging has many prominences, e.g., • • •

Unified systems engineering process of LCS: architecture, analysis, design, construction, performance evaluation, evolution, etc.; Unified concepts, measure, theory, methods, etc. of LCS; Unified studies on different types of real-world LCS.

Given a fixed infrastructure, the self-organization of multi-agent systems makes it possible that a multi-agent system can adapt itself for a wide variety of real-world systems. 3.2

Multi-agent Based System Development

Principal impact of multi-agent paradigm upon system development is the transition from individual focus to interaction focus. High level social behavior may result from a group of low level individuals. Developing systems by multi-agent paradigm immediately turns out developing the corresponding multi-agent systems. To develop a multi-agent system, there are two parts to complete, i.e., first to develop the infrastructure of the multi-agent system, including agents, agent platform(s), agent management, and agent communication language, and then to develop the agent-task association and the social communication. For the former, the development of infrastructure of multi-agent system is just similar to that of any traditional systems. Traditional paradigms of system development, including analysis/design methods and life-cycle models are applicable.

A Study on the Multi-agent Approach to Large Complex Systems

443

Table 2. Multi-agent approach versus traditional complexity research Scope

Destination

Highlight

Conventional complexity research Non-linearity, chaos, selforganization, quantum mechanics, thermodynamics, etc. Basically, physical mechanism oriented, i.e., trying to discover the primary physics of non-linear and thermodynamic complex systems More on microscopic/quantum level, ignoring effectively incorporating humans / computers in the systems, and incapable of dealing with the genuine largeness and structural complexity of a system

Multi-agent approach Very natural to incorporate humans/computers as basic components of the systems, and able to deal with the largeness and architecture of systems Not physical mechanism oriented, but black-box systematic behavior oriented Macroscopic, meta-level, i.e., using coarse-granularity (pseudo-) computational entities (i.e., agents) as the primary elements of characteristics and modeling of LCS. Power is not on the primary elements themselves, but on the dynamic, uncertain, emergent, knowledge-level interactions between these primary elements

For the latter, exactly speaking, the agent-task association and the social communication are not something that can be developed, except influenced. In such a circumstance, traditional paradigms are generally inapplicable. Actually, there is no apparent life-cycle concept. Developers are interactively working with the multi-agent system. In order to exert influence, extra requirements are posed upon the development of infrastructure of multi-agent system. That is, when infrastructure of multi-agent system is developed, consideration should already be given about how they can be influenced later on. Essentially, the geometry of a multi-agent system should have been designed before the development of it infrastructure. By means of multi-agent paradigm, infrastructure of multi-agent systems can be very easily inherited from one system to another. This prominent provision greatly facilitates and alleviates reusability and evolvability of systems.

4

Summary

Advantages of self-organization of multi-agent systems include: • • •

System modeling becomes greatly alleviated. Given a fixed infrastructure, the multi-agent system can adapt itself for many varieties of real-world systems; Completeness and organic links between system perspectives are readily guaranteed; Reusability and evolvability of systems is thoroughly enhanced. Provided with the infrastructure of a multi-agent system, new systems are rapid to be available, considerably short time to market;

444

Huaglory Tianfield

•

Late user requirement can be readily resolved and adapted. Actually users are interactively working with the multi-agent system. Users can change (exactly, influence) the behavior of the multi-agent system. Disadvantages of multi-agent systems may be: • Multi-agent problem solving process is uncertain and self-exploring. These leave the problem solving process to be nontransparent and untraceable to users; • While efficiency of multi-agent problem solving can be influenced by incorporating heuristics into the multi-agent system, the speed of problem solving is uncontrollable.

The traditional models/methods for LCS are relatively dependent upon the originating area, and lack a unified research process for systems from different variety of domains. Multi-agent systems provide a unified approach to the analysis, design, construction, performance, evolution, etc. of LCS. A comparison is made as in Table 2.

References [1] [2] [3] [4] [5]

Tianfield, H.: Formalized analysis of structural characteristics of large complex systems. IEEE Trans. on Systems, Man, and Cybernetics, Part A, 31 (2001) 559-572 Tianfield, H.: Structuring of large-scale complex hybrid systems: From illustrative analysis toward modeling. J. of Intelligent and Robotic Systems: Theory and Applications, 30 (2001) 179-208 Tianfield, H.: An innovative tutorial on large complex systems. Artificial Intelligence Review, 17 (2002) 141-165 Tianfield, H.: Agentization and coordination. Int. J. of Software Engineering and Knowledge Engineering, 11 (2001) 1-5 Tianfield, H.: Towards advanced applications of agent technology. Int. J. of Knowledge-based Intelligent Engineering Systems, 5 (2001) 258-259

Multi-layered Distributed Agent Ontology for Soft Computing Systems Rajiv Khosla School of Business La Trobe University, Melbourne, Victoria – 3086, Australia [email protected]

Abstract. In this paper we outline a multi-layered distributed agent ontology for providing system modeling support to practitioners and problem solvers at four levels, namely, distributed, tool (technology), optimization and task level respectively. We describe the emergent characteristics of the architecture from a architectural viewpoint. We also outline how the ontology facilitates development of humancentered soft computing systems. The ontology has been applied in a range areas including alarm processing, web mining, image processing, sales recruitment and medical diagnosis. We also outline definition of agents in different layers of the ontology.

1

Introduction

Soft computing agents today are being applied in a range of areas including image processing, engineering, process control, data mining , internet and others. In the process of applying soft computing agents to complex real world problems three phenomena have emerged. Firstly, the application of soft computing agents in distributed environments has resulted in merger of techniques from soft computing area with those in distributed artificial intelligence. Secondly, in an effort to improve the quality of soft computing solution researchers have been combining and fusing technologies. This has resulted in hybrid configurations of soft computing technologies. Finally, from a practitioners perspective, as the soft computing technologies have moved out of laboratories into the real world the need for knowledge or task level practitioner-centered technology independent constructs has emerged. In this paper we describe four levels of intelligent soft computing agent design which correspond to the three phenomena described above. These four levels or layers are distributed, tool, optimization and problem solving or task layer. The distributed level support is provided among other aspects for fetching and depositing data across different machines in a distributed environment. Tool support is provided in terms of applying various soft and hard computing technologies like fuzzy logic, neural networks, genetic algorithms, and knowledge based systems. Optimization level support is provided in terms of hybrid configurations of soft computing technologies for designing and developing optimum models. The optimization level V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 445-452, 2003.  Springer-Verlag Berlin Heidelberg 2003

446

Rajiv Khosla

assists in improving the quality of solution of the technologies in the tool layer and range of tasks covered by them. . Finally, task level support is provided in terms of modelling user's tasks and problem solving models in a technology independent manner. The layered architecture is also motivated by the human-centred approach and criteria outlined in the 1997 NSF workshop on human-centered systems and consistent problem solving structures/strategies employed by practitioners while designing solutions to complex problems or situations. The paper is organised as follows. Section 2 describes work done in the process of developing the ontology. Section 3 outlines the multi-layered agent ontology for soft computing systems and some aspects of the emergent behavior or characteristics of multi-layered ontology. Section 4 concludes the paper.

2

Background

In this section we firstly look at evolution of intelligent technologies along two dimensions, namely, quality of solution and range of tasks. Then we outline some of the existing work on problem solving ontologies. 2.1

Evolution of Intelligent Technologies

The four most commonly used intelligent technologies are symbolic knowledge based systems (e.g. expert systems artificial neural networks, fuzzy systems and genetic algorithms) [3, 5, 8].The computational and practical issues associated with intelligent technologies have led researchers to start hybridizing various technologies in order to overcome their limitations. However, the evolution of hybrid systems is not only an outcome of the practical problems encountered by these intelligent methodologies but is also an outcome of deliberative, fuzzy, reactive, self-organizing and evolutionary aspects of the human information processing system [2]. Hybrid systems can be grouped into three classes, namely, fusion systems, transformation systems, combination systems [3, 6, 7]. These classes along with individual technologies are shown in Fig. 1 along two dimensions, namely, quality of solution and range of tasks. In fusion systems, the representation and/or information processing features of in technology A are fused into the representation structure of another technology B. From a practical viewpoint, this augmentation can be seen as a way by which a technology addresses its weaknesses and exploits its existing strengths to solve a particular real-world problem. Transformation systems are used to transform one form of representation into another. They are used to alleviate the knowledge acquisition problem. For example, neural nets are used for transforming numerical/continuous data into symbolic rules which can then be used by a symbolic knowledge based system for further processing. Combination system involve explicit hybridization. Instead of fusion, they model the different levels of information processing and intelligence by using technologies that best model a particular level. These systems involve a modular arrangement of two or more technologies to solve real-world problems.

Multi-layered Distributed Agent Ontology for Soft Computing Systems

447

Associate Systems

Fusion Systems Transformation Systems

Quality of Solution

Symbolic AI

Fuzzy System

Genetic Algorithm

Neural Networks

Combination Systems

Range of Tasks

Fig. 1. Technologies, Hybrid Configurations, Quality of Solution and Range of Tasks

However, these hybrid architectures also suffer from some drawbacks. These drawbacks can be explained in terms of the quality of solution and range of tasks covered as shown in Fig. 1. Fusion and transformation architectures on their own do not capture all aspects of human cognition related to problem solving. For example, fusion architectures result in conversion of explicit knowledge into implicit knowledge, and as a result lose on the declarative aspects of problem solving Thus, they are restricted in terms of the range of tasks covered by them. The transformation architectures with bottom-up strategy get into problems with increasing task complexity. Therefore the quality of solution suffers when there is heavy overlap between variables, where the rules are very complicated, the quality of data is poor, or data is noisy. The combination architectures cover a range of tasks because of their inherent flexibility in terms of selection of two or more technologies. However, because of lack of (or minimal) knowledge transfer among different modules the quality of solution suffers for the very reasons the fusion and transformation architectures are used. It is useful to associate these architectures in a manner so as to maximize the quality as well as range of tasks that can be covered. These class of systems are called associative systems as shown in Fig. 1. As may be apparent from Fig. 1, associative systems consider various technologies and their hybrid configurations as technological primitives that are used to accomplish tasks. The selection of these technological primitives is contingent upon satisfaction of task constraints. In summary, it can seen from the discussion in this section that associative systems represent evolution from a technology-centered approach to a task-centered approach 2.2

Strengths and Weaknesses of Existing Problem Solving Ontologies

In order to pursue the task-centered approach one tends to look into work done in the area of problem solving ontologies. The research on problem solving ontologies or knowledge-use level architectures has largely been done in artificial intelligence. The research at the other end of the spectrum (e.g., radical connectionism, soft computing)

448

Rajiv Khosla

is based more on understanding the nature of human or animal behavior rather than developing ontologies for dealing with complex real world problems on the web as well as in conventional applications. A discussion on strengths and some limitations of existing problem solving ontologies can be found in [7, 8]. Some of limitations include lack of modelling constructs for soft computing applications, humancenteredness, context, response time, external or perceptual representations and some others. Firstly, especially with the advent of the internet and the web human (or user)centeredness has become an important issue (NSF workshop on human-centered systems 1997) and Takagi [10]. Secondly, from a cognitive science viewpoint, distributed cognition (which among aspects involves consideration of external and internal representations for task modelling) has emerged as a system modelling paradigm as compared to the traditional cognitive science approach based on internal representations only. Thirdly, Human-centered problem solving among other aspects involves, context, focus on practitioner goals and tasks, human evaluation of tasks modelled by technological artifacts, flexibility, adaptability and learning. Fourthly, unlike knowledge based systems, soft computing systems because of imprecise and approximate nature require modelling of constructs like optimization . Finally, as outlined in the last section there are a range of hard and soft computing technologies and their hybrid configurations which lend flexibility to the problem solver as against trying to force fit a particular technology or method on to a system design (as traditionally has been done). These aspects can be considered as pragmatic constraints. Most existing approaches do not meet one or more of the pragmatic constraints. Besides, from a soft computing perspective most existing do not facilitate component based modelling at optimization, task and technology levels respectively.

3

Distributed Multi-layered Agent Ontology for Soft Computing Systems

The multi-layered agent ontology is shown in Fig. 2. It is derived from integration of characteristics of intelligent artifacts like fuzzy logic, neural network and genetic algorithm, agents, objects and distributed process model with problem solving ontology model [6, 7]. It consists of five layers, namely, the object layer, which defines the data architecture or structural content of an application. The tool or technology agent layer defines the constructs for various intelligent and soft computing tools. The optimization agent layer defines constructs for fusion, combination and transformation technologies which are used for optimizing the quality of solution (e.g., accuracy). Finally, the problem solving ontology (task) agent layer defines the constructs related to the problem solving agents namely, preprocessing, decomposition, control, decision and postprocessing. This layer models practitioners tasks in the domain under study. Some generic goals and tasks associated with each problem solving agent are shown in Table 1. This layer employs the services of the other 4 layers for accomplishing various tasks. The five layers facilitate a component based approach for agent based software design .

Multi-layered Distributed Agent Ontology for Soft Computing Systems

Problem Solving (Task) Agent Layer Optimization Agent Layer Tool Agent Layer Expert System Agent

Global Preprocessing Phase Agent

Fusion Agent

Fuzzy Logic Agent

Distributed Comm. & Processing Agent1

Distributed Agent Layer Distri-

Object Layer

buted Comm. & Processing Agent N

Supervised Neural Network Agent

SelfOrganising Agent

Assoc iative Agent

Postprocessing Phase Agent

Transformation Agent

Genetic Algorithm Agent

Decomposition Phase Agent

Decision Phase Agent

Combination Agent Control Phase Agent

Fig. 2. Multi-Layered Distributed Agent Ontology for Soft Computing

Table 1. Some Goals and Tasks of Problem Solving Agents Phase

Goal

Preprocessing

Improve data quality

Decomposition

Restrict the context of the input from the environment at the global level. By defining a set of orthogonal concepts Reduce the complexity and enhance overall reliability of the computer-based artifact

Control

Determine decision selection knowledge constructs within an orthogonal concept for the problem under study .

Decision

Post-processing

Provide decision instance results in a user defined decision concept. Establish outcomes as desired outcomes

Some Tasks Noise Filtering Input Conditioning

Define orthogonal concepts

Define decision level concepts with in each orthogonal concept as identified by users Determine Conflict Resolution rules Define decision instance of interest to the user Concept validation Decision instance result validation

449

450

N U M B E R o f P A T H S

Rajiv Khosla D ia g n o sis to T re a tm e n t D ia g n o sis to S ym p tom s a n d T re a tm e nt S ym p tom s to D ia g n o sis to T re a tm e nt to

S y ste m D e co m p o sitio n

T re a tm e n t D e c isio n S yste m P o stp ro c e s sin g

D e c o m p o sitio n S y ste m

D ia g n o stic D e cisio n D e co m p o sitio n

T rea tm e n t D e cision C o n tro l

P o stp ro ce ssin g D ia g n o stic D e cisio n T re a tm e n t D e c isio n P o stp ro c e s sin g

D e c is io n P a th s

Fig. 3. Decision Paths Taken by a Medical Practitioner and Agent Sequence In t e l l i g e n t C o n t r o l A g e n t ( I C A ) N eu ra l N e tw o rk A g en t

W a te r Im m e rs io n S e g m e n t a t i o n IP A

M om en t In v a ria n t T ra n sfo rm a tio n A g en t

M a t h e m a t ic a l M o r p h o lo g y S e g m e n ta tio n a g e n t 1 N

Fig. 4. Optimization Level Modelling of a Image Processing Application

3.1

Some Emerging Characteristics of Soft Computing Agent Ontology

In this section we establish the human-centeredness of the multi-layered agent ontology from two perspectives. Firstly, the multi-layered agent ontology facilitates human involvement, evaluation and feedback at three levels, namely, task, optimization and technology respectively. The constructs used at the task level are based on consistent problem solving structures employed by users in developing solutions to complex problems. Thus the five problem solving agents facilitate mapping of user's tasks in a systematic and scalable fashion [7]. Further, the component based nature of the five problem solving agents allows them to be used in different problem solving contexts as shown in Fig. 4. These sequences assist in modeling different problem solving or situational contexts. The optimization level allows an existing technology based solution to be optimized based on human feedback. For example, in an unstained mammalian cell image processing application we employ neural network (as shown in Fig. 4) for prediction and optimization of segmentation quality of segmented images by morphological and water immersion agents. At the technology level human evaluation and feedback is modelled by asking user to define the search parameters or fitness

Multi-layered Distributed Agent Ontology for Soft Computing Systems

451

function to be used by a soft computing agent like GA to create the correct combination of shirt design [7]. Further, the generic tasks of the five problem solving agents have been grounded in human-centered criteria outlined in the 1997 NSF workshop on human-centered systems. These criteria state that a) human-centered research and design is problem driven rather than logic theory or any particular technology; b) human-centered research and design focuses on practitioner's goals and tasks rather than system developer's goals and tasks, and c) human-centered research and design is context bound. The context criterion relates to social/organizational context and representational context (where people use perceptual as well as internal representations to solve problems) and problem solving context. The generic tasks employed by the five problem solving agents are based on consistent problem solving structures used by practitioners solving complex problems in engineering, process control, image processing, management and other areas. The component based nature problem solving agents shown in Fig. 2 enables them to be used in different sequences and different problem solving contexts. Further, the representation (external and internal representations) and social and organization context (not described in this paper) have also been modelled [6,7]. Additionally, the five layers of the agent ontology lead to component based distributed (collaboration and competition) soft computing system design. The ontology also provides flexibility of technologies, learning and adaptation, different forms of knowledge (fuzzy, rule based and distributed pattern based) to be used for modelling component based software design.

4

Conclusion

In this paper we have outlined a multi-layered distributed agent ontology for developing soft computing applications. The ontology provides modelling support at four levels, namely, task level, optimization level, technology level, and distributed processing level. Further, it takes into consideration pragmatic constraints like human-centeredness, context, distributed cognition, and flexibility of technologies. The ontology has been applied in a range of areas including web mining, image processing, e-commerce, alarm processing, medical diagnosis and others

References [1] [2] [3]

Bezdek, J.C. “Pattern Recognition with Fuzzy Objective Function Algorithms,' Advanced Applications in Pattern Recognition,” Plenum Press 1981, USA. Bezdek, J.C., “What is Computational Intelligence?' Computational Intelligence: Imitating Life, Eds. Robert Marks-II et al., IEEE Press, New York, 1994. Chiaberage, M., Bene. G.D., Pascoli, S.D., Lazzerini, B., and Maggiore, A. (1995), “Mixing fuzzy, neural & genetic algorithms in integrated design environment for intelligent controllers,” 1995 IEEE Int Conf on SMC,. Vol. 4, pp. 2988-93.

452

Rajiv Khosla

[4]

Khosla, R., and Dillon, T., “Learning Knowledge and Strategy of a Generic Neuro-Expert System Architecture in Alarm Processing”, in IEEE Transactions on Power Systems, Vol. 12, No. 12, pp. 1610-18, Dec.1997. Khosla, R., and Dillon T., Engineering Intelligent Hybrid Multi-Agent Systems, Kluwer Academic Publishers, MA, USA August 1997. Khosla, R., Sethi, I. and Damiani, E., Intelligent Multimedia Multi-Agent Systems – A Human-Centered Approach, Kluwer Academic Publishers, MA, USA October 2000. Khosla, R. Damiani, E., and Grosky, W. Human-Centered E-Business, Kluwer Academic Publishers, MA, USA, April 2003 . Goldberg, D.E. (1989), Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley, Reading, MA, pp. Takagi, H.K. (2001) “Interactive Evolutionary Computation: Fusion of the Capabilities of EC Optimization and Human Evaluation,” Proceedings of the IEEE, vol. 89, No. 9, September. Zhang, J., Norman, D. A. (1994), "Distributed Cognitive Tasks", Cognitive Science, pp. 84-120.

[5] [6] [7] [8] [9] [10]

A Model for Personality and Emotion Simulation Arjan Egges, Sumedha Kshirsagar, and Nadia Magnenat-Thalmann MIRALab - University of Geneva 24, Rue General-Dufour,1211 Geneva, Switzerland {egges,sumedha,thalmann}@miralab.unige.ch http://www.miralab.ch

Abstract. This paper describes a generic model for personality, mood and emotion simulation for conversational virtual humans. We present a generic model for describing and updating the parameters related to emotional behaviour. Also, this paper explores how existing theories for appraisal can be integrated into the framework. Finally we describe a prototype system that uses the described models in combination with a dialogue system and a talking head with synchronised speech and facial expressions.

1

Introduction

With the emergence of 3D graphics, we are now able to create very believable 3D characters that can move and talk. However, an important part often missing in this picture is the definition of the force that drives these characters: the individuality. In this paper we explore the structure of this entity as well as its link with perception, dialogue and expression. In emotion simulation research so far, appraisal is popularly done by a system based on the OCC model [1]. This model specifies how events, agents and objects from the universe are appraised according to an individual’s goals, standards and attitudes. These three (partly domain-dependent) parameters determine the ‘personality’ of the individual. More recent research indicates that personality can be modelled in a more abstract, domain-independent way [2, 3]. In this paper we will investigate the relationship between such personality models and the OCC model. The effect of personality and emotion on agent behaviour has been researched quite a lot, whether it concerns a general influence on behaviour [4], or a more traditional planning-based method [5]. Various rule based models [6] and probabilistic models [7] have been reported in the past. How behaviour should be influenced by personality and emotion depends on the system that is used and it is out of the scope of this paper. The effect of personality and emotion will also have an effect on expression (speech intonation, facial expressions, etc.) is partly deatl with in Kshirsagar et al. in [8], which describes a system that simulates personalized facial animation with speech and expressions, modulated through mood. There have been very few researchers who have tried to simulate mood [9, 10].

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 453–461, 2003. c Springer-Verlag Berlin Heidelberg 2003


454

Arjan Egges et al.

Fig. 1. Overview of the emotional state and personality in an intelligent agent framework Figure 1 depicts how we view the role of personality and emotion as a glue between perception, dialogue and expression. Perceptive data is interpreted on an emotional level by an appraisal model. This results in an emotion influence that determines, together with the personality what the new emotional state and mood will be. An intelligent agent uses the emotional state, mood and the personality to create behaviour. This paper is organized as follows. Section 2 and Section 3 present the generic personality, mood and emotion model. In Section 4, we investigate the relationship between the OCC model and personality. Finally, we present an example of how to use our model with an expressive virtual character1.

2

A Generic Personality and Emotion Model

The intention of this section is to introduce the concepts that we will use in our model. In the next section we will explain how these concepts interact. We will first start by defining a small scenario. Julie is standing outside. She has to carry a heavy box upstairs. A passing man offers to help her carry the box upstairs. Julies personality has a big influence on her perception and on her behaviour. If she has an extravert personality, she will be happy that someone offers her some help. If she has a highly introvert and/or neurotic personality, she will feel fear and distress and she will respond differently. As someone is never 100% extravert or 100% neurotic, each personality factor will have a weight in determining how something is perceived and what decisions are being taken. Surely, Julies behaviour will not only be based 1

Please read Egges et al. [11] and Kshirsagar et al. [12, 13] for more details.

A Model for Personality and Emotion Simulation

455

on emotional concepts, but also on intellectual concepts. A dialogue system or intelligent agent will require concrete representations of concepts such as personality, mood and emotional state so that it can decide what behaviour it will portray [4, 14]. As such, we need to define exactly what we mean by personality, mood, and emotion before we can simulate emotional perception, behaviour and expression. 2.1

Basic Definitions

An individual is an entity that is constantly changing. So, when we speak of an individual, we always refer to it relative to a time t. The moment that the individual starts existing is defined by t = 0. The abstract entity that represents the individual at a time t we will call It from now on. An individual has a personality and an emotional state (we do not yet take mood into consideration). The model based on this assumption is called PE. The personality is constant and initialized with a set of values on t = 0. The emotional state is dynamic and it is initialized to 0 at t = 0. Thus we define It as a tuple (p, et ), where p represents the personality and et represents the emotional state at time t. In our example, Julie will portray emotions (that change over time) based on what happens, but how she obtains these emotions and the behaviour that results from it, depends on a static part of her being, the personality. There exist many personality models, each of them consisting of a set of dimensions, where every dimension is a specific property of the personality. Take for example the OCEAN model [3], which has five dimensions (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) or the PEN model [2] that has three dimensions. Generalizing from these theories, we assume that a personality has n dimensions, where each dimension is represented by a value in the interval [0, 1]. A value of 0 corresponds to an absence of the dimension in the personality; a value of 1 corresponds to a maximum presence of the dimension in the personality. The personality p of an individual can then be represented by the following vector: 
pT = α1 . . . αn , ∀i ∈ [1, n] : αi ∈ [0, 1]

(1)

Emotional state has a similar structure as personality, but it changes over time. The emotional state is a set of emotions that have a certain intensity. For example, the OCC model defines 22 emotions, while Ekman [15] defines 6 emotions for facial expression classification. We define the emotional state et as an m-dimensional vector, where all m emotion intensities are represented by a value in the interval [0, 1]. A value of 0 corresponds to an absence of the emotion; a value of 1 corresponds to a maximum intensity of the emotion. This vector is given as follows:   β . . . β 1 m , ∀i ∈ [1, m] : βi ∈ [0, 1] if t > 0 eTt = (2) 0 if t = 0

456

Arjan Egges et al.

Furthermore, we define an emotional state history ωt that contains all emotional states until et , thus: ωt = e0 , e1 , . . . , et 

(3)

An extended version of the PE model, the PME model, is given by including mood. We now define the individual It as a triple (p, mt , et ), where mt represents the mood at a time t. We define mood as a rather static state of being, that is less static than personality and less fluent than emotions [8]. Mood can be one-dimensional (being in a good or a bad mood) or perhaps multi-dimensional (feeling in love, being paranoid). Whether or not it is justified from a psychological perspective to have a multi-dimensional mood is not in the scope of this paper. However, to increase generality, we will provide for a possibility of having multiple mood dimensions. We define a mood dimension as a value in the interval [−1, 1]. Supposing that there are k mood dimensions, the mood can be described as follows:   , ∀i ∈ [1, k] : γi ∈ [−1, 1] if t > 0 . . . γ γ 1 k (4) mTt = 0 if t = 0 Just like for the emotional state, there is also a history of mood, σt , that contains the moods m0 until mt : σt = m0 , m1 , . . . , mt 

3

(5)

Emotional State and Mood Updating

From perceptive input, an appraisal model such as OCC will obtain emotional information. This information is then used to update the mood and emotional state. We define the emotional information as a desired change in emotion intensity for each emotion, defined by a value in the interval [0, 1]. The emotion information vector a (or emotion influence) contains the desired change of intensity for each of the m emotions: 
aT = δ1 . . . δm , ∀i ∈ [1, m] : δi ∈ [0, 1]

(6)

The emotional state can then be updated using a function Ψe (p, ωt , a). This function calculates, based on the personality p, the current emotional state history ωt and the emotion influence a, the change of the emotional state. A second part of the emotion update is done by another function, Ωe (p, ωt ) that represents internal change (such as a decay of the emotional state). Given these two components, the new emotional state et+1 can be calculated as follows: et+1 = et + Ψe (p, ωt , a) + Ωe (p, ωt )

(7)

A Model for Personality and Emotion Simulation

457

In the PME model (which includes the mood), the update process slightly changes. When an emotion influence has to be processed, the update now happens in two steps. The first step consists in updating the mood; the second step consists of updating the emotional state. The mood is updated by a function Ψm (p, ωt , σt , a) that calculates the mood change, based on the personality, the emotional state history, the mood history and the emotion influence. The mood is internally updated using a function Ωm (p, ωt , σt ). Thus the new mood mt+1 can be calculated as follows: mt+1 = mt + Ψm (p, ωt , σt , a) + Ωm (p, ωt , σt )

(8)

The emotional state can then be updated by an extended function Ψe that also takes into account the mood history and the new mood. The internal emotion update, which now also takes mood into account, is defined as Ωe (p, ωt , σt+1 ). The new emotion update function is given as follows: et+1 = et + Ψe (p, ωt , σt+1 , a) + Ωe (p, ωt , σt+1 )

4

(9)

Personality, Emotion and the OCC Model of Appraisal

As OCC is the most widely accepted appraisal model, we will present some thoughts about how to integrate personality models with OCC. OCC uses goals, standards and attitudes. These three notions are for a large part domain dependent. As multi-dimensional personality models are domain-independent, we need to define the relationship between this kind of personality model and the personality model as it is used in OCC. We will choose an approach where we assume that the goals, standards and attitudes in the OCC model are fully defined depending on the domain. Our personality model will then serve as a selection criterion that indicates what and how many goals, structures and attitudes fit with the personality. Because the OCEAN model is widely accepted, we will use this model to illustrate our approach. For an overview of the relationship between OCEAN and the goals, standards, and attitudes, see Table 1. The intensity of each personality faction will determine the final effect on the goals, standards and attitudes.

5 5.1

Application Overview

In order to demonstrate the model in a conversational context, we have built a conversational agent represented by a talking head. The update mechanisms of emotions and mood are implemented using a linear approach (using simple matrix representations and computations). The center of the application is a dialogue system, using Finite State Machines, that generates different responses

458

Arjan Egges et al.

Table 1. Relationship between OCEAN and OCC parameters Goals Openness

Standards Attitudes making a shift of attitude towards standards in new sit- new elements uations

Conscientiousness abandoning and adopting goals, determination Extraversion Agreeableness

willingness to communicate abandoning and compromising stan- adaptiveness to adopting goals in dards in favor of oth- other people favor of others ers

Neuroticism

based on the personality, mood and emotional state. The personality and emotion model is implemented using the 5 factors of the OCEAN model of personality, one mood dimension (a good-bad mood axis) and the 22 emotions from OCC plus 2 additional emotions (disgust and surprise) to have maximum flexibility of facial expression. A visual front-end uses the dialogue output to generate speech and facial expressions. The dialogue system annotates its outputs with emotional information. An example of such a response is (the tags consist of a joy emotion percentage of 58): |JO58|Thanks! I like you too! 5.2

Visual Front-End

Our visual front-end comprises of a 3D talking head capable of rendering speech and facial expressions in synchrony with synthetic speech. The facial animation system interprets the emotional tags in the responses, generates lip movements for the speech and blends the appropriate expressions for rendering in real-time with lip synchronization2 . Facial dynamics are considered during the expression change, and appropriate temporal transition functions are selected for facial animation. We use MPEG-4 Facial Animation Parameters as low level facial deformation parameters [12]. However, for defining the visemes and expressions, we use the Principal Components (PCs) [13]. The PCs are derived from the statistical analysis of the facial motion data and reflect independent facial movements observed during fluent speech. The use of PCs facilitates realistic speech animation, especially blended with various facial expressions. We use available text-to-speech (TTS) software that provides phonemes with temporal information. The co-articulation rules are applied based on the algorithm of Cohen et al. [16]. The expressions are embedded in the text in terms of tags 2

For more details on expressive speech animation, see Kshirsagar et al. [13].

A Model for Personality and Emotion Simulation

459

Fig. 2. Julie’s behaviour as an extravert (a) and an introvert (b) personality

and associated intensities. An attack-sustain-decay-release type of envelope is applied for these expressions and they are blended with the previously calculated co-articulated phoneme trajectories. Periodic eye-blinks and minor head movements are applied to the face for increased believability. Periodic display of facial expression is also incorporated, that depends on the recent expression displayed with the speech, as well as the mood of the character. 5.3

Example Interaction

As an example, we have developed a small interaction system that simulates Julie’s behaviour. We have performed these simulations with different personalities, which gives different results in the interaction and the expressions that the face is showing. The interaction that takes place for an extravert personality (90% extravert) is shown in Figure 2(b). An introvert personality (5% extravert) significantly changes the interaction (see Figure 2(b)).

6

Conclusions and Future Work

In this paper we have presented a basic framework for personality and emotion simulation. Subsequently we have shown how this framework can be integrated with an application such as an expressive MPEG-4 talking head with speech synthesis. Our future work will focus on user studies to validate the personality

460

Arjan Egges et al.

and emotion model that is used. We will also explore the possibility of having multiple mood dimensions. Furthermore we will explore how personality and emotion are linked to body behaviour and what computational methods are required to simulate this link.

Acknowledgment This research has been funded through the European Project MUHCI (HPRNCT-2000-00111) by the Swiss Federal Office for Education and Science (OFES).

References [1] Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press (1988) 453 [2] Eysenck, H.J.: Biological dimensions of personality. In Pervin, L.A., ed.: Handbook of personality: Theory and research. New York: Guilford (1990) 244–276 453, 455 [3] Costa, P.T., McCrae, R.R.: Normal personality assessment in clinical practice: The NEO personality inventory. Psychological Assessment (1992) 5–13 453, 455 [4] Marsella, S., Gratch, J.: A step towards irrationality: Using emotion to change belief. In: Proceedings of the 1st International Joint Conference on Autonomous Agents and Multi-Agent Systems, Bologna, Italy (2002) 453, 455 [5] Johns, M., Silverman, B.G.: How emotions and personality effect the utility of alternative decisions: a terrorist target selection case study. In: Tenth Conference On Computer Generated Forces and Behavioral Representation. (2001) 453 [6] Andr´e, E., Klesen, M., Gebhard, P., Allen, S., Rist, T.: Integrating models of personality and emotions into lifelike characters. In: Proceedings International Workshop on Affect in Interactions. Towards a New Generation of Interfaces. (1999) 453 [7] Ball, G., Breese, J.: Emotion and personality in a conversational character. In: Proceedings of the Workshop on Embodied Conversational Characters. (1998) 83–84 and 119–121 453 [8] Kshirsagar, S., Magnenat-Thalmann, N.: A multilayer personality model. In: Proceedings of 2nd International Symposium on Smart Graphics, ACM Press (2002) 107–115 453, 456 [9] Vel´ asquez, J.: Modeling emotions and other motivations in synthetic agents. In: Proceedings of AAAI-97, MIT Press (1997) 10–15 453 [10] Moffat, D.: Personality parameters and programs. In: Lecture Notes in Artificial Intelligence : Creating Personalities for Synthetic Actors: Towards Autonomous Personality Agents. (1995) 453 [11] Egges, A., Kshirsagar, S., Zhang, X., Magnenat-Thalmann, N.: Emotional communication with virtual humans. In: The 9th International Conference on Multimedia Modelling. (2003) 454 [12] Kshirsagar, S., Garchery, S., Magnenat-Thalmann, N.: Feature Point Based Mesh Deformation Applied to MPEG-4 Facial Animation. In: Deformable Avatars. Kluwer Academic Publishers (2001) 33–43 454, 458

A Model for Personality and Emotion Simulation

461

[13] Kshirsagar, S., Molet, T., Magnenat-Thalmann, N.: Principal components of expressive speech animation. In: Proceedings Computer Graphics International. (2001) 59–69 454, 458 [14] Elliott, C.D.: The affective reasoner: a process model of emotions in a multiagent system. PhD thesis, Northwestern University, Evanston, Illinois (1992) 455 [15] Ekman, P.: Emotion in the Human Face. Cambridge University Press, New York (1982) 455 [16] Cohen, M.M., Massaro, D. In: Modelling co-articulation in synthetic visual speech. Springer-Verlag (1993) 139–156 458

Using Loose and Tight Bounds to Mine Frequent Itemsets Lei Jia, Jun Yao, and Renqing Pei School of Mechatronics and Automation Shanghai University, Shanghai, 200072, China {jialei7829,junyao0529,prq44}@hotmail.com

Abstract. Mining frequent itemsets forms a core operation in many data mining problems. The operation, however, is data intensive and produces a large output. Furthermore, we also have to scan the database many times. In this paper, we propose to use loose and tight bounds to mine frequent itemsets. We use loose bounds to remove the candidate itemsets whose support cannot satisfy the preset threshold. Then, we find whether we can determine the frequency of the remainder candidate itemsets with the tight bounds. According to the itemsets that cannot be treated, we scan the database for them. Using this new method, we can decrease not only the candidate frequent itemsets have to be tested, but also the database scan times.

1

Introduction

Data mining is to efficiently discover interesting rules from large collections of data. Mining frequent itemsets forms a core operation in many data mining problems. The frequent itemset problem is stated as follows. Assume we have a finite set of items I. A transaction is a subset of I, together with a unique identifier. A database D is a finite set of transactions. A subset of I is called an itemset. The support of an itemset equals to the fraction of the transactions in D that contains it. If the support is above the preset threshold, we call it frequent itemset. Many researchers devote to designing efficient structures or algorithms [1,2,3,4,5,6,7,8] to mine frequent itemsets. We find, in these algorithms, we often spend a lot of time in scanning the database. If we can optimize the scanning processes, we will improve the efficiency of the mining dramatically. In this paper, unlike the existing algorithms, we propose to use loose and tight bounds to mine the frequent itemsets. We use loose bounds to remove the candidate itemsets whose support cannot satisfy the preset threshold. Then, we find whether we can determine the frequency of the remainder candidate itemsets with the tight bounds. According to the itemsets that cannot be treated, we scan the database for them. In section 2, we introduce loose bounds and tight bounds. In section 3, we show how to use loose and tight bounds to mine frequent itemsets by LTB algorithm (loose V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 462-468, 2003.  Springer-Verlag Berlin Heidelberg 2003

Using Loose and Tight Bounds to Mine Frequent Itemsets

463

and tight bounds based algorithm). We present the experiment result in section 4 and section 5 is a brief conclusion.

2

Loose Bounds and Tight Bounds

2.1

Loose Bounds

Suppose we have two itemsets A,B and their support sup(A), sup(B). Minsup is the preset support threshold. According to the Apriori-trick, if at least one of them is not frequent, we should not consider AB. If A and B are both frequent, we have to generate and test the candidate frequent itemsets AB. Not only do we generate it, but also scan the database to determine its frequency. We know sup(AB) ∈ [max(0,sup(A)+sup(B)-1), min(sup(A),sup(B))]. Because we have already known sup(A) and sup(B) at the last level, we can easily induce it is a frequent itemset if max(0,sup(A)+sup(B)-1) ≥ minsup. In the same way, if min(sup(A),sup(B))<minsup, we confirm it is not a frequent itemset. We call this bound is a loose bound. We define a symbol ∇ to denote the cross union between the itemsets A and B. Given a database D, s1 and s2 as the minimum and maximum support level, we define Fs as the itemsets with the frequency above s, Fs1,s2 to be frequent itemsets occurring at least s1, but less than s2 percent of transactions in D. We can get Fs1Fs2 ⊆ Fs1,s2, also can we get Fs1,s2 ∪ Fs2 ⊆ Fs1. Now we can use the definition to the loose bound. We let ∆ s=s1+s2-1, if ∆ s 〉 0,

then Fs1,s2 ∇ Fs3,s4 ⊆ F ∆s ,min(s2,s4). For example, if sup(A)=60%, sup(B)=70%, sup(C)=10%, sup(D)=30%, minsup=20%. We can easily get 30% ≤ sup(AB) ≤ 60%, 0 ≤ sup(AC) ≤ 10%, 0 ≤ sup(AD) ≤ 30%. So we conclude that itemset AB must be frequent itemsets, AC cannot be frequent itemset and we don't know AD is frequent or not. In the existing algorithm, we have to scan the database for the itemsets like AB and AD. But now, we can use the following tight bounds to determine frequency of the itemset like AB. 2.2

Tight Bounds

We find, using the tight bounds [9,10], the numbers of database scans and the itemsets that have to be counted can be reduced significantly. Consider the following example. Suppose we know: sup(A)=sup(B)=sup(C)=2/3, sup(AB)=sup(AC)=sup(BC)= 1/3 and minsup=1/3, we cannot judge the itemsets ABC whether it is frequent or not with the idea in Apriori-like algorithm. However, we can calculate the frequency of it with the following inequalities.

464

Lei Jia et al.

We get the following results:

So we conclude that sup(ABC)=0. We find we can determine the frequency without scanning the database. This method is called tight bounds. We define tx as the frequency of x in the transactions if transaction.item=x. For example: tAB means the frequency of tranactions contains AB in the database and it is independent to tA and tB. Then sup(A)=tA+tAB+tAC+…+tABC+tABD+…+tI. Then we have the following equalities.

We know any one of the tx is above zero. So we can solve the above equalities recursively from (1) to (N) to get the following solutions.

We find if we know all the subsets of the itemsets I, we can accurately calculate the lower and upper limits of its support with above solutions. If the lower and upper limits are the same, we can determine its frequency without scanning the databases. We prune the superset of I with the downward-closed property (Apriori-trick). When we cannot decide the support just because the low and upper limits of the tight bounds are not the same, we have to scan the database for them. Also do we scan database for the itemsets we cannot decide like AD in the last subsection.

Using Loose and Tight Bounds to Mine Frequent Itemsets

3

465

LTB Algorithm (Loose and Tight Bounds Based Algorithm)

Through the above analysis, we find if we make full use of the loose and tight bounds, we can decrease the number of the itemsets that need to be tested and the times we have to scan the database. So we propose the LTB algorithm in this section. LTB algorithm: Input: database D, minsup and t (the highest level we use the loose bounds); Output: Frequent itemsets L; L= φ ; L1={frequent 1-itemsets}; L1=order L1 according to their frequency in ascend order; 4) T[1]=get_filtered(D,L1); 5) for(k=2,Lk-1 ≠ φ ,k++) 6) Ck=apriori_gen(Lk-1); 7) if(k ≤ t) 8) use the loose bounds to get L' whose support is above minsup; 9) L''=Ck-L'; 10) Calculate the support of itemsets M1(M1 ⊆ L') with the tight bounds; 11) if the low and upper limits are the same; 12) M2=L'-M1; 13) Lk=Lk ∪ M1; 14) end; 15) Scan the database T[1] for the itemsets x (x ⊆ (L'' ∪ M2)); 16) if sup(x) ≥ minsup 17) Lk=Lk ∪ x; 18) else 19) Calculate the support of itemsets in Ck with the tight bounds; 20) if the low and upper limits of M3(M3 ⊆ Ck) are the same; 21) M4=Ck-M3; 22) Lk=Lk ∪ M3; 23) end; 24) Scan the database T[1] for the itemsets y in M4; 25) If sup(y) ≥ minsup 26) Lk=Lk ∪ y; 27) end 28) end Answer L= ∪ kLk; 1) 2) 3)

The algorithm can be divided into three parts. From line 1 to line 4, we find the frequent itemsets at level 1 and use them to filter the database D to T [1]. If some database has infrequent 1-item, this filtering can shorten the database significiently.

466

Lei Jia et al.

From line 5 to 17, we use the loose and tight bounds to mine the frequent itemsets below level t where t is the preset highest level we use loose bound. We utilize the apriori_gen function as the traditional Apriori Algorithm to make full use of the downward-closed property to prune the search space. We use the loose bounds to select the itemsets that is needed to calculate their frequency with the tight bounds. Some of these have the same low and upper limits, so we can determine its support. We scan the database for the others and the itemsets we cannot determine whether it is frequent or not by loose bounds. The last part is from line 15 to the end. In this part, because the loose bounds is too loose, we only calculate their supports with tight bounds. We scan the database again if we cannot determine the support of them. Using this algorithm, we can use the loose and tight bounds to mine frequent itemsets efficiently.

4

Experiment

To study the effectiveness and efficiency of the algorithm we proposed in the above section, we implemented it in VB and tested on a 1GHz Pentium PC computer with 128 megabytes of the main memory. The test consists of a set of synthetic transaction database generated using a randomized itemset generation algorithm similar to the algorithm described in [2]. Table 1 shows the database used, in which N is the average size of these itemsets and T is the size of item sets we randomly choose from. Table 1.

Database 1 2

N 2 3

T

Size of transactions 100K 100K

5

N=2,MinSup=20%

N=3,MinSup=20%

5000

15000 Apriori LTB

Apriori LTB

4500 4000 3500

10000 Time(ms)

Time(ms)

3000 2500 2000

5000

1500 1000 500 0

0

10

20

30

40 50 60 70 Size of Transactions(k)

80

90

100

Fig. 1. Database1, N=2, MinSup=20%

0

0

10

20

30

40 50 60 70 Size of Transactions(k)

80

90

100

Fig. 2. Database2, N=3, MinSup=20%

Using Loose and Tight Bounds to Mine Frequent Itemsets N=2,MinSup=40%

N=3,MinSup=40%

3500

7000 Apriori LTB

3000

6000

2500

5000

2000

4000

Time(ms)

Time(ms)

Apriori LTB

1500

3000

1000

2000

500

1000

0

467

0

10

20

30

40 50 60 70 Size of Transactions(k)

80

90

100

Fig. 3. Database1, N=2, MinSup=40%

0

0

10

20

30

40 50 60 70 Size of Transactions(k)

80

90

100

Fig. 4. Database2, N=3,MinSup=40%

We compare our LTB algorithm with the classic Apriori algorithm [2]. Figure 1 and Figure 2 show the running time of the two approaches in relevance to the number of transactions in database1 and database2 where the support threshold MinSup is 20%. In the same way, we get figure 3 and figure 4 respectively when we change the MinSup to 40%. When n=2, LTB algorithm is about 6% (MinSup=20%) or 3% (MinSup=40%) less efficient than Aprori algorithm. The reason is that we cannot find the itemsets that could satisfy the tight bounds. We have to scan the database for them all. We waste the time we spend in calculating the tight bounds. When n=3, LTB approach is about 20% (MinSup=20%) or 12% (MinSup=40%) more efficient just because we can determine the support of some itemsets without scanning the database. In practice, there are many candidate frequent itemsets and the database is very large. So, we can save more time if we use LTB algorithm.

5

Conclusion

We propose to use a new method, loose and tight bounds, to mine frequent itemsets in this paper. The algorithm we propose is called LTB algorithm (loose bounds and tight bounds based algorithm). The loose bounds can help us narrow the search space and the tight bounds can assist us to get the frequency of the itemsets without scanning the database. We also use the order and apriori-trick in the algorithm to make the mining process efficiently. The detail experiments prove the effectiveness of our method.

References [1] [2]

R.Agrawal, T.Imielinski and A.Swami. Mining association rules between sets of items in large databases. SIGMOD (1993) 207–216 R.Agrawal and R.Srikant. Fast slgorithms for mining association rules. VLDB (1994) 487–499

468

Lei Jia et al.

[3]

J.Han and Y.Fu. Discovery of multiple-level association rules from large databases. VLDB (1995) 420–431 R.Srikant,Q.Vu,and R.Agrawal. Mining associstion rules with item constraints. In Proc. 1997 Int. Conf. Knowledge Discovery and Data mining (KDD'97) 67–73 J.Pei,J.Han,and L.V.S.Lakshmanan. Mining frequent itemsets with convertible constraints. In ICDE'01, 323–332 J.Han,J,Pei and Y.Yin. Mining frequent patterns without candidate generation. In SIGMOD'00, 1–12 N.Pasquier,Y.Bastide,R.Taouil, and L.Lakhal. Efficient mining of association rules using closed itemset lattices. Information Systems (1999) 25–46 J.-F.Boulicaut and A.Bykowski. Frequent closures as a concise representation for binary data mining. In Proceedings of the Fourth Pacif-Asia Conference on Knowledge Discovery and Data Mining (PAKDD'00) 62–73 T.Calders. Deducing bounds on the frequency of itemsets. In EDBT Workshop DTDM Database Techniques in Data Mining, 2002 T.Calders and B.Goethals. Mining all non-deriable frequent itemsets. Proceedings of the 6th European Conference on Principles of Data Mining and Knowledge Discovery. 2002

[4] [5] [6] [7] [8] [9] [10]

Mining Association Rules with Frequent Closed Itemsets Lattice Lei Jia, Jun Yao, and Renqing Pei School of Mechatronics and Automation Shanghai University, Shanghai, 200072, China {jialei7829,junyao0529,prq44}@hotmail.com

Abstract. One of the most important tasks in the field of data mining is the problem of finding association rules. In the past few years, frequent closed itemsets mining has been introduced. It is a condensed representation of the data and generates a small set of rules without information loss. In this paper, based on the theory of Galois Connection, we introduce a new framework called frequent closed itemsets lattice. Compared with the traditional itemsets lattice, it is simple and only contains the itemsets that can be used to generate association rules. Using this framework, we get the support of frequent itemsets and mine association rules directly. We also extend it to fuzzy frequent closed itemsets lattice, which is more efficient at the expense of precision.

1

Introduction

Data mining is to extract the previous unknown and potentially useful information from the large database. The problem of mining association rules [1,2] has been the subjects of numerous studies. However, the traditional methods generate too many frequent itemsets. We know if the size of largest itemsets is N, the candidate frequent itemsets space is 2N. It is really a complex and untrivial work. In the past few years, based on the theory of Formal Concept Analysis, a new technology, frequent closed itermsets mining, has been introduced [3,4,5,6,7]. It extracts a particular subset of the frequent itemsets that can regenerate the whole ones without information loss. In this paper, we introduce a new framework. It is called Frequent Closed Itemsets Lattice. We use Apriori-style algorithm to build the framework. It makes full use of the Galois Connection theory and only contains the itemsets that can be used to form association rules. We also extend the frequent closed itemsets lattice to fuzzy frequent closed itemsets lattice. It is more efficient at the expense of information loss. In Section 2, we present theoretical basis of the frequent closed itemsets lattice. In Section 3, we introduce Apriori-style FCIL algorithm to build the new framework. In section 4, we discuss how to generate the informative association rules under the V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 469-475, 2003.  Springer-Verlag Berlin Heidelberg 2003

470

Lei Jia et al.

framework. In section 5, we extend the framework to fuzzy frequent closed itemsets lattice. We present our experiments in section 6 and section 7 is a brief conclusion.

2

Theoretical Basis

In this section, we define the data mining context, Galois connection, Galois closure operator, closed itemsets and frequent closed itemsets lattice.[8] Definition 1 (Data mining context). A data mining context is a triple D=( Γ , τ ,R). The rows are transactions Γ and the columns are the itemsets τ . R ⊆ Γ × τ is a relation between the transactions and itemsets. Definition 2 (Galois Connection). Let D=( Γ , τ ,R) be data mining context. For T ⊆ Γ and I ⊆ τ , we define: Γ

τ

τ

Γ

p(T): 2 → 2 p(T)={i ∈ τ | ∀ t ∈ T,(t,i) ∈ R} q(I): 2 → 2 Γ

q(I)={t ∈ Τ | ∀ i ∈ I,(t,i) ∈ R}

τ

Because 2 and 2 are power set of Γ and

τ

, (p,q)

forms a Galois connec-

tion. The connection has the following property.

I1 ⊆ I2 → q(I1) ⊇ q(I2)

T1 ⊆ T2 → p(I1) ⊇ p(I2) T ⊆ q(I) → I ⊆ p(T) Definition 3 (Galois closure operator). Given the Galois connection, if we choose h=p(q(I)), g=q(p(T)), I,I1,I2 ⊆ τ ,T,T1,T2 ⊆ Γ , we have the following properties: Extension:

I ⊆ h(I)

T ⊆ g(T)

Idempotency:

h(h(I))=h(I)

Monotonicity:

I1 ⊆ I2 → h(I1) ⊆ h(I2)

g(g(T))=g(T)

T1 ⊆ T2 → g(T1) ⊆ g(T2) Definition 4 (Closed itemsets and Frequent closed itemsets). The closure of an itemset I is the maximal superset of I which has the same support as I. The closed itemset C is the itemset whose support is equal to its closure. If the support is higher than the minimum support, the closed itemsets called frequent closed itemsets. Definition 5 (Frequent closed itemsets lattice). We use all the itemsets C' in the closure to build a complete lattice (C', ≤ ) called frequent closed itemsets lattice.

Mining Association Rules with Frequent Closed Itemsets Lattice

471

The lattice has the following properties: 1) The closure lattice is simple than the frequent itemsets lattice. It is composed by the frequent closed itemsets. 2) All sub-itemsets in the closure lattice of a frequent closed itemsets are frequent (anti-monotonicity property). 3) We can get the associations directly from the lattice as shown in section 4.

3

Building Frequent Closed Itemsets Lattice with Apriori-Style FCIL Algorithm

Considering that frequent closed itemsets lattice has the anti-monotonicity property, we build the framework with the Apriori-style algorithm. Suppose we have 6 transactions in table1. The second column is the items contain in the transaction and the class label is in the third column. Now, we start to build the lattice. Generally, we sort the itemsets in lexicographic order, but nowadays we find if we put items in specific order, it will improve the efficiency of the process. We use the support-based order introduced in paper [7]. Also do we sort the frequent 1-item in support-based ascend order and use the same Itemset-Transaction structure. Table 1.

1 2 3 4 5 6

Item ACTW CDW ACTW ACDW ACDTW CDT

Fig.1. Frequent closed itemsets lattice

Class 1 2 1 1 2 2

472

Lei Jia et al.

The level-wise algorithms like Apriroi [2] have been proved to be one of the most efficient algorithms in the field of association rules mining. So we propose the Apriori-style FCIL algorithm (Frequent Closed Itemsets Lattice algorithm). The FCIL algorithm: Input:

T[1](in ascend order), MinSup is the Minimum Support threshold and function t(x) represents the transactions that contain x. Output: frequent closed itemsets lattice 1)L1={large 1-itemsets X1 × t(X1) in ascend order}; 2)T[2]=order T[1] according to L1; 3)Construct the lattice of level-1;

4)for(k=2;Lk-1 ≠ φ ;k++) 5) Ck=apriori_gen(Lk-1) where Lk=Lk-1 ∪ Lk-1,t(Lk)= t(Lk-1) ∩ t(Lk-1); 6) forall transactions t ∈ T[2] 7) Ct=subset(Ck,t); 8) forall candidates c ∈ Ct do 9) c.count++; 10) end 11) end 12) Lk={c ∈ Ck| c.count ≥ MinSup}; 13) if t(x)=t(y) where x ∈ Lk, y ∈ Lk-1 14) add the item to the lattice and connect them; 15) else add the item to the lattice; 16) end 17)end return the frequent closed itemsets lattice Using the FCIL algorithm, we can get the frequent closed itemsets collections from the lattice easily. In each {} are the frequent itemsets have the same closure. We show the transactions that contain the itemsets in ( ). They are {D,DC(2456)}, {DW,DWC(245)},{T,TC(1356)},{TA,TW,TAW,TAC,TWC(135)},{A,AW,AC,AW C(1345)},{W,WC(12345)} and {C(123456)}.

4

Forming Association Rules

After building the frequent closed itemsets lattice, we try to form association rules from the structure directly. Association rules can be divided into two classes. According to the rule I1 → I2-I1, if both I1 and I2 are in the same closure, the confidence of the rule is 1. It is so-called exact rules. If I1 and I2 are in the different closure, the rule maybe belongs to approximate rules. We should compare the confidence with the preset confidence threshold to determine whether they are our targeted association rules. We use ⇒ and → to show exact and approximate rules respec-

Mining Association Rules with Frequent Closed Itemsets Lattice

473

tively, we also present the confidence of the approximate rules in the brackets behind the rule. Using the following inference technology, we elicit the informative rules directly. 1) The informative rules mean we can infer the other rules from them. We should pay attention to the following cases: i. If the antecedents of the rules are the same, which one has the largest consequent is the informative one. ii. If the conjunctions of the antecedent and consequent of the rule is the same, which rule has the smallest antecedent is the informative rule. 2) According to the Guigues-Duquenne basis and Luxenburger basis [8], we can prune some rules. (1) Guigues-Duquenne basis: i. X ⇒ Y, W ⇒ Z a XW ⇒ YZ ii. X ⇒ Y, Y ⇒ Z a X ⇒ Z We can also get: X → Y, Y ⇒ Z a X → Z (2) Luxenburger basis: i. The association rule X → Y has the same support and confidence as the rule close(X) → close(Y). The close(x) means the closed itemsets of x. ii. For any three closed itemsets I1,I2 and I3, such that I1 ⊆ I2 ⊆ I3, the confidence of the rule I1 → I3 is equal to the product of the confidences of the rules I1 → I2 and I2 → I3, and its support is equal to the support of the rule I2 → I3. (3) When two rules are equipotent (The support of the antecedent and the conjunctions of the antecedent and consequent of the two rules are the same.), we can delete one according to the ascend order. We get the informative ones of approximate rules and exact rules respectively, then compare them with the above principle again to get the final results. According to the transactions in table 1, we get the final informative rules: C → W(5/6), W → AC(0.8), D ⇒ C, T ⇒ C and TW ⇒ AC. We note that, unlike the traditional algorithm, we do not generate the rule like W → A, then prune it when we find W → AC. Based on the frequent closed itemsets lattice and above inference technology, we can get W → AC directly (The rule is determined by the smallest itemset in one closure with the largest itemset in another closure). We are sure it will cover the information of rules like W → A. Through analyzing the example, we find that if we combine frequent closed itemsets lattice and inference technology, we can mine association rules efficiently.

5

Fuzzy Frequent Closed Itemsets Lattice

In reality, the more correlated the data is, the more association rules we will find. In such case, the collections of frequent closed itemsets are more compact than frequent itemsets. However, we often find the following situation: itemsets A and itemsets B

474

Lei Jia et al.

are both occur in almost the same transactions except small exceptions. Case 1: |A|=|B|, just means they are in the same level. Case2: |A|=|B|-n, just means A is the subset of B. In the above definition, if the frequent itemsets have the same support (occur in the same transactions), they are in the same frequent itemsets closure. The frequent closed itemsets is a concise representation. However, in many applications, it will cost a lot of time and memories. So we propose to loose the concise representation to ε -adequate representation, we can gain efficiency at the expense of precision. We introduce the notation fuzzy frequent closed itemsets. Formally, if itemsets X occurs in t transactions with the database, we say that the itemsets Y is in the same fuzzy frequent closed itemsets of X if the difference of sup(X) and sup(X ∪ {Y}) is less than the threshold δ . If δ =0, the fuzzy frequent closed itemsets degenerate to frequent closed itemsets. The fuzzy frequent closed itemsets form the fuzzy frequent closed itemsets lattice. In the FCIL algorithm, if we change step 13 to the step 13': if |t(x) -t(y)| ≤ δ where x ∈ Lk-1,y ∈ Lk, we can construct the fuzzy frequent closed itemsets as well.

6

Experiment

To study the effectiveness and efficiency of the algorithm we proposed in the above section, we implemented it in Basic and tested on a 1GHz Pentium PC computer with 128 megabytes of the main memory. The test consists of a set of synthetic transaction database generated using a randomized itemset generation algorithm similar to the algorithm described in [2]. The average size of these itemsets N is 3, the size of item sets T we randomly choose from is 5 and the size of database is 10K. The minimum support threshold is 70%. Using the traditional algorithm like Apriori [2], we have 35 connections if we want to build the frequent itemsets lattice. Now, we use FICL algorithm to build frequent closed itemsets lattice, the number reduces to 17. After building the lattice, we begin to mine the association rules. Within Apriori algorithm, we firstly generate candidate rules, then using the inference technology to prune the useless rules. If we have already built the frequent closed itemsets lattice, we can directly induce the exact association rules and approximation association rules respectively. We only use the inference technology to find whether some exact rules can be covered by some approximation rules. The time used to create all candidate rules and prune uninformative ones can be reduced. The experiment result is summarized in Table 2. Because the time spent in finding frequent itemset is same, we only show the time used to induce the association rules in the time list. Clearly, in practice, we will save more time when N increases. Table 2. Algorithm

Connections

Apriori FCIL

35 17

Candidate Rules 43

(xact+Approximation) Rules 7+5

Result Rules 3

Time (ms) 13 2

Mining Association Rules with Frequent Closed Itemsets Lattice

7

475

Conclusion

Association rule mining has been extensively studied since its introduction. It has been used in many applications. However, we are often in trouble just because we have to face a large number of candidate frequent itemsets. In the past few years, frequent closed itemsets mining has been introduced. It will generate a small set of rules compared with the traditional frequent itemsets mining without information loss. In this paper, we introduce a new framework called frequent closed itemsets lattice to mine association rules. Compared with the traditional itemsets lattice, the framework is simple and only contains the itemsets we need to form association rules. Under this framework, we get the support of the frequent itemsets and mine association rules directly. We also extend the structure to fuzzy frequent closed itemsets lattice, which is more efficient at the expense of precision. Finally, through the experiment, we prove the effectiveness of our method.

References [1] [2] [3] [4] [5] [6] [7] [8]

R.Agrawal,T.Imielinski, and A.Swami. Mining association rules between sets of items in large databases. SIGMOD 93, 207–216 R.Agrawal and R.Srikant. Fast slgorithms for mining association rules. VLDB 94, 487-499 N.Pasquier,Y.Bastide,R.Taouil, and L.Lakhal. Discovering frequent closed itemsets for association rules. In 7th Intl. Conf. on Database Theory 1999 D.Cristofor, L.Cristofor, and D.Simovici. Galois connection and data mining. Journal of universal Computer Science (2000) 60–73 J.Pei,J.Han,and R.Mao. Closet: An efficient algorithm for mining frequent closed itemsets. In SIGMOD Int'l Workshop on Data Mining and Knowledge Discovery, 2000 N.Pasquier,Y.Bastide,R.Taouil, and Lotfi Lakhal. Efficient mining of associaion rules using closed itemset lattice. Information Systems (1999) 25–46 M.Zaki. Generating non-redundant association rules. In Proceedings of the 6th ACM-SIGKDD Intermational Conference on Knowledge Discovery and Data Mining (2000)34–43 B.A.Davey and H.A.Priestley. Introduction to lattices and Order. Cambridge University Press, Four edition (1994)

Mining Generalized Closed Frequent Itemsets of Generalized Association Rules Kritsada Sriphaew and Thanaruk Theeramunkong Sirindhorn International Institute of Technology, Thammasat University 131 Moo 5, Tiwanont Rd., Bangkadi, Muang, Pathumthani, 12000, Thailand {kong,ping}@siit.tu.ac.th

Abstract. In the area of knowledge discovery in databases, the generalized association rule mining is an extension from the traditional association rule mining by given a database and taxonomy over the items in database. More initiative and informative knowledge can be discovered. In this work, we propose a novel approach of generalized closed itemsets. A smaller set of generalized closed itemsets can be the representative of a larger set of generalized itemsets. We also present an algorithm, called cSET, to mine only a small set of generalized closed frequent itemsets following some constraints and conditional properties. By a number of experiments, the cSET algorithm outperforms the traditional approaches of mining generalized frequent itemsets by an order of magnitude when the database is dense, especially in real datasets, and the minimum support is low.

1

Introduction

The task of association rule mining (ARM) is one important topic in the area of knowledge discovery in databases (KDD). ARM focuses on finding a set of all subsets of items (called itemsets) that frequently occur in database records or transactions, and then extracting the rules representing how a subset of items influences the presence of another subset [1]. However, the rules may not provide informative knowledge in the database. It may be limited with the granularity over the items. For this purpose, generalized association rule mining (GARM) was developed using the information of pre-defined taxonomy over the items. The taxonomy may classify products (or items) by brands, groups, categories, and so forth. Given a taxonomy where only leaf items present in the database, more initiative and informative rules (called generalized association rules) can be mined from the database. Each rule contains a set of items from any levels of the taxonomy. In the past, most previous works focus on efficient finding all generalized frequent itemsets. As an early intensive work, Srikant et al. [2] proposed five algorithms that apply the horizontal database format and breath-first search strategy like Apriori algorithm. These algorithms waste a lot of time in multiple scanning a database. As a more recent algorithm, Prutax, was proposed in [3] by V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 476–484, 2003. c Springer-Verlag Berlin Heidelberg 2003


Mining Generalized Closed Frequent Itemsets

477

applying a vertical database format to reduce the time needed in database scanning. Nevertheless, a limitation of this work is the cost of checking whether their ancestor itemsets are frequent or not using a hash tree. There exists a slightly different task for dealing with multiple minimum supports as shown in [4, 5, 6]. A parallel algorithm was proposed in [7]. The recent applications of GARM were shown in [8, 9]. Our efficient approach to mine all generalized frequent itemsets is presented in [10]. Furthermore to improve time complexity of the mining process, the concepts of closed itemsets have been proposed in [11, 12, 13]. The main idea of these approaches focus to find only a small set of closed frequent itemsets, which is the representative of a large set of frequent itemsets. This technique helps us reduce the computational time. Thus, we intend to apply this traditional concept to deal with the generalized itemsets in GARM. In this work, we propose a novel concept of generalized closed itemsets, and present an efficient algorithm, named cSET, to mine only generalized closed frequent itemsets.

2

Problem Definitions

A generalized association rule can be formally stated as follows. Let I = {A, B, C, D, E, U, V, W} be a set of distinct items, T = {1, 2, 3, 4, 5, 6} be a set of transaction identifiers (tids). The database can be viewed into two formats, i.e. horizontal format as shown in Fig. 1A, and vertical format as shown in Fig. 1B. Fig. 1C shows the taxonomy, a directed acyclic graph on items. An edge in a taxonomy represents is-a relationship. V is called an ancestor item of U, C, A and B. A is called a descendant item of U and V. Note that only leaf items of a taxonomy are presented in the original database. Intuitively, the database can be extended to contain the ancestor items by adding the record of ancestor items of which tidsets are given by the union of their children as shown in Fig. 1D. A set IG ⊆ I is called a generalized itemset (GI) when IG is a set of items where no any items in the set is an ancestor item of the others. The support of IG , denoted by σ(IG ), is defined as a percentage of the number of transactions in which IG occurs as a subset to the total number of transactions. Only the GI that has its support greater than or equal to a user-specified minimum support (minsup) is called a generalized frequent itemset (GFI). A rule is an implication → I2 , where I1 , I2 ⊆ I, I1 ∩ I2 = ∅, I1 ∪ I2 is GFI, and no of the form R: I1  item in I2 is an ancestor of any items in I1 . The confidence of a rule, defined as σ(I1 ∪ I2 )/σ(I1 ), is the conditional probability that a transaction contains I2 , given that it contains I1 . The rule is called a generalized association rule (GAR) if its confidence is greater than or equal to a user-specified minimum confidence (minconf). The task of GARM can be divided into two steps, i.e. 1) finding all GFIs and 2) generating the GARs. The second step is straightforward while the first step takes intensive computational time. We try to improve the first step by exploiting the concept of closed itemsets to GARM, and find only a small set of generalized closed itemsets to reduce the computational time.

478

Kritsada Sriphaew and Thanaruk Theeramunkong

Fig. 1. Databases and Taxonomy

3

Generalized Closed Itemset (GCI)

In this section, the concept of GCI is defined by adapting the traditional concept of closed itemsets in ARM [11, 12, 13]. We show that a small set of generalized closed frequent itemsets is sufficient to be the representative of a large set of GFIs. 3.1

Generalized Closed Itemset Concept

Definition 1. (Galois connection): Let the binary relation δ ⊆ I × T be the extension database. For an arbitrary x∈I and y∈T, xδy can be denoted when x is related to it y in database. Let it X ⊆ I, and Y ⊆ T. Then the mapping functions, t:I → T, t(X) = {y ∈ T | ∀x ∈ X, xδy} i:T → I, i(Y) = {x ∈ I | ∀y ∈ Y, xδy} define a Galois connection between the power set of I (P(I)) and the power set of T (P(T)). The following properties hold for all X,X1 ,X2 ⊆I and Y,Y1 ,Y2 ⊆T: 1. X1 ⊆ X2 =⇒ t(X1 ) ⊇ t(X2 ) 2. Y1 ⊆ Y2 =⇒ i(Y1 ) ⊇ i(Y2 ) 3. X ⊆ i(t(X) and Y ⊆ t(i(Y)) Definition 2. (Generalized closure): Let X ⊆ I, and Y ⊆ T, the composition of two mappings gcit :P(I)  → P(I) and gcti : P(T)  → P(T) are generalized closure operator on itemset and tidset respectively. The mapping of gcit (X) = i ◦ t(X) = i(t(X)) while gcti (Y) = t ◦ i(Y) = t(i(Y)). Definition 3. (Generalized closed itemset and tidset): X is called a generalized closed itemset (GCI) when X = gcit (X), and Y is called a generalized closed tidset (GCT) when Y = gcti (Y).

Mining Generalized Closed Frequent Itemsets

479

Fig. 2. Galois Lattice of Concepts For X ⊆ I and Y ⊆ T, the generalized closure operators gcit and gcti satisfy the following properties (Galois property): 1. Y ⊆ gcti (Y). 2. X ⊆ gcit (X). 3. gcit (gcit (X)) = gcit (X), and gcti (gcti (Y)) = gcti (Y). For any GCI X, there exists a corresponding GCT Y, with the property that Y = t(X) and X = i(Y). Such a GCI and GCT pair X × Y is called a concept. All possible concepts can be formed a Galois lattice of concepts as shown in Fig. 2. 3.2

Generalized Closed Frequent Itemsets (GCFIs)

The support of a concept X × Y is the size of GCT (i.e. |Y|). A GCI is frequent when its support is greater than or equal to minsup. Lemma 1. For any generalized itemset X, its support is equal to the support of its generalized closure (σ(X) = σ(gcit (X))). Proof. Given X, its support σ(X) = |t(X)|/|T|, and the support of its generalized closure σ(gcit (X)) = |t(gcit (X))|/|T|. To prove the lemma, we have to show that t(X) = t(gcit (X)). Since gcti is the generalized closure operator, it satisfies the first property that t(X) ⊆ gcti (t(X)) = t(i(t(X))) = t(gcit (X)). Thus t(X) ⊆ t(gcit (X)). The gcit (X) provides the GI that is the maximal superset of X and has the same support as X. Then X ⊆ gcit (X), and t(X) ⊇ t(gcit (X)) due to the Galois property [11]. We can conclude that t(X) = t(gcit (X)). Implicitly, the lemma states that all GFIs can be uniquely determined by the GCFIs since the support of any GIs will be equal to its generalized closure. In the worst case, the number of GCFIs is equal to the number of GFIs, but typically it is much smaller. From the previous example, there are 10 GCIs, which are the representatives of a large amount of all GIs as shown in Fig. 2. With minsup=50%, only 7 concepts (in bold font) are GCFIs.

480

4 4.1

Kritsada Sriphaew and Thanaruk Theeramunkong

Algorithm cSET Algorithm

In our previous work [10], all GFIs can be enumerated by applying two constraints, i.e. subset-superset and parent-child, on GIs for pruning. We propose an algorithm called cSET algorithm, which specifies the order of set enumeration by using these two constraints and the generalized closures to generate only GCIs. Two constraints stated that only descendant and superset itemsets of GFIs should be considered in the enumeration process. For generating only GCFIs, the following conditional properties must be checked when generating the child itemsets by joining X1 × t(X1 ) with X2 × t(X2 ). 1. If t(X1 ) = t(X2 ), then (1) replace X1 and children under X1 with X1 ∪ X2 , (2) generate taxonomy-based child itemsets of X1 ∪ X2 , and (3) remove X2 (if any). 2. If t(X1 ) ⊂ t(X2 ), then (1) replace X1 with X1 ∪ X2 and (2) generate taxonomy-based child itemsets of X1 ∪ X2 . 3. If t(X1 ) ⊃ t(X2 ), then (1) generate join-based child itemset of X1 with X1 ∪ X2 , (2) add hash table with X1 ∪ X2 , and (3) remove X2 (if any). 4. If t(X1 ) = t(X2 ) and t(X1 ∪ X2 ) is not contain in hash, then generate join-based child itemset of X1 with X1 ∪ X2 . Using the given example in Fig. 1 with minsup=50%, the cSET algorithm starts with an empty set. Then, we add all frequent items in the second level of the taxonomy, that are item V and W, and form the second level of the tree shown in Fig. 3. Each itemset has to generate two kinds of child itemsets, i.e. taxonomybased and join-based itemsets, respectively. We first generate taxonomy-based itemset by joining last items in itemsets by its child according to taxonomy. One taxonomy-based itemset of V is VU. The first property holds for VU, which results in replacing V with VU and then generating VUA and VUB. The second taxonomy-based itemset is joined with the current itemset (VU), which produces VUC. Again, the first property holds for VUC, which results in replacing VU and the children in tree under VU with VUC. Next, the join-based child itemset of V, VW, is generated. The third property holds for VW, which results in removing W and then generating VW under V. In the same approach, the process recursively occurs until no new GCFIs are generated. Finally, a complete itemset tree is constructed without excessive checking cost as shown in Fig. 3. All remaining itemsets in Fig. 3, except ones of crossed itemsets, are GCFIs. 4.2

Pseudo-code Description

The formal pseudo-code of cSET, extended from SET in [10], is shown below. The main procedure is cSET-MAIN and a function, called cSET-EXTEND, creates a subtree followed by a proposed set enumeration. cSET-EXTEND is executed recursively to create all itemsets under the root itemsets. The NewChild function

Mining Generalized Closed Frequent Itemsets

481

Fig. 3. Finding GCFIs using cSET with minsup=50%

creates a child itemset. For instance, NewChild(V,U) creates a child itemset VU of a parent itemset V, and adds the new child in a hash table. The GenTaxChild function returns the taxonomy-based child itemsets of that GI. Line 8-11 generates the join-based child itemsets. The function, called cSET-PROPERTY, checks for four conditional properties of GCIs and makes the operations with the generated itemset. Following the cSET algorithm, we will get a tree of all GCFIs. cSET-MAIN (Database,Taxonomy,minsup): 1. Root = Null Tree //Root node of set enumeration 2. NewChild(Root, GFIs from second level of taxonomy) 3. cSET-EXTEND(Root) cSET-EXTEND(Father) 4. For each Fi in Father.Child 5. C = GenTaxChild(Fi ) //Gen taxonomy-based child itemset 6. If supp(C) ≥ minsup then 7. cSET-PROPERTY(Nodes,C) 8. For j = i+1 to |Father.Child| //Gen join-based child itemset 9. C = Fi ∪ Fj 10. If supp(C) ≥ minsup then 11. cSET-PROPERTY(Nodes,C) = NULL then cSET-EXTEND(Fi ) 12. If Fi .Child  cSET-PROPERTY(Node,C) 13. if t(Fi ) = t(Fj ) and Child(Fi )= ∅ then //Prop.1 14. Remove(Fj ); Replace(Fi ) with C 15. else if t(Fi ) ⊂ t(Fj ) and Child(Fi )= ∅ then //Prop.2 16. Replace(Fi ) with C 17. else if t(Fi ) ⊃ t(Fj ) then //Prop.3 18. Remove(Fj ); if !Hash(t(C)) then NewChild(Fi ,C) 19. else if !Hash(t(C)) then NewChild(Fi ,C) //Prop.4

482

5

Kritsada Sriphaew and Thanaruk Theeramunkong

Experimental Results

Since the novel concept of GCIs has never appeared in any researches, there are no existing algorithms for finding GCFIs. In our experiment, the cSET algorithm is evaluated by comparing with the current efficient algorithms for mining GFIs, i.e. SET algorithm [10]. All algorithms are coded in C language and the experiment was done on a 1.7GHz PentiumIV with 640Mb of main memory running Windows2000. The syntactic and real datasets are used in our experiment. The syntactic datasets are automatically generated by a generator tool from IBM Almaden with some slightly modified default values. Two real datasets from the UC Irvine Machine Learning Database Repository, i.e. mushroom and chess, are used with our own generated taxonomies. The original items contain in the leaf-level of taxonomy. Table. 1 shows the comparison of using SET and cSET to enumerate all GFIs and GCFIs, respectively. In real datasets, the number of GCFIs is much smaller than that of GFIs. With the same datasets, the ratio of the number of GFIs to that of GCFIs typically increases when we lower minsup. The higher the ratio is, the more time reduction is gained. The ratio can grow up to around 7,915 times, which results in reduction of running time around 3,878 times. Note that in syntactic datasets, the number of GFIs is slightly different from the number of GCFIs. This indicates that the real datasets are dense but the syntactic datasets are sparse. This result makes us possible to reduce more computational time by using cSET in real situations.

Table 1. Number of itemsets and Execution Time (GFIs vs. GCFIs)

Mining Generalized Closed Frequent Itemsets

6

483

Conclusion and Further Research

A large number of generalized frequent itemsets may cause of high computational time. Instead of mining all generalized frequent itemsets, we can mine only a small set of generalized closed frequent itemsets and then result in reducing computational time. We proposed an algorithm, named cSET, by applying some constraints and conditional properties to efficiently enumerate only generalized closed frequent itemsets. The advantage of cSET becomes more dominant when minimum support is low and/or the dataset is dense. This approach makes us possible to mine the data in real situations. In further research, we intend to propose a method to extract only a set of important rules from these generalized closed frequent itemsets.

Acknowledgement This paper has been supported by Thailand Research Fund (TRF), and NECTEC under project number NT-B-06-4C-13-508.

References [1] Agrawal, R., Imielinski, T., Swami, A. N.: Mining association rules between sets of items in large databases. In Buneman, P., Jajodia, S., eds.: Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, Washington, D. C. (1993) 207–216 476 [2] Srikant, R., Agrawal, R.: Mining generalized association rules. Future Generation Computer Systems 13 (1997) 161–180 476 [3] Hipp, J., Myka, A., Wirth, R., G¨ untzer, U.: A new algorithm for faster mining of generalized association rules. In: Proceedings of the 2nd European Conference on Principles of Data Mining and Knowledge Discovery (PKDD ’98), Nantes, France (1998) 74–82 476 [4] Chung, F., Lui, C.: A post-analysis framework for mining generalized association rules with multiple minimum supports (2000) Workshop Notes of KDD’2000 Workshop on Post-Processing in Machine Learing and Data Mining 477 [5] Han, J., Fu, Y.: Mining multiple-level association rules in large databases. Knowledge and Data Engineering 11 (1999) 798–804 477 [6] Lui, C. L., Chung, F. L.: Discovery of generalized association rules with multiple minimum supports. In: Proceedings of the 4th European Conference on Principles of Data Mining and Knowledge Discovery (PKDD ’2000), Lyon, France (2000) 510–515 477 [7] Shintani, T., Kitsuregawa, M.: Parallel mining algorithms for generalized association rules with classification hierarchy. In: Proceedings of the 1998 ACM SIGMOD International Conference on Management of Data. (1998) 25–36 477 [8] Michail, A.: Data mining library reuse patterns using generalized association rules. In: International Conference on Software Engineering. (2000) 167–176 477 [9] Hwang, S. Y., Lim, E. P.: A data mining approach to new library book recommendations. In: Lecture Notes in Computer Science ICADL 2002, Singapore (2002) 229–240 477

484

Kritsada Sriphaew and Thanaruk Theeramunkong

[10] Sriphaew, K., Theeramunkong, T.: A new method for fiding generalized frequent itemsets in generalized association rule mining. In Corradi, A., Daneshmand, M., eds.: Proc. of the Seventh International Symposium on Computers and Communications, Taormina-Giardini Naxos, Italy (2002) 1040–1045 477, 480, 482 [11] Pasquier, N., Bastide, Y., Taouil, R., Lakhal, L.: Discovering frequent closed itemsets for association rules. Lecture Notes in Computer Science 1540 (1999) 398–416 477, 478, 479 [12] Pasquier, N., Bastide, Y., Taouil, R., Lakhal, L.: Efficient mining of association rules using closed itemset lattices. Information Systems 24 (1999) 25–46 477, 478 [13] Zaki, M. J., Hsiao, C. J.: CHARM: An efficient algorithm for closed itemset mining. In Grossman, R., Han, J., Kumar, V., Mannila, H., Motwani, R., eds.: Proceedings of the Second SIAM International Conference on Data Mining, Arlington VA (2002) 477, 478

Qualitative Point Sequential Patterns Aomar Osmani LIPN- UMR CNRS 7030 Universit´e de Paris 13, 99 avenue J.-B. Cl´ement 93430 Villetaneuse, France [email protected]

Abstract. We have introduced [Osm03] a general model for representing and reasoning about STCSP (Sequential Temporal Constraint Satisfaction Problems) to deal with patterns in data mining and applications which generate great quantity of data used to understand or to explain some given situations (diagnostic of dynamic systems, alarms monitoring, event prediction, etc.). One important issue of sequence reasoning concerns the recognition problem. This paper presents the STCSP model with qualitative point primitives using frequency evaluation function. It gives the problem formalization; usual problems concerned with this kind of approach and it proposes some algorithms to deal with sequences.

1

Introduction

Sequences generation, and more generally reasoning about sequences,problem in many disciplines including artificial intelligence, databases, cognitive sciences, and engineering techniques [Se02]. A sequence reasoning called also sequence mining is the discovery of sets of characteristics shared through time by a great number of objects ordered in time [Zak01]. Various problems are concerned with reasoning about sequences i.e. sequence prediction, sequence generation, and sequence recognition. The most considered one is the task of discovering of all frequent sequences in large databases. It is a quite challenging problem, for instance, the search space is extremely large: with m variables there are O(mk ) potentially frequent sequences of length at most k. Many techniques have been proposed to mine temporal databases for the frequently occurring sequences [Zak01, JBS99, Lin92, Se02, aSD94, BT96].

2

Definitions

Let us consider the set E = {e1 , . . . , en } of qualitative points and their possible relationships {<, =, >}. Let (e1 , . . . , em ) a vector of observations or events, such that ∀i ∈ {1..m}, ei ∈ E and R a matrix of binaries relations on E. Definition 1. A sequential pattern P is defined as a couple ((e1 , . . . , en ), R) such that (∀i), i ∈ {1..n − 1} ei {<, =}ei+1 .

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 485–492, 2003. c Springer-Verlag Berlin Heidelberg 2003


486

Aomar Osmani

For instance, consider the following situation: when you install this software, you will see in the screen the message m1 , then m2 finally the messages m1 and m3 in the same window (at the same time). The pattern P corresponding to this situation will be represented by the vector (m1 , m2 , m1 , m3 ) and the matrix R = ((=, <, <, <), (<, =, <, <)(<, <, =, =)(<, <, =, =)). Using the property 1 proved in [Osm03], the pattern P may be represented by ((m1 , m2 , m1 , m3 ), (<, <, =)). Definition 2. (Subsequence) A partial order set(poset) S is a subsequence of a poset P with partial order relation , denoted S ⊆ P , if – there exists an injective application φ from P onto S such that if x y in S then φ(x) φ(y) in P; – each object of S and P is represented by pair (identifier, time). For all x = (ix , tx ), φ(x) = (ix , φ(tx )). Example 1. [(a, c), (<, <)] ⊆ [(a, b, c, a), (<, =, <, <)] In several applications all variables instances are important, instances masking is not authorized. For that, we define the contiguous subsequence: s1 is a contiguous subsequence of s, denoted S  P if S ⊆ P , and for all x, y ∈ S such that x y, if ∃z ∈ S such that x z y then ∃t ∈ P such that φ(x) t φ(y). In the case where is a poset ( = {α1 , . . . , αn }), the last condition becomes: x αi y, if ∃z ∈ S such that x αj z αk y then ∃t ∈ P such that φ(x) αl t αm φ(y) for all i, j, k, l, m ∈ {1..n} (gaps in objects and gaps in relations are prohibited). In sequential pattern reasoning, data are given by large sequences of variables instances with infinite length in theory. For most of the applications it is not necessary or not possible to observe all data in the same time. If we admit that data are organized in a time line, then we take the assumption that if advancing on time line from the past to the future at each time we have the possibility to observe events (variable instances) in a finite window. For instance, if we consider the monitoring task of dynamic systems, the operators observe alarms in the screen and they can only make deduction about finite sequence length. The maximal length is called observation window. Definition 3. the observation window Wq on the sequence s is the number of variables instances of S observable simultaneously. Example 2. Let us consider the pattern ((m1 , m2 , m1 , m3 ), (<, <, =)). If we assume that no masking is authorised then (1) if Wq = 2 then the possible observations are {(m1 , m2 ), (m2 , m1 , (m2 , m3 ), (m1 , m3 )} (2) if Wq = 3 then the possible observations will be {(m1 , m2 , m1 ), (m1 , m2 , m3 , and (m2 , m1 , m3 ).

3

Qualitative Points Model

A Sequence Temporal Constraint Satisfaction Problem (STCSP) is defined [Osm03] as a 5-uplet M =< O, L, f, C, S >

Qualitative Point Sequential Patterns

487

Where: O describes the considered objects(qualitative points), L defines the sequences language(Vilain and Kautz point algebra [KV86], f defines the evaluation function (in the considered model we treat only the frequency evaluation function), C is the set of constraints applied to the considered functions (C is used to select the significant patterns: fixed frequency threshold is the main used constraint), and S is the solver (evaluation algorithm). f (M, M1 ) defines the frequency of the subsequence M1 in the sequence M . For instance – f ( (a1 , a2 , a1 , a2 ), a1 ) = 2. – f ( (a1 , a2 , a1 , a2 ), (a1 , a2 )) = 3 if we accept redundancy, and – f ( (a1 , a2 , a1 , a2 ), (a1 , a2 )) = 2 otherwise. 3.1

Constraint Network

A qualitative point STCSP network of M =< O, L, f, C, S > is a multi-valuated graph G = (N, V ) where each node n ∈ N represents an object of O and V is a set of couple (time point relation, position). Example 3. Let us consider the following temporal pattern [(a1 , a2 , a3 , a3 , a4 ), (= , <, <, =)]. Figure 3.1 gives the corresponding STCSP network. To simplify the representation we may omit the relationship in the arcs: an oriented arc corresponds to the relation ”<”. The non oriented arc refers to the relation ”=” (see figure 3.1(b) ). 3.2

Qualitative Point STCSP Algorithms

The type of the selected algorithms depends on the considered application and the kind of problem to solve. If we consider a deterministic environment, the main studied problems are [Se02]:

(=,1)

a2

1

(<,2)

a1

a2

2

3

a

a1 (<,3)

a

3

a4 (a)

(=,4)

3

a4

4

(b)

Fig. 1. The qualitative point TSCSP network

488

Aomar Osmani

– Prediction problem: given a pattern [(a1 , . . . , an ), R], we want to predict the object an+j , j > 0 such that (∀i ∈ 1..n) ai < an+1 . When j=1, we make predictions based only on the immediately preceding object; – Sequence generation. This task has the same formulation as the prediction problem but consider the following situation: given a pattern [(a1 , . . . , an ), R], we want to generate an object an+j such that (∀i ∈ 1..n) ai < an+1 ; – Sequence recognition. This problem consists in verifying the consistency of a given pattern [(a1 , . . . , an ), R]. The response will be the pattern is valid or not regarding to the set of an a priori defined criteria. To solve these problems we must first solve some generic problems like: (1)checking the θ-consistency of a given pattern, (2)checking the θn -consistency of the network, (3)finding all solutions, and (4) finding minimal set of solutions. For specific algorithms let us consider the telecommunication network monitoring real problem. The telecommunication network equipments generate an infinite sequences of alarms1 . These alarms arrive to the supervision centre. One considered problem is the detection of ”regular” sequences of alarms received by the supervisor centre when a known breakdown situation is happened. Definition 4. A pattern p = [(a1 , . . . , an ), R] is θ-consistent in the network G if there exists at least θ instances ”without redundancy” 2 of the pattern p in G. Example 4. In the following network G = [(a, b, a, b), (<, <, <)], the pattern p = [(a, b), (<)] is 2-consistant: (a,b,a,b), and (a,b,a,b). But, if we consider the redundancy there are three instances of p in G: (a,b,a,b), (a,b,a,b), and (a,b,a,b). Definition 5. A network G is θ-consistent if there is at last one θ-consistent pattern p = [(a1 , . . . , an ), R] in the network G. G is θn -consistent if there exist at last one θ-consistent pattern p of the length n in G 3 . Example 5. In the following network G = [(a, b, a, b), (<, <, <)], the pattern p = [(a, b), (<)] is 2-consistent. But it is not 3-consistant. However, if we consider the redundancy there are three instances of p in G. Checking the θ-Consistency Algorithm. A temporal pattern s is θ-consistent on the network G if there are more that θ instances of s in G such that the intersection between instances of s is empty.

1

2 3

To simplify the problem, we define an alarm as an event in the network received by the supervisor centre identified by the alarm name, equipment identifiers, and the time when the event is happening. Each label of each arc of G appears in at most in one instance of the pattern p. The pattern length is the number of objects instances in the pattern.

Qualitative Point Sequential Patterns

489

Algorithm 1 Checking the θ-consistency of a given pattern The global network: G = [(a1 , . . . , am ), (ra1,2 , dots, ram−1,m )] s = [(b1 , . . . , bn ), (r1,2 , . . . , rn−1,n )] Output : s is θ-consistent? begin 1 result = 0; 2 Ga = AllNextGen(G); // for all values of variable computes the possible successors 3 currentNodeG= s1 ; //such that aj = s1 ) starts the generation from the heat of s 4 treeSolutionsRoot = VirtualRootNode; //TreeSolution includes the tree of potential solutions 5 treeSolutions.add(virtualNode,s1 ); 6 currentSolution =s1 ; 7 currentPositionInGa = ai ; 8 while ((result  = θ) and (currentNodeG has next Node)) do 9 for all next node nd of currentPositionGa do 10 if ((currentNodeInGa, nd)=(si , si+1 )) 11 treeSolutions.add(si , si+1 ); 12 if (i=n) 13 result++; 14 for (j=n downto 1) do 15 treeSolutions.delete(sj ); 16 if (the node (sj−1 , s1 ) exists) treeSolutions.add(virtualRoot, s1 ); = s1 ; 17 treeSolutions.delete(sj , a) for all a  18 if (result= θ) return (s is θ-consistent in G; // (end) 19 if (currentNodeG has not next Node) 20 return (s is θ-inconsistent in the G; // (end) EndAlgorithm Input :

The algorithm allN extGen(G uses the point algebra composition operation to generate, for each value of all variables in the STCSP, possible successors). Example 6. Let us consider the following network: G = [(a1 , a2 , a3 , a3 , a4 ), (= , <, <, =)]. After the application of the allN extGen() algorithm, the resulting network will be: Ga = [(a1 , a2 , a3 , a3 , a4 ), (r1,2 = r3,4 = {=}, r3,3 = {<}, r2,3 = {=}, r1,3 = r1,4 = r2,4 = {<})] (see figure 6).

Finding θ-Solutions Algorithm. This algorithm generates all possible θsolutions of the network. We propose an incremental algorithm which first computes arc θ-solutions, then path θ-solutions, then 4th θ-solutions, etc.

490

Aomar Osmani

1

a2

2

2

1

a2

2

3

a

1

a1

a1 3

a

3

3

1

a4 (a)

4

a4

4

(b)

Fig. 2. (a) Example of STCSP network (b) application of the algorithm AllNextGen() on (a) Definition 6. A network G is arc θ−consistent if there is at least one arc with θ labels. Definition 7. A network G is path θ−consistent if there is at least one sequence s of the length 3 such that s is θ−consistent in G. Definition 8. A network G is n θ−consistent if there is at least one sequence s of the length n such that s is θ−consistent in G. The proposed finding θ−solutions algorithm uses the following properties: (1) the trivial one is if the temporal pattern s is θ−consistent then all subpatterns of s are also θ−consistent, (2) if we have all θ−consistent solutions of the length n, then solutions of the length n+ 1 may be matching sequences of length n sharing the same prefix subsequence of the length (n-1). The same propriety is applied in Apriori algorithm proposed in [AS95]. Algorithm 2 Finding θ-solutions Input : network G Output : all θ− solutions begin 1 solutions = ∅ ; stepSolutions =0; l =1 ; 2 for all arc in G if (θ ≤ nbLabels of (si , sj , l) then 3 solutions =solutions ∪(si , sj , l); stepSolutions++; 4 while (stepSolutions  = 0) do 5 stepSolutions =0; l++; 6 for all couple (si , sj ) such that si = ((a1 , . . . , al−1 , a), (r1,1 , . . . , r1,l−1 , ra ))and sj ((a1 , . . . , al−1 , b), (r1,1 , . . . , r1,l−1 , rb )) 7 check the θ− consistency of the sequences s = ((a1 , . . . , al−1 , a, b), (r1,1 , . . . , r1,l−1 , ra , r)) such that r = {<, =, >} 8 for each case if s if θ−consistent then 9 solutions=solutions ∪s; 10 stepSolutions++; EndAlgorithm

Qualitative Point Sequential Patterns

491

Checking the Minimal θ-Consistency Algorithm. The previous algorithm for checking the θ− consistency may be used with the following operation: after the line (9) we add solutions = solutions/{si, sj }. Both of them may be generated from the solution s. Checking the θn -Consistency Algorithm. A network G is θn -consistent if there exists a θ-consistent sequence s of the length n. One trivial algorithm to solve this problem is to use the previous finding θ-solutions algorithm. It’s possible to propose a more efficient algorithm. We can also us the finding θ− solutions algorithm and stop the computation when the value of l is n. For the contextual problems, let us consider the telecommunication networks presented before. It is a prediction problem. We can distinguish two sequential pattern problems: (1) supervised rules generation and (2)rules discovery. Supervised Rules Generation. Let F = {f1 , . . . , fn } be a set of predefined situations (breakdown situations in the our considered application)4 , |w| the observation window width and S all observations on the system. For each i, i ∈ {1..n} there exists a set of sequences s = ((e1 , . . . , em ), R), n < |w| such that if we observe s then fi is happened. The problem is, let us consider a continuous observations on the system, when it is possible to predict elements of F and which possible elements we can predict at each moment? Algorithm 3 Supervised Rules generation Input : network G Output : all θ− solutions begin 1 solutions = ∅ ; stepSolutions =0; l =1 ; 2 for all arc in G if (θ ≤ nbLabels of (si , sj , l) then 3 solutions =solutions ∪(si , sj , l); stepSolutions++; 4 while (stepSolutions  = 0) do 5 stepSolutions =0; l++; 6 for all couple (si , sj ) such that si = ((a1 , . . . , al−1 , a), (r1,1 , . . . , r1,l−1 , ra ))and sj ((a1 , . . . , al−1 , b), (r1,1 , . . . , r1,l−1 , rb )) 7 check the θ− consistency of the sequences s = ((a1 , . . . , al−1 , a, b), (r1,1 , . . . , r1,l−1 , ra , r)) such that r = {<, =, >} 8 for each case if s if θ−consistent then 9 solutions=solutions ∪s; 10 stepSolutions++; EndAlgorithm

4

fi may be an element of E .

492

Aomar Osmani

Rules Discovery. Let E  be a subset of E, |w| the observation window width and S all observations on the system. The problem is, let us consider a continuous observations on the system, which possible elements of E  we can predict at each moment?

4

Conclusion

This paper presents a formulation of the sequential patterns reasoning using constraint satisfaction problems formalism. A classification of the main kinds of temporal primitives is presented. A Model of qualitative point STCSP is introduced with a model of constraints network representation. In this paper, we consider the sequence pattern problem with frequency evaluation function. Some problems and associated resolution algorithms are proposed. This work is still in progress, and raises many questions especially how to manage constraints and which kind of knowledge we can explore when we consider systems where the values of variables change among the time.

References R. Agrawal and R. Srikant. Mining sequentiel motifs. In International Conference on Data Engineering (ICDE), Taipei, Taiwan. Expanded version is available as IBM Research Report RJ9910, October 1994, 1995. 490 [aSD94] N. Lavrac an S. Dzeroski. Inductive Logic Programming. Ellis Horword, New York, 1994. 485 [BT96] D. Bertsekas and J. Tsitsiklis. Neuro-dynamic programming. In Athena Scientic, Belmont, MA., 1996. 485 [JBS99] S. D. Johnson, D. S. Battisti, and E. S. Sarachik. Empirically derived markov models and prediction of tropical sea surface temperature anomalies. In Journal of Climate, 1999. 485 [KV86] H. Kautz and M. Vilain. Constraint propagation algorithms for temporal reasoning. In In Proceeding AAAI-86, pages 377–382, 1986. 487 [Lin92] L. Lin. Self-improving reactive agents based on reinforcement learning, planning, and teaching. Machine Learning, 8:293–321, 1992. 485 [Osm03] A. Osmani. Stcsp: A representation model for sequential patterns. In Foundations and Applications of Spatio-Temporal Reasoning, March 2003. 485, 486 [Se02] R. Sun and L. Gilles editors. Introduction to sequence Learning. (a completer), 2002. 485, 487 [Zak01] D. J. Zaki. On spade´ a:an efficient algorithm for mining frequent sequences. Machine Learning, 42:31–60, 2001. 485 [AS95]

Visualization and Evaluation Support of Knowledge Discovery through the Predictive Model Markup Language Dietrich Wettschereck1 , Al´ıpio Jorge2, and Steve Moyle3 1

The Robert Gordon University, School of Computing St. Andrew Street, Aberdeen, AB25 1HG, UK [email protected] 2 LIACC-University of Porto Rua do Campo Alegre, 823, 4150 Porto, Portugal [email protected] 3 Oxford University Computing Laboratory Wolfson Building, Parks Road, Oxford, OX1 3QD, UK [email protected]

Abstract. The emerging standard for the platform- and system-independent representation of data mining models PMML (Predictive Model Markup Language) is currently supported by a number of knowledge discovery support engines. The primary purpose of the PMML standard is to separate model generation from model storage in order to enable users to view, post-process, and utilize data mining models independently of the tool that generated the model. In this paper two systems, called VizWiz and PEAR, are described. These software packages allow for the visualization and evaluation of data mining models that are specified in PMML. They can be viewed as decision support systems, since they enable non-expert users of data mining results to interactively inspect and evaluate these results.

1

Introduction

The Predictive Model Markup Language (PMML, [1]) industrial standard for the representation of data mining models has gained enough momentum to motivate the development of a new generation of decision support tools that utilize this standard, while avoiding the complexity of current knowledge discovery support engines (KDDSE). Such tools typically do not include the actual analysis methods, but concentrate on one or more of the other steps in the CRISP-DM [2] process. Such a separation of the actual model generation step from model handling enables users to view, post-process and utilize data mining models independently of the KDDSE that generated the model. The current PMML standard (version 2.0) supports the following types of models: decision trees, neural networks, center and density based clusters, general and polynomial regression, Naive Bayes, and association and sequence rules. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 493–501, 2003. c Springer-Verlag Berlin Heidelberg 2003


494

Dietrich Wettschereck et al.

Neither propositional nor first-order classification rule models are currently supported. We, therefore, also developed a non-standard extension to the PMML model for association rules to define (first-order) classification and regression rules as well as subgroups [4]. Further information about PMML and the Data Mining Group (DMG) can be obtained at the DMG.org web site [1]. A software architecture that integrates several key data mining and decision support techniques is proposed in [3]. The architecture aims at providing a modular knowledge discovery and decision support system for geographically dispersed groups that collaborate in different modes, depending on the complexity of the project and the resources available. The integration of models takes place in this architecture at the representation level and is based on PMML. At this point the visualization and evaluation of data models is a key issue. This issue is addressed in this paper. A further motivation for using PMML in this architecture is its key relevance in centralized model evaluation [5]. Only a standardized model representation language will allow for rapid, practical, reliable, and reproducible model handling in decision making. The paper is organized as follows: two software systems for the visualization and evaluation of data mining models are described in Section 2 in some detail. First, a visualization tool (VizWiz) for PMML models that is independent of a specific KDDSE is presented; and second, a post-processing methodology and an optimized tool (PEAR) for browsing and visualizing large sets of association rules. Section 3 concludes the paper with a discussion on how these tools can be utilized as decision support tools.

2

Model Visualization and Evaluation

Data visualization methods have been a part of statistics and data analysis research for many years. This research has concentrated primarily on plotting one or more independent variables against a dependent variable in support of explorative data analysis. The visualization of analysis results, however, only recently gained some attention with the proliferation of data mining and decision support techniques. This recent interest was spawned by the often overwhelming number and the complexity of data mining results. Within the framework described in the previous section, visualization serves four purposes: (1) to better illustrate the model to the end user, (2) to enable the comparison of models, (3) to increase model acceptance, and (4) to enable model editing and support for “what-if” questions. The ultimate goal of visualization is to support the user in making better decisions. Below, we describe two software systems that visualize data mining models which are specified in PMML. 2.1

VizWiz - A KDDSE-Independent PMML Visualizer

Within the SolEuNet project we developed a software component called VizWiz that visualizes PMML data mining models. The software includes visualizations for decision and regression trees, for subgroups, for propositional and first-order

Visualization and Evaluation Support of Knowledge Discovery

495

Fig. 1. A visualization of a decision tree

Fig. 2. A VizWiz visualization of a set of association rules

rules, and for association rules (see figures below). It is a Java implementation that can be run as an application or as an Applet.1 It allows for viewing of models by users that either do not have access to the actual KDDSE, that want to avoid the overhead of starting a KDDSE, or want to present their results via the internet. The original visualization wizard has been extended with model editing and model evaluation facilities (see [3], Chapter 5). Hence, the philosophy of VizWiz is to support all steps within the CRISP-DM [2] process that follow the actual modeling step. The remainder of this section will show several sample visualizations for the currently supported model types. Figure 1 shows a decision tree for the well known Play/Dont Play domain [6]. The tree is not fully expanded and is normally shown in color, where differently colored bars in the nodes denote the number of instances from each class contained in that node. The user can browse through the tree and open or close sub-trees as needed, the visualization can be changed from bar charts to pie charts and more detailed views can be shown for selected nodes. Figure 2 shows an interactive visualization for association rules. Confidence and support values are displayed for each rule. The bar above each rule graphically displays these two numbers where bar length represents the support and bar color represents its confidence (from red denoting low confidence to green 1

For a demonstration see: http://soleunet.ijs.si/website/other/pmml.html.

496

Dietrich Wettschereck et al.

Fig. 3. A visualization of propositional rules generated from the Cleveland Heart Disease domain denoting high confidence). Sliders at the bottom of the display (not shown) allow rules to be filtered out that lie outside selected minimal and maximal confidence and support values. An extension to the PMML model for association rules allows the definition of propositional and first-order rules (see [7, 8] for details). Figure 3 displays a set of propositional rules in the Cleveland Heart Disease domain [9]. The visualization is very similar to that of a set of multivariate decision trees: each tree displays the set of rules that predict a certain class. The leaves list the conditions that must be satisfied such that a certain rule fires. The display mode can be changed between bar charts to pie charts as shown for the second to last rule. Figure 4 displays a hand coded set of first-order rules that predict whether an actor is likely to reappear in a Star Trek Enterprise episode or not. The rules for each class are summarized by the bars at the top of the figure. The left bar shows the number of instances correctly covered, and the right bar the number of exceptions covered by the rule. The figure shows that rules may contain literals with bound or unbound variables, and that it is possible to replace Prolog-like notation with pretty text. Figure 5 displays a set of subgroups discovered by Midos [4] in a multirelational medical application. The size of each subgroup is shown, how it compares to the entire population, and the distribution of the target values within each subgroup. Experience gained from working with non-technical end users has shown that a pie chart visualization is more appealing to these users because they closely resemble business charts. Pie charts, however, often mislead the per-

Fig. 4. A visualization of three first-order classification rules

Visualization and Evaluation Support of Knowledge Discovery

497

ception of the user due to difficulties with relating the size of pie slices to actual values. Hence, alternative visualizations are possible (see, for example [10]).

Fig. 5. Visualization of selected subgroups learned from a multi-relational task VizWiz has also been extended with model evaluation facilities that allow the user to evaluate entire data sets against given models and to plot the evaluation results on a ROC Curve [11]. It is also possible to select single records and VizWiz will highlight the rule or tree node of the model that classifies this instance, and finally a rule or tree node can be selected and all instances covered by this partial model will be highlighted (Figure 6). This combination of model visualization and model evaluation offers an essential utility for collaborative data mining. The model evaluation features of VizWiz enable the user to interactively evaluate partial or entire data mining models on selected data sets or data records. The aim of such an evaluation phase would be to increase the end-user’s understanding of and confidence in the model. This is particularly valuable when the model was produced by someone else. The integration of VizWiz with the ROC Curve viewer permits the comparison and selection of models based on their predictive power. This integrated tool can, therefore, be seen as a powerful decision support tool for users of data mining results and as an effective presentation tool for producers of data mining results. 2.2

Post-processing Association Rules

PEAR is a Post-processing environment for association rules [12] that reads association models represented in PMML and allows the exploration of large sets of rules through rule set selection and visualization. PEAR allows the user to browse a large set of rules to ultimately find the subset of interesting/actionable rules. The central idea is that a large set of rules can be visited like a web site. PEAR is implemented as a set of dynamic web pages that can be accessed with an ordinary web browser. After loading a PMML file, an initial selection of the rules can be seen (Figure 7). This starting set is the index page of the whole set. To get to another subset of rules, the user can choose one of a list of intuitive operators (or change minimal support or minimal confidence). As an example, suppose we want to study the behavior of the users of one web site, by analyzing association rules obtained from web access data. In this case, one transaction corresponds to the set of page categories visited by one registered user. Rules indicate association between preferred page categories.

498

Dietrich Wettschereck et al.

Fig. 6. The evaluation of a selected data record (highlighted in table on the left). The path in the decision tree that leads to the leaf classifying this data record is marked in bold

Fig. 7. Subset of association rules as shown by PEAR

Visualization and Evaluation Support of Knowledge Discovery

499

Table 1. Some of the initial rules Rule Economics-and-Finance <= Population-and-Social-Conditions & Industry-and-Energy & External-Commerce Commerce-Tourism-and-Services <= Economics-and-Finance & Industry-and-Energy & General-Statistics Industry-and-Energy <= Economics-and-Finance & Commerce-Tourism-and-Services & General-Statistics Environment-and-Territory <= Population-and-Social-Conditions & Industry-and-Energy & General-Statistics General-Statistics <= Commerce-Tourism-and-Services & Industry-and-Energy & Environment-and-Territory External-Commerce <= Economics-and-Finance & Industry-and-Energy & General-Statistics Agriculture-and-Fishing <= Commerce-Tourism-and-Services & Environment-and-Territory & General-Statistics

Sup Conf 0.038 0.94 0.036 0.93 0.043 0.77 0.043 0.77 0.040 0.73 0.036 0.62 0.043 0.51

Table 2. Applying the consequent generalization operator Rule Environment-and-Territory Environment-and-Territory Environment-and-Territory Environment-and-Territory Environment-and-Territory

<= <= <= <= <=

Sup Conf Population-and-Social-Conditions & Industry-and-Energy & General-Statistics 0.043 0.77 Population-and-Social-Conditions & Industry-and-Energy 0.130 0.41 Population-and-Social-Conditions & General-Statistics 0.100 0.63 Industry-and-Energy & General-Statistics 0.048 0.77 General-Statistics 0.140 0.54

After rule generation, the PMML model is loaded. An initial page shows the 30 rules with highest support. Other possibilities for the initial page are the 30 rules with highest confidence or a set of rules involving different items to guarantee variability. Table 1 shows some of the rules in the initial page. The user then finds the rule on “Environment and Territory” relevant for structuring the categories on the site. By applying the consequent generalization operator to this rule, a new page with a subset of rules appears (Table 2). This operator results in the rules with the same antecedent but a more general consequent. From here, we can see that “Population and Social Conditions” is not relevantly associated to “Environment and Territory”. The user can now, for example, look into rules with “Population and Social Conditions” by applying the focus on the antecedent operator (results not shown here) to see what the main associations to this item are. For each page, the user can also select a graphical visualization that summarizes the set of rules on the page. Currently, the available visualizations are confidence/support plot and confidence/support histograms (Figure 8). The charts produced are interactive and indicate the rule that corresponds to the point under the mouse. After being loaded, rules are internally stored in a relational database, allowing the implementation of the operators as SQL queries. The charts are generated dynamically as Scalable Vector Graphics (SVG) pages, an XML based language.

3

Discussion

The visualization tools presented are simple, yet powerful, tools that can function as dissemination tools for data mining results. Their simplicity ensures that non-KDD users can operate the tools and interpret the results obtained by a data mining expert. Java/HTML technology ensures that platform issues are secondary, and that results could even be part of online content management or workgroup support systems. The philosophy behind VizWiz and PEAR differs in the sense that VizWiz attempts to cover as many different model types as

500

Dietrich Wettschereck et al.

Fig. 8. PEAR plot of confidence vs. support values for a set of association rules

possible while PEAR is a highly efficient and intuitive tool for one specific type of model.

Acknowledgements This work has been supported in part by the EU funded project Data Mining and Decision Support for Business Competitiveness: A European Virtual Enterprise (IST-1999-11495). Some of the visualizations presented here were developed by A. and G. Andrienko, AIS, FhG, Sankt Augustin, Germany. A. Jorge has also been supported by the POSI/2001/Class Project sponsored by Funda¸c˜ao Ciencia e Tecnologia, FEDER e Programa de Financiamento Plurianual de Unidades de I & D.

References [1] Data Mining Group, PMML specification, see http://www.dmg.org 493, 494 [2] Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C. and Wirth, R. (2000). CRISP-DM 1.0: step-by-step data mining guide. 493, 495 [3] Mladenic, D., Lavrac, N., Bohanec, M., Moyle, S. (editors): Data Mining and Decision Support: Integration and Collaboration, Kluwer Publishers, to appear. 494, 495 [4] Wrobel, S. (1997). An algorithm for multi-relational discovery of subgroups. Proc. First European Symposium on Principles of Data Mining and Knowledge Discovery, 78–87, Springer. 494, 496

Visualization and Evaluation Support of Knowledge Discovery

501

[5] Blockeel, H., Moyle, S. (2002). Centralized model evaluation for collaborative data mining. In M. Grobelnik, D. Mladenic, M. Bohanec, and M. Gams, editors, Proceedings A of the 5th International Multi-Conference nformation Society 2002: Data Mining and Data Warehousing/Intelligent Systems, pages 100–103. Jozef Stefan Institute, Ljubljana, Slovenia. 494 [6] Quinlan, J. R. (1993) C4.5: Programs for Machine Learning. Machine Learning. Morgan Kaufmann, San Mateo, CA, USA. 495 [7] Wettschereck, D., & M¨ uller, S. (2001). Exchanging data mining models with the predictive model markup language. In Proc. of the ECML/PKDD-01 Workshop on Integration of Data Mining, Decision Support and Meta-Learning, pp. 55-66. 496 [8] Wettschereck, D. (2002). A KDDSE-independent PMML Visualizer, in Proc. of IDDM-02, workshop on Integration aspects of Decision Support and Data Mining, (Eds.) Bohanec, M., Mladenic, D., Lavrac, N., associated to the conferences ECML/PKDD. 496 [9] Blake, C., Keogh, E., Merz, C. J. (1999). UCI repository of Machine Learning databases (Machine Readable data repository). Irvine, CA: Department of Information and Computer Science, University of California at Irvine. http://www.cs.uci.edu/ mlearn/MLRepository.html]. 496 [10] Gamberger, D., Lavraˇc, N., Wettschereck, D. (2002) Subgroup Visualization: A Method and Application in Population Screening. ECAI 2002 Workshop on INTELLIGENT DATA ANALYSIS IN MEDICINE AND PHARMACOLOGY, (2002). 497 [11] Provost, F., Fawcett, T. (2001). Robust Classification for Imprecise Environments. Machine Learning 42(3): 203-231. 497 [12] Jorge, A., Po¸cas, J., Azevedo P. (2002). Post-processing operators for browsing large sets of association rules, in Proceedings of Discovery Science 02, Luebeck, Germany, LNCS 2534, Eds. Steffen Lange, Ken Satoh, Carl H. Smith, SpringerVerlag. 497

Detecting Patterns of Fraudulent Behavior in Forensic Accounting Boris Kovalerchuk1 and Evgenii Vityaev2 1

Dept. of Computer Science, Central Washington University Ellensburg, WA 98926, USA [email protected] 2 Institute of Mathematics, Russian Academy of Sciences, Novosibirsk, 630090, Russia [email protected]

Abstract. Often evidence from a single case does not reveal any suspicious patterns to aid investigations in forensic accounting and other forensic fields. In contrast, correlation of sets of evidence from several cases with suitable background knowledge may reveal suspicious patterns. Link Discovery (LD) has recently emerged as a promising new area for such tasks. Currently LD mostly relies on deterministic graphical techniques. Other relevant techniques are Bayesian probabilistic and causal networks. These techniques need further development to handle rare events. This paper combines firstorder logic (FOL) and probabilistic semantic inference (PSI) to address this challenge. Previous research has shown this approach is computationally efficient and complete for statistically significant patterns. This paper shows that a modified method can be successful for discovering rare patterns. The method is illustrated with an example of discovery of suspicious patterns.

1

Introduction

Forensic accounting is a field that deals with possible illegal and fraudulent financial transactions [3]. One current focus in this field is the analysis of funding mechanisms for terrorism where clean money (e.g., charity money) and laundered money are both used [1] for a variety of activities including acquisition and production of weapons and their precursors. In contrast, traditional illegal businesses and drug trafficking make dirty money appear clean [1]. There are many indicators of possible suspicious (abnormal) transactions in traditional illegal business. These include (1) the use of several related and/or unrelated accounts before money is moved offshore, (2) a lack of account holder concern with commissions and fees [2], (3) correspondent banking transactions to offshore shell banks [2], (4) transferor insolvency after the transfer or insolvency at the time of transfer, (5) wire transfers to new places [4], (6) transactions without identifiable business purposes, and (7) transfers for less than reasonably equivalent value [5]. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 502-509, 2003.  Springer-Verlag Berlin Heidelberg 2003

Detecting Patterns of Fraudulent Behavior in Forensic Accounting

503

Some of these indicators can be easily implemented as simple flags in software. However, indicators such as wire transfers to new places produce a large number of 'false positive' suspicious transactions. Thus, the goal is to develop more sophisticated mechanisms based on interrelations among many indicators. To meet these challenges link analysis software for forensic accountants, attorneys and fraud examiners such as NetMap, Analyst's Notebook and others [4-7] have been and are being developed. Here we concentrate on fraudulent activities that are closely related to terrorism such as transactions without identifiable business purposes. The problem is that often an individual transaction does not reveal that it has no identifiable business purpose or that it was done for no reasonably equivalent value. Thus, we develop a technique that searches for suspicious patterns in the form of more complex combinations of transactions and other evidence using background knowledge. The specific tasks in automated forensic accounting related to transaction monitoring systems are the identification of suspicious and unusual electronic transactions and the reduction in the number of 'false positive' suspicious transactions by using inexpensive, simple rule-based systems, customer profiling, statistical techniques, neural networks, fuzzy logic and genetic algorithms [1]. This paper combines the advantages of first-order logic (FOL) and probabilistic semantic inference (PSI) [8] for these tasks. We discover the following transaction patterns from ordinary or distributed databases that are related to terrorism and other illegal activities: • a normal pattern (NP) – a Manufacturer Buys a Precursor & Sells the Result of manufacturing (MBPSR); • a suspicious (abnormal) pattern (SP) – a Manufacturer Buys a Precursor & Sells the same Precursor (MBPSP); • a suspicious pattern (SP) – a Trading Co. Buys a Precursor and Sells the same Precursor Cheaper (TBPSPC ); • a normal pattern (NP) -- a Conglomerate Buys a Precursor & Sells the Result of manufacturing (CBPSR).

2

Example

Consider the following example. Table 1 contains transactions with the attributes seller, buyer, item sold, amount, cost and date and Table 2 describes the types of companies and items sold. Table 1. Transactions records

Record ID 1 2 3 4 5 6

Seller Aaa Bbb Ttt Qqq Ccc Ddd

Buyer Ttt Ccc Qqq Ccc Ddd Ccc

Item sold Td Td Td Pd Td Pd

Amount 1t 2t 1t 1.5t 2.0t 3.0t

Cost $100 $100 $100 $100 $200 $400

Date 03/05/99 04/06/98 05/05/99 05/05/99 08/18/98 09/18/98

504

Boris Kovalerchuk and Evgenii Vityaev

We assemble a new Table 3 from Tables 1 and 2 to look for suspicious patterns. For instance, row 1 in Table 3 is a combination of row 1 from Table 1 and rows 1 and 4 from Table 2 that contain types of companies and items. Table 3 does not indicate suspicious patterns immediately, but we can generate pairs of records from Table 3 that can be mapped to patterns listed above using a pattern-matching algorithm A. The algorithm A analyzes pairs of records in Table 3. For simplicity, we can assume that a new table with 18 attributes is formed to represent pairs of records from Table 3. Each record in Table 3 contains nine attributes. Table 2. Company types and item types

Record ID 1 2 3 4 5 6

Company name Company type (seller/buyer) Aaa Trading Bbb Unknown Ccc Trading Ttt Manufacturing Ddd Manufacturing Qqq Conglomerate

Item

Item type in process PP

Td Pd Rd

Precursor Product Precursor

Table 3. Combined data records

Record Seller Seller ID type 1 2 1 Aaa trading 2 3 4 5 6

Buye r 3 Ttt

Buyer type 4 Manuf . Bbb unknown Ccc Tradin g Ttt manuf. Qqq Congl. Qqq Congl. Ccc Tradin g Ccc Trading Ddd Manuf . Ddd Manuf Ccc Tradin g

Item Item type Amount Price Date sold 5 6 7 8 9 Td Precursor 1t $100 03/05/99 Td

Precursor 2t

$100 04/06/98

Td pd

Precursor 1t Product 1.5t

$100 05/05/99 $100 06/23/99

td

Precursor 2.0t

$200 08/18/98

pd

Product

$400 09/18/98

3.0t

Thus, we map pairs of records in Table 3 into patterns: A(#5,#6)=MBPSR, that is a pair of records #5 and #6 from Table 3 indicates a normal pattern -- a manufacturer bought a precursor and sold product ; In contrast, two other pairs indicate suspicious patterns: A(#1,#3)= MBPSP, that is a manufacturer bought a precursor and sold the same precursor; A(#2,#5)= TBPSPC, that is a trading company bought a precursor and sold the same precursor cheaper.

Detecting Patterns of Fraudulent Behavior in Forensic Accounting

505

Now let us assume that we have a database of 105 transactions as in Table 1. Then Table 3 will have all pairs of them, i.e., about 5*109. Statistical computations can reveal a distribution of these pairs into patterns as shown in Table 4. Table 4. Statistical characteristics

Pattern

Type

Frequency, %

Approximate number of cases

MBPSR normal 55 0.55*5*109 MBPSP suspicious 0.1 100 CBPSR normal 44.7 0.44*5*109 TBPSPC suspicious 0.2 200 Thus, we have 300 suspicious transactions. This is 0.3% of the total number of transactions and about 6*10-6% of the total number of pairs analyzed. It shows that finding such transactions is like finding a needle in a haystack. The automatic generation of patterns/hypotheses descriptions is a major challenge. This includes generating MBPSP and TBPSPC descriptions automatically. We do not assume that we already know that MBPSP and TBPSPC are suspicious. One can ask: “Why do we need to discover these definitions (rules) automatically?” A manual way can work if the number of types of suspicious patterns is small and an expert is available. For multistage money-laundering transactions, this is difficult to accomplish manually. It is possible that many laundering transactions were processed before money went offshore or was used for illegal purposes. Our approach to identify suspicious patterns is to discover highly probable patterns and then negate them. We suppose that a highly probable pattern should be normal. In more formal terms, the main hypothesis (MH) is: If Q is a highly probable pattern (>0.9) then Q constitutes a normal pattern and not(Q) can constitute a suspicious (abnormal) pattern.

Table 5 outlines an algorithm based on this hypothesis to find suspicious patterns. The algorithm is based first-order logic and probabilistic semantic inference [8]. To minimize computations we generate randomly a representative part of all possible pairs of records such as shown in Table 4. Then an algorithm finds highly probable (P>T) Horn clauses. Next, these clauses are negated as described in Table 5. After that, a full search of records in the database is performed to find records that satisfy the negated clauses. According to our main hypothesis (MH) this set of records will contain suspicious records and the search for “red flag” transactions will be significantly narrowed. Use of the property of monotonicity is another tool we use to minimize computations. The idea is based on a simple observation: If A1&A2&…&An-1⇒ B represents a suspicious pattern then A1&A2&…&An-1&An⇒ B is suspicious too. Thus, one does not need to test clause A1&A2&…&An-1 &An⇒ B if A1&A2&…&An-1 ⇒ B is already satisfied.

506

Boris Kovalerchuk and Evgenii Vityaev

Table 5. Algorithm steps for finding suspicious patterns based on the main hypotheis (MH)

1

2

3

4

3

Discover patterns in a database such as MBPSR in a form MBP ⇒SR, that is, as a Horn clause A1&A2&…&An-1⇒ An (see [8] for mathematical detail). 1.1.Generate a set of predicates Q={Q1,Q2,…,Qm} and first order logic sentences A1,A2,…,An based on those predicates. For instance, Q1 and A1 could be defined as follows: Q1 (x)=1 Ù x is a trading company and A1(a)= Q1 (a)& Q1 (b), where a and b are companies. 1.2. Compute a probability P that pattern A1&A2&…&An-1⇒ An is true on a given database. This probability is computed as a conditional probability of conclusion An under assumption that If-part A1&A2&…&An-1 is true, that is P(An/A1&A2&…&An-1)= =N(An/A1&A2&…&An-1)/N(A1&A2&…&An1&An), where N(An/A1&A2&…&An-1) is the number of An/A1&A2&…&An-1 cases and N(A1&A2&…&An-1&An) is the number of A1&A2&…&An-1&An cases. 1.3. Compare P(A1&A2&…&An-1⇒ An) with a threshold T, say T=0.9. If P(A1&A2&…&An-1⇒ An )>T then a database is “normal”. A user can select another value of threshold T, e.g., T=0.98. If P(MBP⇒ SR)=0.998, then DB is normal for 0.98 too. 1.4. Test statistical significance of P(A1&A2&…&An-1⇒ An). We use the Fisher criterion [8] to test statistical significance. Negate patterns. If database is “normal” (P(A1&A2&…&An-1⇒ An) >T=0.9 and A1&A2&…&An-1⇒ An is statistically significant then negate A1&A2&…&An-1⇒ An to produce a negated pattern ┐( A1&A2&…&An-1⇒ An). Compute the probability of the negated pattern P(┐(A1&A2&…&An-1⇒ An)) as P(A1&A2&…&An-1⇒ ┐An ) = 1- P( ┐(A1&A2&…&An-1⇒ An). In the example above, it is 1-0.998=0.002. Analyze database records that satisfy A1&A2&…&An-1 & ┐An. for possible false alarm. Really suspicious records satisfy the property A1&A2&…&An- & ┐ An , but normal records also can satisfy this property.

Hypothesis Testing

One of the technical aims of this paper is to design tests and simulation experiments for this thesis. We designed two test experiments: 1.

2.

Test 1: Generate a relatively large Table 4 that includes a few suspicious records MBPSP and TBPSPC. Run a data-mining algorithm (MMDR [8]) to discover as many highly probable patterns as possible. Check that patterns MBPSR and CBPSR are among them. Negate MBPSR and CBPSR to produce patterns MBPSP and TBPSPC. Run patterns MBPSP and TBPSPC to find all suspicious records consistent with them. Test 2: Check that other highly probable patterns found are normal; check that their negations are suspicious patterns (or contain suspicious patterns).

Detecting Patterns of Fraudulent Behavior in Forensic Accounting

507

A positive result of Test 1 will confirm our hypothesis (statement) for MBPSR and CBPSR and their negations. Test 2 will confirm our statement for a wider set of patterns. In this paper we report results of conducting Test 1. The word “can” is the most important in our statement/hypothesis. If the majority of not(Q) patterns are consistent with an informal and intuitive concept of suspicious pattern then this hypothesis will be valid. If only a few of the not(Q) rules (patterns) are intuitively suspicious then the hypothesis will not be of much use even if it is formally valid. A method for Test 1 contains several steps: • •

• • • • •

Create a Horn clause: MBP ⇒ SR. Compute a probability that MBP ⇒ SR is true on a given database. Probability P(MBP =>SR) is computed as a conditional probability P(SR/MBP)=N(SR/MBP)/N(MBP), where N(SR/MBP) is the number of MBPSR cases and N(MBP) is the number of MBP cases. Compare P(MBP ⇒ SR) with 0.9. If P(MBP ⇒ SR)>0.9 then a database is ‘normal”. For instance, P(SR/MBP) can be 0.998. Test the statistical significance of P(MBP ⇒SR). We use Fisher criterion [8] to test statistical significance. If the database is “normal” (P(MBP ⇒ SR) >T=0.9) and if P(MBP ⇒SR) is statistically significant then negate MBP=>SR to produce ┐(MBP ⇒SR). Threshold T can have another value too. Compute probability P(┐(MBP ⇒SR)) as P(MBP ⇒ ┐(SR)) = P(┐(SR)/MBP)=1P(MBP⇒SR). In the example above it is 1-0.998=0.002. Analyze database records that satisfy MBP and ┐(SR). For instance, really suspicious MBPSP records satisfy property MBP and ┐(SR), but other records also can satisfy this property too. For instance, MBPBP records (a manufacturer bought a precursor twice) can be less suspicious than MBPSP.

Thus, if the probability P(SR/MBP) is high (0.9892) and statistically significant then a normal pattern MBPSR is discovered. Then suspicious cases are among the cases where MBP is true but the conclusion SR is not true. We collect these cases and analyze the actual content of the then-part of the clause MBP =>SR. The set ┐SR can contain a variety of entities. Some of them can be very legitimate cases. Therefore, this approach does not guarantee that we find only suspicious cases, but the method narrows the search to a much smaller set of records. In the example above the search is narrowed to 0.2% of the total cases.

4

Experiment

We generated two synthesized databases with attributes shown in Table 4. The first one does not have suspicious records MBPSP and TBPSPC. A second database contains few such records. Using a Machine Method for Discovery Regularities (MMDR) [8] we were able to discover MBPSR and CBPSR normal patterns in both databases.

508

Boris Kovalerchuk and Evgenii Vityaev Table 6. Database with suspicious cases

Pattern Normal pattern, MBP ⇒ SR Negated ┐(MBP ⇒SR), MBP ⇒ ┐(SR) Normal pattern CBP=> SR Negated pattern ┐(CBP ⇒ SR), CBP ⇒ ┐(SR)

Probability P(A1&A2&…&An-1⇒ An ) In database without In database with suspicious cases suspicious cases > 0.95 >0. 9 < 0.0.5 < 0.1 >0.95 > 0.9 <0.05 < 0.05

The MMDR method worked without any advanced information that these patterns are in data. In the database without suspicious cases, negated patterns MBP ⇒ ┐(SR) and CBP ⇒ ┐(SR) contain cases that are not suspicious. For instance, MBP ⇒BP, that is, a manufacturer that already bought precursors (transaction record 1) bought them again (transaction record 2). The difference in probabilities for MBP ⇒ ┐(SR) in the two databases points out actually suspicious cases. In our computational experiments, the total number of regularities found is 41. The number of triples of companies (i.e., pairs of transactions) captured by regularities is 1531 out of total 2772 triples generated in the experiment. Table 7 depicts some statistically significant regularities found. Attributes New_Buyer__type and New_Item_type belong to the second record in a pair of records (R1,R2). Individual records are depicted in table 3. Table 7. Computational experiment: examples discovered regularities

# Discovered regularity Frequency 1 IF Seller_type = Manufacturing AND Buyer__type = 72 / (6 + 72) = Manufacturing 0.923077 THEN New_Item_type = product 2 IF Seller_type = Manufacturing AND New_Buyer__type = 72 / (6 + 72) = Manufacturing THEN New_Item_type = product 0.923077 3 IF Seller_type = Manufacturing AND Item_type = precursor 152 / (59 + THEN New_Item_type = product 152) = 0.720379 4 IF Seller_type = Manufacturing AND Price_Compare = 1 AND 47 / (2 + 47) = New_Buyer__type = Trading THEN New_Item_type = product 0.959184 5 IF Seller_type = Manufacturing AND Price_Compare = 1 AND 79 / (5 + 79) = Item_type = precursor THEN New_Item_type = product 0.940476

5

Conclusion

The method outlined in this paper advances pattern discovery methods that deal with complex (non-numeric) evidences and involve structured objects, text and data in a variety of discrete and continuous scales (nominal, order, absolute and so on). The paper shows potential application of the technique for forensic accounting. The technique combines first-order logic (FOL) and probabilistic semantic inference (PSI). The approach has been illustrated with an example of discovery of suspicious patterns in forensic accounting.

Detecting Patterns of Fraudulent Behavior in Forensic Accounting

509

References [1] [2]

[3] [4] [5] [6] [7] [8]

Prentice, M., Forensic Services - tracking terrorist networks,2002, Ernst & Young LLP, UK, http://www.ey.com/global/gcr.nsf/UK/Forensic_Services_tracking_terrorist_networks Don Vangel and Al James Terrorist Financing: Cleaning Up a Dirty Business, the issue of Ernst & Young's financial services quarterly, Spring 2002. http://www.ey.com/GLOBAL/content.nsf/International/Issues_&_Perspectives _-Library_-Terrorist_Financing_Cleaning_Up_a_Dirty_Business IRS forensic accounting by TPI, 2002, http://www.tpirsrelief.com/forensic_accounting.htm Chabrow, E. Tracking The Terrorists, Information week, Jan. 14, 2002, http://www.tpirsrelief.com/forensic_accounting.htm How Forensic Accountants Support Fraud Litigation, 2002, http://www.fraudinformation.com/forensic_accountants.htm i2 Applications-Fraud Investigation Techniques, http://www.i2.co.uk/applications/fraud.html Evett, IW., Jackson, G. Lambert, JA , McCrossan, S. The impact of the principles of evidence interpretation on the structure and content of statements. Science & Justice 2000; 40: 233–239 Kovalerchuk, B., Vityaev, E., Data Mining in Finance: Advances in Relational and Hybrid Methods, Kluwer, 2000

SPIN!—An Enterprise Architecture for Spatial Data Mining Michael May and Alexandr Savinov Fraunhofer Institute for Autonomous Intelligent Systems Schloss Birlinghoven, Sankt-Augustin, D-53754 Germany {michael.may,alexandr.savinov}@ais.fraunhofer.de

Abstract. The rapidly expanding market for Spatial Data Mining systems and technologies is driven by pressure from the public sector, environmental agencies and industry to provide innovative solutions to a wide range of different problems. The main objective of the described spatial data mining platform is to provide an open, highly extensible, ntier system architecture based on Java 2 Platform, Enterprise Edition (J2EE). The data mining functionality is distributed among (i) Java client application for visualization and workspace management, (ii) application server with Enterprise Java Bean (EJB) container for running data mining algorithms and workspace management, and (iii) spatial database for storing data and spatial query execution.

1

Introduction

Data mining is the partially automated search for hidden patterns in typically large and multi-dimensional databases. It draws on results in machine learning, statistics and database theory [7]. Data mining methods have been packaged in data mining platforms, which are software environments providing support for the application of one or more data-mining algorithms. So far Data Mining and Geographic Information Systems (GIS) have existed as two separate technologies, each with its own methods, traditions and approaches to visualization and data analysis. Recently, the task of integrating these two technologies has become highly actual [3,8,9] especially as various public and private sector organizations possessing huge databases with thematic and geographically referenced data began to realize the huge potential of information hidden there. As a response to this demand a prototype has been developed [1,2] which demonstrates the potential of combining data mining and GIS. This initial prototype encouraged the development of the SPIN! [4,11,12] system the overall objective of which consists in developing a spatial data mining platform by integrating state of the art Geographic Information System (GIS) and data mining functionality in a closely coupled open and extensible system architecture. This paper describes an open, extensible architecture for spatial data mining, which pays special attention to such features as scalability, security, multi-user access, roV. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 510-517, 2003.  Springer-Verlag Berlin Heidelberg 2003

SPIN!—An Enterprise Architecture for Spatial Data Mining

511

bustness, platform independence and adherence to standards. It integrates Geographic Information System for interactive visual data exploration and Data Mining functionality specially adapted for spatial data. The system is built on the Java 2 Enterprise Edition (J2EE) architecture and particularly uses Enterprise Java Bean (EJB) technology for implementing remote object functionality. The flexibility and scalability of the J2EE platform has made it the platform of choice for building different multitiered enterprise applications so using it as a basis for a spatial data mining platform in SPIN! project [4] is a natural extension. EJB is a server-side component architecture, which cleanly separates the “business logic” (the analysis tools, in our case) from server issues, shielding the method developers from many technicalities involved in client-server programming. This choice allows us to meet the requirements often found in business applications, e.g. security, scalability, platform independence, in a principled manner. The system is tightly integrated with a relational database and can serve as data access and transformation tool for spatial and non-spatial data. Analysis tools can be integrated either as stand-alone modules or, more tightly, by distributing the analysis functionality between the database and the core algorithm. Particularly, spatial database is used to execute complex spatial queries generated by analysis algorithms. The final system integrates several data mining methods adapted to the analysis of spatial data, e.g., multi-relational subgroup discovery, rule induction and spatial cluster analysis, and combines them with rich interactive functionality for visual data exploration, thus offering an integrated distributed environment for spatial data analysis.

2

N-tier EJB-Based Architecture

The general SPIN! architecture is shown in Fig. 1. It is a n-tier Client/Serverarchitecture based on Enterprise Java Beans for the server side components. A major advantage of using Enterprise Java Beans is that such tasks as controlling and maintaining user access rights, handling multi-user access, pooling of database connections, caching, handling persistency and transaction management are delegated to the EJB container. The architecture has the following major subsystems: client, application server with one or more EJB containers, one or more database servers and optionally compute servers. The SPIN! client is a standalone Java application. It always creates one server side representative in the form of session bean the methods of which are accessed through the corresponding remote reference via Java RMI or CORBA IIOP protocol. The client session bean executes various server side tasks on behalf of the client. In particular, it may load/save workspace objects from/in its persistent state. The client is based on component connectivity framework, which is implemented in Java as connectivity library (CoCon). The idea is that the workspace consists of components each of which is considered a storage for a set of parameters and pieces of functionality (e.g., algorithms). The system functionality is determined by a set of available components.

512

Michael May and Alexandr Savinov Enterprise Java Bean Container

Database

Algorithm Session Bean

Data

Client Entity Bean

Workspace Entity Bean

Database Persistent object

Client Workspace Visual Component

Algorithm Component

JDBC (Connections) RMI/IIOP (References)

Fig. 1: SPIN! platform architecture. Main components are a Java-based client, an Enterprise Java Beans Container and one or more databases serving spatial and non-spatial data

The workspace components and connections among them can be edited in two views: tree view and graph view. In the tree view components from the system repository can be added into the workspace (Fig. 2, left top). User connections among workspace components can be established in the Connection Editor dialog. A more user friendly way of editing workspaces is through a Clementine-style workspace graph view, which shows both components and their user connections (Fig. 2, left bottom). In this view components can be added by selecting them from the system tool bar and connecting them by drawing arrows between graph nodes. It is also very important that components can be arranged within views into visually expressive diagrams. The application server is an Enterprise Java Bean container. It manages the client workspace, analysis tasks, data access and persistency. There may be more than one simultaneously running container on one or more servers so that, e.g., different algorithms and other tasks can be executed on different computers under different restrictions. The SPIN! system uses an EJB container for making workspaces persistent in the database and for remote computations. For the first task the client creates a special session bean, which is responsible on the server side for workspace persistence and access. Particularly, if the client needs to load or save a workspace it delegates this task to this session bean. The client creates one remote object for each analysis task to be run so that data is transferred directly from the database to the algorithm. After the analysis is finished its result is transferred to the client for visualization. User data are stored in primary data storage, which is a relational database system (it may be the same machine as the application server). There may be one or more optional secondary databases. In addition, data can be loaded from other sources – databases, ASCII files in the file system or Excel files. It is important that for remote computations in application server data is transferred directly into the remote algorithm bypassing the client. It is only a set of components (subgraph of the workspace) that is transferred between application server and client.

SPIN!—An Enterprise Architecture for Spatial Data Mining

513

Fig. 2. SPIN! client. The workspace consists of interconnected components such as database connections, database queries, data mining algorithms, analysis results and spatial object visualizers

3

Remote Algorithm Management

The developed architecture supposes that all algorithms are executed on compute servers. For each running algorithm a separate session bean is created which implements high-level methods for controlling its behavior, particularly, starting/stopping the execution, getting/setting parameters, setting the data to process, and getting the result. The session bean then is responsible for the methods implementation. There are several ways how it can be done. ƒ ƒ

ƒ

A clean and very convenient but in some cases inefficient approach is using Java for implementing the complete algorithm directly within the corresponding EJB, loading all data via JDBC into the workspace. A second approach divides the labor between the EJB container and the relational database. We have implemented a multi-relational spatial subgroup-mining algorithm [6] that does most of the analysis work (especially the spatial analysis) directly in the database. The EJB part retrieves summary statistics, manages hypotheses and controls the search. A third approach consists in implementing computationally intensive methods in native code wrapped into shared library by means of Java Native Interface (JNI). A rule induction algorithm based on finding largest empty intervals in data

514

ƒ ƒ

Michael May and Alexandr Savinov

[13,14] has been implemented in this way, namely, as a dynamically linked library the functions of which are called from the algorithm EJB. A fourth option is that the algorithm session bean directly calls an external executable module. This approach has been used to run SPADA algorithm [10]. And finally other remote objects (e.g. CORBA) can be used to execute the task.

The algorithm parameters are formed in the client and transferred to the algorithm EJB as a workspace component before the execution. In particular, data to be processed by the algorithm has to be specified. It is important that only a data description is specified and not the complete data set is transferred. In other words, the algorithm EJB gets information where and how to take data and what kind of restrictions to use. Thus when the algorithm starts, the data is directly retrieved by the algorithm EJB rather than passes through the client. For example, assume that we need to find interesting subgroups in spatially referenced data [6]. The data is characterised by both thematic attributes, e.g., population, and spatial attributes, e.g., proximity to highway or percentage of forests in the area. The data to be analysed is specified in the corresponding component where we can choose tables, columns, join and restriction conditions including spatial operators supported by the underlying database system. The algorithm component is connected to the data component and the subgroup pattern component. The algorithm component creates a remote algorithm object in the EJB container as a session bean and transfers to it all necessary components such as the data description. The remote object (EJB) starts computations while its local counterpart periodically checks its state until the process is finished. During computations the remote object retrieves data, analyses it and stores the result in the result component. Note that each client may start several local and remote analysis algorithms simultaneously and for each of them a separate thread is created. Once interesting subgroups have been discovered and stored in a component they can be visualised in a special view, which provides a list of all subgroups with all parameters as well as a two-dimensional chart where each subgroup is represented by one point according to its coverage and strength. Additionally, the data analysed by subgroup discovery data mining algorithm can be viewed in a geographic information system and analysed by visual analysis methods.

4

Workspace Management

One task, which is very important in distributed environment is workspace management. During one session user loads into the client and works with one workspace from some central storage. As the work is finished the workspace is stored back into its initial or new location. There are several alternatives how persistent workspace can be implemented: (i) the whole workspace is serialized and stored in one object like local/remote file or database record, and (ii) the workspace components and connections are stored separately in different database records. The first approach is much simpler but it is difficult to share workspaces. The second approach allows us to treat workspace components as individual objects even within persistent storage, i.e., the whole workspace graph structure is represented in the storage.

SPIN!—An Enterprise Architecture for Spatial Data Mining

Connection object Component object

Global graph in database

GRAPH PERSISTENCE MANAGER

Local (sub)graph in client

515

Connection table

Component table

Fig. 3. Workspace is a graph where nodes are components and edges are connections between them. All workspaces are stored in a database and retrieving a workspace means finding its component and connection objects. The persistent workspace management functionality is implemented as a session bean, which manipulates two types of entity beans: workspace components and workspace connections

We implemented both approaches and in both cases the workspace is represented as special graph object, i.e., a set of its nodes (workspace components) and a set of its edges (workspace connections). The graphs can be created from existing run-time workspace objects by specifying constraints on its nodes and connections. For example, for loading and storing workspaces view connections are ignored. Then the selected subgraph is passed to the persistence manager. If it needs to be stored as one object then the whole graph is serialized. Otherwise individual node and edge objects are serialized. We used XML for serialization, i.e., any object state is represented as an XML text. The functionality of remote workspace management is implemented by a special session bean. This EJB has functions for loading and storing workspaces. If the workspace is stored as a set of its constituents then the session bean uses entity beans, which correspond to the workspace components. The state of such workspaces is stored in two tables: one for nodes and one for edges. There exist two classes of entity beans, which are used to manipulate these two tables. The workspace management architecture for this case is shown in Fig. 3.

5

Conclusion

We have described the general architecture of the SPIN! spatial data mining platform. It integrates GIS and data mining algorithms that have been adapted to spatial data. The choice of J2EE technology allows us to meet requirements such as security, scal-

516

Michael May and Alexandr Savinov

ability, platform independence, in a principled manner. The system is tightly integrated with a RDBMS and can serve as data access and transformation tool for spatial and non-spatial data. The client has been implement in Java using Swing for its visual interface. Jboss 3.0 [5] has been used as an application server. Oracle 9i database has been used as a spatial data and workspace storage. In future it would be very interesting to add the following features to this architecture: persistent algorithms running with no client, web interface to data mining algorithms via conventional browser, data mining functionality as web services via XML-based SOAP protocol, shared workspaces where components can belong to more than one workspace.

Acknowledgement Work on this paper has been partially funded by the European Commission under IST-1999-10536-SPIN!

References [1]

Andrienko, N., G. Andrienko, A. Savinov, and D. Wettschereck, “Descartes and Kepler for Spatial Data Mining”, ERCIM News, No. 40, January 2000, 44–45. [2] Andrienko, N., Andrienko, G., Savinov, A., Voss, H. and Wettschereck, D., “Exploratory Analysis of Spatial Data Using Interactive Maps and Data Mining”, Cartography and Geographic Information Science 28(3), July 2001, 151-165. [3] Ester, M., Frommelt, A., Kriegel, H.P, Sander, J., “Spatial Data Mining: Database Primitives, Algorithms and Efficient DBMS Support”, in Data Minining and Knowledge Discovery, an International Journal, 1999. [4] European IST SPIN!-project web site, http://www.ccg.leeds.ac.uk/spin/. [5] JBoss Application Server, www.jboss.org. [6] W. Klösgen, May, M. Spatial Subgroup Mining Integrated in an ObjectRelational Spatial Database, PKDD 2002, Helsinki, Finland, August 2002, 275-286. [7] Klösgen, W., Zytkow, J. (eds.), Handbook of Data Mining and Knowledge Discovery. Oxford University Press, 2002. [8] Koperski, K., Adhikary, J., Han, J., 1996. Spatial Data Mining, Progress and Challenges, Vancouver, Canada, Technical Report [9] Koperski, K., Han, J. “GeoMiner: A System Prototype for Spatial Mining”, Proceedings ACM-SIGMOD, Arizona, 1997. [10] Lisi, F.A., Malerba, D., SPADA: A Spatial Association Discovery System. In A. Zanasi, C.A. Brebbia, N.F.F. Ebecken and P. Melli (Eds.), Data Mining III, Series: Management Information Systems, Vol. 6, 157-166, WIT Press, 2002.

SPIN!—An Enterprise Architecture for Spatial Data Mining

517

[11] May, M.: Spatial Knowledge Discovery: The SPIN! System. Fullerton, K. (ed.) Proceedings of the 6th EC-GIS Workshop, Lyon, 28-30th June, European Commission, JRC, Ispra. [12] May, M., Savinov, A. An integrated platform for spatial data mining and interactive visual analysis, Data Mining 2002, Third International Conference on Data Mining Methods and Databases for Engineering, Finance and Other Fields, 25-27 September 2002, Bologna, Italy, 51-60. [13] Savinov, A.: Mining Interesting Possibilistic Set-Valued Rules. In: Da Ruan and Etienne E. Kerre (eds.), Fuzzy If-Then Rules in Computational Intelligence: Theory and Applications, Kluwer, 2000, 107-133. [14] Savinov, A.: Mining Spatial Rules by Finding Empty Intervals in Data. Proc. of the 7th International Conference on Knowledge-Based Intelligent Information & Engineering Systems (KES'03), 3-5 September 2003, University of Oxford, United Kingdom (accepted).

The Role of Discretization Parameters in Sequence Rule Evolution Magnus Lie Hetland1 and P˚ al Sætrom2 1

Norwegian University of Science and Technology Dept. of Computer and Information Science Sem Sælands vei 9, NO–7491 Trondheim, Norway [email protected] 2 Interagon AS, Medisinsk-teknisk senter, NO–7489 Trondheim, Norway [email protected]

Abstract. As raw data become available in ever-increasing amounts, there is a need for automated methods that extract comprehensible knowledge from the data. In our previous work we have applied evolutionary algorithms to the problem of mining predictive rules from time series. In this paper we investigate the effect of discretization on the predictive power of the evolved rules. We compare the effects of using simple model selection based on validation performance, majority vote ensembles, and naive Bayesian combination of classifiers.

1

Introduction

As raw data become available in ever-increasing amounts, there is a need for automated methods that extract comprehensible knowledge from the data. The process of knowledge discovery and data mining has been the subject of much research interest in later years, and one recent subfield is that of sequence mining. In our previous work [3] we have applied evolutionary algorithms to the problem of mining predictive rules from time series. Our method involves discretizing the time series, in order to be able to evaluate our rules on them (which, in turn, allows us to use genetic programming to find such rules). This process of discretization discards some information about the time series, and the parameters chosen (such as the width of the discretization window) will determine which features are available for the mining algorithm. In this paper we investigate the effect of discretization on the predictive power of the evolved rules. We evolve rules for different discretization parameter settings and design predictors using the following methods: 1. We test rules evolved for different settings on a validation set. We then take the rule with the best performance to be our predictor. 2. We select the best rule for each of five discretization resolutions and combine them to form a predictor ensemble by simple majority vote. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 518–525, 2003. c Springer-Verlag Berlin Heidelberg 2003


The Role of Discretization Parameters in Sequence Rule Evolution

519

3. We select the best rule for each of five discretization resolutions and combine them by means of a naive Bayesian model. Majority vote ensembles are a simple way of combining several predictors to achieve increased predictive power. To make a prediction, all the component predictors make their predictions; a simple majority vote is used to decide which prediction “wins”. The naive Bayesian predictor is a simple probabilistic model that assumes conditional independence among the predictor variables. Even though this assumption may be too strong in many cases, there is much empirical evidence suggesting that the method is quite robust.

2

Method

In this section we first describe the general approach, as developed in [3], as well as the extensions introduced here, that is, combining predictors for several resolutions. Finally, the discretization method is discussed. 2.1

The General Mining Approach

The basic method works by using an evolutionary algorithm (EA) to develop rules that optimize some score (or fitness) function, which describes what qualities we are looking for. Three components are of central importance: The rule format, the mechanism for rule evaluation, and the fitness function. For the purposes of this discussion, we only consider one rule format: The rule consequent is some given event that may occur at given times, possibly extrinsic to the series itself, while the antecedent (the condition) is a pattern in a very expressive pattern language called IQL [5]. This language permits, among other things, such constructs as string matching with errors, regular expressions, shifts (latency), and Boolean combinations of expressions. In our EA system, these expressions are represented as syntax trees, in the standard manner. In order to calculate the fitness of each rule, we need to know at which positions in the data the antecedent is true. To ascertain this, we use the antecedent as a query in an information retrieval system, the Pattern Matching Chip (PMC) [4], for which our pattern language was designed. Such a chip is able to process about 100 MB/s. There are many possible fitness functions that measure the precision, degree of recall, or interestingness of a query or rule. For the relatively straightforward prediction task undertaken here, simple correlation is quite adequate. We use a supervised learning scheme, in which we mark the positions where want the rule to make a prediction, and then, for each rule, calculate the correlation between its behaviour and the desired responses. This can be calculated from the confusion matrix of the rule (true and false positives and negatives) [3]. It is worth mentioning that even though we focus mainly on predictive power here, one of the strengths of the method is the transparency of its rule format. Rather than a black-box predictor, the user receives a rule in a human-readable language. This can be an advantage in data mining contexts.

520

2.2

Magnus Lie Hetland and P˚ al Sætrom

Selecting and Combining Predictors

We want to investigate the effect of the parameters of the discretization process, or, more specifically, of the resolution and the alphabet size, on the predictive power of the developed rules. The specific meaning of resolution and alphabet size is given in Sect. 2.3; informally, the resolution refers to the width of each (overlapping) discretized segment of the time series, while the alphabet size is simply the cardinality of the alphabet used in the resulting strings. As mentioned in the introduction, we will compare three methods of exploiting the variability introduced by these parameters: model selection through cross-validation, majority vote ensembles, and naive Bayesian combination of predictors. The rules that are developed by our system have their performance tested on a validation set (done through a form of k-fold cross-validation). In the simple case, a single resolution and alphabet size is used, and the rule that has the best performance on the validation set is selected and tested on a separate test set. Instead of this simple approach, we use several resolutions and alphabet sizes, run the simple process for each parameter setting, and choose the rule (and, implicitly, the parameter setting) that has the highest validation rating. This way we can discover the resolution and alphabet size that best suits a given time series, without having to arbitrarily set these beforehand. To demonstrate the effect of this procedure, we also select the rule (and parameter setting) that has the lowest validation score, as a baseline. The next method is the majority vote ensemble. For this, we run the basic process for each parameter setting, and for each resolution we choose the rule (and, implicitly, the alphabet size) that has the best validation performance. We then combine these rules to form a single predictor, through voting. In other words, when we observe a time series (for example, the test set) we discretize it at all the resolutions, with the given alphabet sizes, (possibly incrementally) and run each rule in its own resolution. The prediction (“up” or “down”) that is prevalent (simple majority) among the rules is taken to be the decision of the combined predictor. Ensembles are described in more detail in [1]. The basic idea behind them is that if each of a set of classifiers (or predictors) is correct with a probability p ≥ 0.5 and the classifiers are generally not wrong at the same time (that is, they are somewhat independent) then a majority vote will increase the probability of a correct classification.1 More sophisticated ensemble methods exist; see [1] for details. The naive Bayesian predictor is a simple probabilistic model that assumes conditional independence among the predictor variables. Assume that we have a family {Xi } of predictor variables, and one dependent (predicted) variable Y . Assume that P (Y = y) is the a priori probability of a given value y being observed for Y , and P (Xi = xi | Y = y) is the conditional probability of 1

Note that because of this requirement, rules with lower than 50% validation accuracy were excluded from the ensembles.

The Role of Discretization Parameters in Sequence Rule Evolution

521

observing x for Xi , given that Y = y. Then, the naive Bayesian model states that our predicted class should be
P (Xi = xi |Y = y) . (1) ζ(x) = arg max P (Y = y) y

i

The independence assumption may in many cases be quite strong, but the naive Bayesian model can be surprisingly robust, and, as shown in [2], may even be optimal in cases where the independence assumption is violated. 2.3

Feature Extraction and Discretization

Many features may potentially be extracted from time series; for the task of prediction, we need to extract features in a way that lets us access the features of a sequence prefix. A natural approach is to divide the sequence into (overlapping) windows of width w, and to extract features from each of these. This discretization approach is described in the following. Our basic discretization process is a simple one, used, for example, in [7]. It works as follows. A time series x = (xi )ni=1 , where xi ∈ R, is discretized by the following steps: 1. Use a sliding window of width k to create a sequence w of m overlapping vectors. More formally: Create a sequence of windows w = (wi )m i=1 , r(i) where wi = (xj )j=l(i) , such that l(1) = 1, r(m) = n, r(i) − l(i) = k − 1 for 1 ≤ i ≤ m, and l(i) = l(i − 1) + 1 for 1 < i ≤ m. The window width is also referred to as the resolution. 2. For some feature function f : Rk → R, create a feature sequence f = (f (wi ))m i=1 . 3. Sort f and divide it into a segments of equal length. We refer to a as the alphabet size. 4. Use the limit elements in the sorted version of f to classify each element in f , creating a new sequence s where si is the number of the interval where fi is found. After this discretization process, we have a sequence s, where si is an integer such that 1 ≤ si ≤ a, and each integer occurs the same number of times in s. For convenience, we map these integers to lowercase letters (so that for a = 3 our alphabet is a. . . c). This discretization is performed on the training set, and the resulting limits are used to classify the features of the test set. (This means that there is no guarantee that each character is represented an equal number of times in the test set.) Many feature functions f are possible, such as average value or signal-to-noise ratio. Since we are interested in the upward and downward movement of the series, we have chosen to use the slope of the line fitted to the points in each window through linear regression.

522

Magnus Lie Hetland and P˚ al Sætrom 1.0 0.8 Accuracy

Worst 0.6

Best

0.4

Ensemble Bayesian

0.2

ng ch a Ex

ua hq rt Ea

e

ke

k et wo r N

ts Su

ns

po

G EC

R

an

do

m

0

Fig. 1. Performance comparison

3

Experiments

In our experiments we use six data sets, available from the UCR Time Series Data Mining Archive [6] (the first five) and the Federal Reserve Board [8] (the last data set): Random Walk. A standard random walk, where each value is equal to the previous one, plus a random number. ECG. ECG measurements from several subjects, concatenated. Earthquake. Earthquake-related seismic data. Sunspots. Monthly mean sunspot numbers from 1749 until 1990. The series is periodic, with one period of eleven years, and one of twenty-seven days. Network. Packet round trip time delay for packets sent from UCR to CMU. Exchange Rates. Daily exchange rate of Pound Sterling to US Dollars. For each of the data sets, the target prediction was when the series moved upward, that is, when the next value was greater than the current. It is to be expected that good predictors can be found for the ECG data (as it is quite regular and periodic), whereas the random walk data, and, to some extent, the exchange rate data, function as baselines for comparison. Finding predictors for random data would clearly indicate that our experiments were flawed, and finding good predictors for exchange rate data would also be surprising, as this is considered a quite difficult (if at all possible) task. For each of the data sets we performed experiments with alphabet sizes 2 (the minimum non-trivial case), 5, 10, and 20, as well as window sizes (resolutions) 2, 4, 8, 16, and 32. This was done with tenfold cross-validation2 and early 2

Note that full cross-validation was not used, due to the temporal nature of the data. The validation was constrained so that the elements of the training data occurred at an earlier time than the elements of the testing set.

The Role of Discretization Parameters in Sequence Rule Evolution

Random Walk 0.5004 0.5002 0.5 0.4998 0.4996 0.4994 0.4992 0.499

2

4

a-b a-e

0.62 0.6 0.58 0.56 0.54 0.52 0.5

2

a-b

2

a-b a-e

ECG

16

32 a-t

0.72 0.7 0.68 0.66 0.64 0.62 0.6 0.58

2

a-o Sunspots

4

a-e 0.64 0.62 0.6 0.58 0.56 0.54 0.52 0.5

8 a-j

8

a-j

8

a-j

a-o

a-b

4

a-e

16

32 a-t

0.66 0.64 0.62 0.6 0.58 0.56 0.54

2

a-o Earthquake

4

523

a-b

16

32 a-t

2

a-b a-e

a-j

16

32 a-t

16

32 a-t

16

32 a-t

a-o Network

4

a-e 0.53 0.525 0.52 0.515 0.51 0.505 0.5 0.495 0.49 0.485

8

8

a-j

a-o Exchange

4

8 a-j a-o

Fig. 2. Accuracy as function of window size and alphabet

stopping; that is, rules were selected based on their performance on a separate validation set before they were tested. As described in Sect. 2.2, results were used to select the alphabet size that gave the best single-predictor performance for each resolution, and these rules were then used in constructing ensembles and Bayesian classifiers. The rules in the ensembles were developed individually, that is, only their individual performance was used in calculating their fitness. The performance of the ensemble was then calculated by performing a simple majority vote among the rules for each position. The results are summarized in Fig. 1. The percentages (predictive accuracy) are averaged over the ten folds. For the ECG, sunspot, network, and earthquake data sets, the difference between the worst single classifier and the best single classifier is statistically significant (p < 0.01 with Fisher’s

524

Magnus Lie Hetland and P˚ al Sætrom

exact test), while the differences between the best single classifier, the ensemble, and the Bayesian classifier are not statistically significant (p > 0.05 for all except the difference between the best single classifier and the Bayesian combination for the Network data, where p = 0.049). For the random and exchange rate data sets there are no significant differences, as expected. Figure 2 shows how rule accuracy is related to window and alphabet size. For the random and exchange rate data, no clear trend is discernible. Although there are clearly problem specific differences, for the ECG, sunspot, and network data, there seems to be a rough inverse relationship between window size and accuracy, regardless of alphabet size. For the earthquake data, a large alphabet makes up for poor resolution, giving peak performance for the two largest window sizes and the most fine-grained alphabet.

4

Discussion

In this paper we have examined the role of discretization when evolving time series predictor rules. We used three main techniques to improve the basic evolution presented in [3]: Model selection based on validation performance, majority vote ensembles, and naive Bayesian classifiers. Prior to our empirical study, we expected the ensembles to outperform the simple selection, and the Bayesian classifiers to outperform both of the other methods. As it turns out, on our data, there was no statistically significant difference between the three methods, even though the difference between the result they produced and that produced by unfavorable parameter settings (discretization resolution and alphabet size) was highly significant. This leads to the conclusion that, given its simplicity, plain model selection may well be the preferred method. Our experiments also showed that the relationship between discretization resolution, alphabet size, and prediction accuracy is highly problem dependent, which means that no discretization parameters can be found that work equally well in all cases.

References [1] Thomas G. Dietterich. Ensemble methods in machine learning. Lecture Notes in Computer Science, 1857:1–15, 2000. 520 [2] Pedro Domingos and Michael J. Pazzani. On the optimality of the simple bayesian classifier under zero-one loss. Machine Learning, 29(2–3):103–130, 1997. 521 [3] Magnus Lie Hetland P˚ al Sætrom. Temporal rule discovery using genetic programming and specialized hardware. In Proc. of the 4th Int. Conf. on Recent Advances in Soft Computing, 2002. 518, 519, 524 [4] Interagon AS. Digital processing device. PCT/NO99/00308, Apr 2000. 519 [5] Interagon AS. The Interagon query language : a reference guide. http://www.interagon.com/pub/whitepapers/IQL.reference-latest.pdf, sep 2002. 519 [6] E. Keogh and T. Folias. The UCR time series data mining archive. http://www.cs.ucr.edu/~eamonn/TSDMA, sep 2002. 522

The Role of Discretization Parameters in Sequence Rule Evolution

525

[7] E. Keogh, S. Lonardi, and W. Chiu. Finding surprising patterns in a time series database in linear time and space. In Proc. of the 8th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pages 550–556, 2002. 521 [8] Federal Reserve Statistical Release. Foreign exchange rates 1971–2002. http://www.federalreserve.gov/Releases/H10/Hist, oct 2002. 522

Feature Extraction for Classification in Knowledge Discovery Systems Mykola Pechenizkiy1, Seppo Puuronen1, and Alexey Tsymbal2 1

University of Jyväskylä Department of Computer Science and Information Systems, P.O. Box 35, FIN-40351, University of Jyväskylä, Finland {mpechen, sepi}@cs.jyu.fi 2 Trinity College Dublin Department of Computer Science College Green, Dublin 2, Ireland [email protected]

Abstract. Dimensionality reduction is a very important step in the data mining process. In this paper, we consider feature extraction for classification tasks as a technique to overcome problems occurring because of “the curse of dimensionality”. We consider three different eigenvector-based feature extraction approaches for classification. The summary of obtained results concerning the accuracy of classification schemes is presented and the issue of search for the most appropriate feature extraction method for a given data set is considered. A decision support system to aid in the integration of the feature extraction and classification processes is proposed. The goals and requirements set for the decision support system and its basic structure are defined. The means of knowledge acquisition needed to build up the proposed system are considered.

1

Introduction

Data mining applies data analysis and discovery algorithms to discover information from vast amounts of data. A typical data-mining task is to predict an unknown value of some attribute of a new instance when the values of the other attributes of the new instance are known and a collection of instances with known values of all the attributes is given. In many applications, data, which is the subject of analysis and processing in data mining, is multidimensional, and presented by a number of features. The so-called “curse of dimensionality” pertinent to many learning algorithms, denotes the drastic raise of computational complexity and classification error with data having big amount of dimensions [2]. Hence, the dimensionality of the feature space is often tried to be reduced before classification is undertaken. Feature extraction (FE) is one of the dimensionality reduction techniques. FE extracts a subset of new features from the original feature set by means of some V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 526-532, 2003.  Springer-Verlag Berlin Heidelberg 2003

Feature Extraction for Classification in Knowledge Discovery Systems

527

functional mapping keeping as much information in the data as possible [5]. Conventional Principal Component Analysis (PCA) is one of the most commonly used feature extraction techniques. PCA extracts the axes on which the data shows the highest variability [7]. There exist many variations of the PCA that use local and/or non-linear processing to improve dimensionality reduction, though they generally do not use class information [9]. In our research, beside the PCA, we consider also two eigenvector-based approaches that use the within- and between-class covariance matrices and thus do take into account the class information. We analyse them with respect to the task of classification with regard to the learning algorithm being used and to the dynamic integration of classifiers (DIC). During the last years data mining has evolved from less sophisticated firstgeneration techniques to today's cutting-edge ones. Currently there is a growing need for next-generation data mining systems to manage knowledge discovery applications. These systems should be able to discover knowledge by combining several available techniques, and provide a more automatic environment, or an application envelope, surrounding this highly sophisticated data mining engine [4]. In this paper we consider a decision support system (DSS) approach that is based on the methodology used in expert systems (ES). The approach combines feature extraction techniques with different classification tasks. The main goal of such a system is to automate as far as possible the selection of the most suitable feature extraction approach for a certain classification task on a given data set according to a set of criteria. In the next sections we consider the feature extraction process for classification and present the summary of achieved results. Then we consider a decision support system that integrates the feature extraction and classification processes, describing its goals, requirements, structure, and the ways of knowledge acquisition. As a summary the obtained preliminary results are discussed and the focus of further research is described.

2

Eigenvector-Based Feature Extraction

Generally, feature extraction for classification can be seen as a search process among all possible transformations of the feature set for the best one, which preserves class separability as much as possible in the space with the lowest possible dimensionality [5]. In other words we are interested in finding a projection w: y = wT x

(1)

where y is a p'×1 transformed data point (presented using p' features), w is a p × p ' transformation matrix, and x is a p × 1 original data point (presented using p features). In [10] it was shown that the conventional PCA transforms the original set of features into a smaller subset of linear combinations that account for the most of the variance of the original data set. Although it is the most popular feature extraction technique, it has a serious drawback, namely the conventional PCA gives high

528

Mykola Pechenizkiy et al.

weights to features with higher variabilities irrespective of whether they are useful for classification or not. This may give rise to the situation where the chosen principal component corresponds to the attribute with the highest variability but having no discriminating power. A usual approach to overcome the above problem is to use some class separability criterion [1], e.g. the criteria defined in Fisher linear discriminant analysis and based on the family of functions of scatter matrices: J (w ) =

wT S B w w T SW w

(2)

where SB is the between-class covariance matrix that shows the scatter of the expected vectors around the mixture mean, and SW is the within-class covariance, that shows the scatter of samples around their respective class expected vectors. A number of other criteria were proposed in [5]. Both parametric and nonparametric approaches optimize the criterion (2) by using the simultaneous diagonalization algorithm [5]. In [11] we analyzed the task of eigenvector-based feature extraction for classification in general; a 3NN classifier was used as an example. The experiments were conducted on 21 data sets from the UCI machine learning repository [3]. The experimental results supported our expectations. Classification without feature extraction produced clearly the worst results. This shows the so-called “curse of dimensionality” with the considered data sets and the classifier supporting the necessity to apply some kind of feature extraction in that context. In the experiments, the conventional PCA was the worst feature extraction technique on average. The nonparametric technique was only slightly better than the parametric one on average. However, the nonparametric technique performed much better on categorical data. Still, it is necessary to note that each feature extraction technique was significantly worse than all the other techniques at least on a single data set. Thus, among the tested techniques there does not exist “the overall best” one for classification with regard to all given data sets.

3

Managing Feature Extraction and Classification Processes

Currently, as far as we know, there is no feature extraction technique that would be the best for all data sets in the classification task. Thus the adaptive selection of the most suitable feature extraction technique for a given data set needs further research. Currently, there does not exist canonical knowledge, a perfect mathematical model, or any relevant tool to select the best extraction technique. Instead, a volume of accumulated empirical findings, some trends, and some dependencies have been discovered. We consider a possibility to take benefit of the discovered knowledge by developing a decision support system based on the methodology of expert system design [6] in order to help to manage the data mining process. The main goal of the system is to recommend the best-suited feature extraction method and a classifier for a given data set. Achieving this goal produces a great benefit because it might be

Feature Extraction for Classification in Knowledge Discovery Systems

529

possible to reach the performance of the wrapper type approach by using the filter approach. In the wrapper type approach the interaction between the feature selection process and the construction of the classification model is applied and the parameter tuning for every stage and for every method is needed. In the filter approach the evaluation process is independent from the learning algorithm and the methods, and their parameters' selection process is performed according to a certain set of criteria in advance. However, the additional goal of the prediction of model's output performance requires also further consideration. The “heart” of the system is the Knowledge Base (KB) that contains a set of facts about the domain area and a set of rules in a symbolic form describing the logical references between a concrete classification problem and recommendations about the best-suited model for a given problem. The Vocabulary of KB contains the lists of terms that include feature extraction methods and their input parameters, classifiers and their input and output parameters, and three types of data set characteristics: simple measures such as the number of instances, the number of attributes, and the number of classes; statistical measures such as the departure from normality, correlation within attributes, the proportion of total variation explained by the first k canonical discriminants; and information-theoretic measures such as the noisiness of attributes, the number of irrelevant attributes, and the mutual information of class and attribute. Filling in the knowledge base is among the most challenging tasks related to the development of the DSS. There are two potential sources of knowledge to be discovered for the proposed system. The first is the background theory of the feature extraction and classification methods, and the second is the set of field experiments. The theoretical knowledge can be formulated and represented by an expert in the area of specific feature extraction methods and classification schemes. Generally it is possible to categorise the facts and rules that will be present in the Knowledge Base. The categorisation can be done according to the way the knowledge has been obtained – has it been got from the analysis of experimental results or from the domain theory. Another categorisation criterion is the level of confidence of a rule. The expert may be sure in a certain fact or may just think or to hypothesize about another fact. In a similar way, a rule that has been just generated from the analysis of results by experimenting on artificially generated data sets but has been never verified on realworlds data sets and a rule that has been verified on a number of real-world problems. In addition to the “trust“ criteria due to the categorisation of the rules it is possible to adapt the system to a concrete researcher's needs and preferences by giving higher weights to the rules that actually are the ones of the user.

4

Knowledge Acquisition from the Experiments

Generally, the knowledge base is a dynamic part of the decision support system that can be supplemented and updated through the knowledge acquisition and knowledge refinement processes [6]. Potential contribution of knowledge to be included into the KB might be found discovering a number of criteria from the experiments conducted on artificially generated data sets with pre-defined characteristics. The results of experiments can be

530

Mykola Pechenizkiy et al.

examined looking at the dependencies between the characteristics of a data set in general and the characteristics of every local partition of the instance space in particular. Further, the type and parameters of the feature extraction approach best suited for the data set will help to define a set of criteria that can be applied for the generation of rules of KB. The results of our preliminary experiments support that approach. The artificially generated data sets were manipulated by changing the amount of irrelevant attributes, the level of noise in the relevant attributes, the ratio of correlation among the attributes, and the normality of the distributions of classes. In the experiments, supervised feature extraction (both the parametric and nonparametric approaches) performed better than the conventional PCA when noise was introduced to the data sets. The similar trend was found with the situation when artificial data sets contained missing values. The finding was supported by the results of experiments on the LED17, Monk-3 and Voting UCI data sets (Table 1) that are known as ones that contain irrelevant attributes, noise in the attributes and a plenty of missing values. Thus, this criterion can be included in the KB to be used to give preference to supervised methods when there exist noise or missing values in a data set. Nonparametric feature extraction essentially outperforms the parametric approach on the data sets, which include significant nonnormal class distributions and are not easy to learn. This initial knowledge about the nature of the parametric and nonparametric approaches and the results on artificial data sets were supported by the results of experiments on Monk-1 and Monk-2 UCI data sets (Table 1). Table 1. Accuracy results of the experiments

Dataset LED17 MONK-1 MONK-2 MONK-3 Voting

5

PCA .395 .767 .717 .939 .923

Par .493 .687 .654 .990 .949

NPar .467 .952 .962 .990 .946

Plain .378 .758 .504 .843 .921

Discussions

So far we have not found a simple correlation-based criterion to separate the situations when a feature extraction technique would be beneficial for the classification. Nevertheless, we found out that there exists a trend between the correlation ratio in a data set and the threshold level used in every feature extraction method to address the amount of variation in the data set explained by the selected extracted features. This finding helps in the selection of the initial threshold value as a start point in the search for the optimal threshold value. However, further research and experiments are required to check these findings. One of our further goals is to make the knowledge acquisition process semiautomatic using the possibility of deriving new rules and updating the old ones based on the analysis of results obtained during the self-run experimenting. This process will include generating artificial data sets with known characteristics (simple,

Feature Extraction for Classification in Knowledge Discovery Systems

531

statistical and information-theoretic measures); running the experiments on the generated artificial data sets; derivation of dependencies and definition of criteria from the obtained results and updating the knowledge base; validating the constructed theory with a set of experiments on real-world data sets, and reporting on the success or failure of certain rules. We consider a decision tree learning algorithm as a mean of automatic rule extraction for the knowledge base. Decision tree learning is one of the most widely used inductive learning methods [12]. A decision tree is represented as a set of nodes and arcs. Each node contains a feature (an attribute) and each arc leaving the node is labelled with a particular value (or range of values) for that feature. Together, a node and the arcs leaving it represent a decision about the path an example follows when being classified by the tree. Given a set of training examples, a decision tree is induced in a “top-down” fashion by repeatedly dividing up the examples according to their values for a particular feature. In this context, mentioned above data set characteristics and a classification model's outputs that include accuracy, sensitivity, specificity, time complexity and so on represent instance space. And the combination of a feature extraction method's and a classification model's names with their parameter values represent class labels. By means of analysing the tree branches it is possible to generate “if-then” rules for the knowledge base. A rule reflects certain relationship between meta-data-set characteristics and a combination of a feature extraction method and a classification model.

6

Conclusions

Feature extraction is one of the dimensionality reduction techniques that are often used to cope with the problems caused by the “curse of dimensionality”. In this paper we considered three eigenvector-based feature extraction approaches, which were applied for different classification problems. We presented the summary of results that shows a high level of complexity in dependencies between the data set characteristics and the data mining process. There is no feature extraction method that would be the most suitable for all classification tasks. Due to the fact that there is no well-grounded strong theory that would help us to build up an automated system for such feature extraction method selection, a decision support system that would accumulate separate facts, trends, and dependencies between the data characteristics and output parameters of classification schemes performed in the spaces of extracted features was proposed. We considered the goals of such a system, the basic ideas that define its structure and methodology of knowledge acquisition and validation. The Knowledge Base is the basis for the intelligence of the decision support system. That is why we recognised the problem of discovering rules from the experiments of an artificially generated data set with known predefined simple, statistical, and informationtheoretic measures, and validation of those rules on benchmark data sets as a prior research focus in this area. It should be noticed that the proposed approach has a serious limitation. Namely the drawbacks can be expressed in the terms of fragmentariness and incoherence (disconnectedness) of the components of knowledge to be produced. And we

532

Mykola Pechenizkiy et al.

definitely do not claim the completeness of our decision support system. Otherwise, certain constrains and assumptions to the domain area were considered, and the limited sets of feature extraction methods, classifiers and data set characteristics were considered in order to guarantee the desired level of confidence in the system when solving a bounded set of problems.

Acknowledgments This research is partly supported by the COMAS Graduate School of the University of Jyväskylä, Finland and Science Foundation, Ireland. We would like to thank the UCI ML repository of databases, domain theories and data generators for the data sets, and the MLC++ library for the source code used in this study.

References [1]

Aivazyan, S.A.: Applied Statistics: Classification and Dimension Reduction. Finance and Statistics, Moscow, 1989. [2] Bellman, R., Adaptive Control Processes: A Guided Tour, Princeton University Press, 1961. [3] Blake, C.L., Merz, C.J. UCI Repository of Machine Learning Databases [http://www.ics.uci.edu/~mlearn/ MLRepository.html]. Dept. of Information and Computer Science, University of California, Irvine CA, 1998. [4] Fayyad U.M. Data Mining and Knowledge Discovery: Making Sense Out of Data, IEEE Expert, Vol. 11, No. 5, Oct., 1996, pp. 20-25. [5] Fukunaga, K. Introduction to Statistical Pattern Recognition. Academic Press, London, 1991. [6] Jackson P. Introduction to Expert Systems, 3rd Edn. Harlow, England: Addison Wesley Longman, 1999. [7] Jolliffe, I.T. Principal Component Analysis. Springer, New York, NY. 1986. [8] Kohavi, R., Sommerfield, D., Dougherty, J. Data mining using MLC++: a machine learning library in C++. Tools with Artificial Intelligence, IEEE CS Press, 234-245, 1996. [9] Liu H. Feature Extraction, Construction and Selection: A Data Mining Perspective, ISBN 0-7923-8196-3, Kluwer Academic Publishers, 1998. [10] Oza, N.C., Tumer, K. Dimensionality Reduction Through Classifier Ensembles. Technical Report NASA-ARC-IC-1999-124, Computational Sciences Division, NASA Ames Research Center, Moffett Field, CA, 1999. [11] Tsymbal A., Puuronen S., Pechenizkiy M., Baumgarten M., Patterson D. Eigenvector-based feature extraction for classification, In: Proc. 15th Int. FLAIRS Conference on Artificial Intelligence, Pensacola, FL, USA, AAAI Press, 354-358, 2002. [12] Quinlan, J.R. 1993. C4.5 Programs for Machine Learning. San Mateo CA: Morgan Kaufmann.

Adaptive Per-application Load Balancing with Neuron-Fuzzy to Support Quality of Service for Voice over IP in the Internet Sanon Chimmanee, Komwut Wipusitwarakun, and Suwan Runggeratigul Sirindhorn International Institute of Technology (SIIT) Thammasat University, Thailand. {Sanon,komwut,suwan}@siit.tu.ac.th [email protected]

Abstract. This paper presents a per-application load balancing with adaptive feedback control to optimize both QoS requirement for VoIP application (first target) and effective link usages (second target) over the Internet. In order to forward the Internet applications over the route coupling with respect to QoS for VoIP, the perceptron neural net is used to classify the applications with similar properties belong to one of two groups. However, the perceptron has some limitations such as nonoptimization for several targets. Therefore, this paper proposes to develop the perceptron with fuzzy control approach to eliminate the limitation of perceptron and enhance the capability of perceptron.

1

Introduction

Nowadays, real-time connectionless technology applications such as Voice over IP (VoIP) are becoming increasingly popular over the Internet. However, TCP/IP protocols are originally developed without the idea of supporting the quality of service (QoS) for real-time applications. The load balancing approach is one possible method for improving the QoS of VoIP application over the Internet since a) the load balancing approach allows the system to exploit the availability of the network resources to increase throughput and reduce end-to-end delay of the system, which results in a better QoS [1], and b) it can provide additional bandwidths to alleviate the degrading bandwidth performance due to the network unreliability of the Internet [2]. Although the existing load balancing tools that are per-packet and per-destination [1],[2] are already implemented on routers, they can not support both the QoS and the equal usage of links simultaneously. In addition, they cannot selectively control the QoS of specific real-time applications such as the VoIP traffic because packets of different applications such FTP trains and VoIP stream may be distributed to the same link. Traditionally , the time-delay problem of delay-sensitive applications arises in IP networks when their data streams are behind large data transfers such as FTP trains [2]. In other words, they are developed without considering the properties of the network applications, and hence introduce a low QoS for VoIP application. This has V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 533-541, 2003.  Springer-Verlag Berlin Heidelberg 2003

534

Sanon Chimmanee et al.

led [1],[2] to present ARLB that can “selectively” control the QoS for VoIP stream with static rule approach. However, the Internet traffic pattern is uncertain, which results in unequal link usages. Therefore, the challenge in designing load balancing tools is how to optimize both QoS requirements and link usages. The proposed method, which is referred to as Adaptive Per-Application Load Balancing (APALB), uses the concept of application classification to enable the system to selectively control the QoS, and then tries to optimize the link usages and maintain the QoS by using a feedback control. In other words, the APALB is an extension version of ARLB, that is the per-application load balancing, with adaptive approach. The merit of the new version is to allow applications can belong to one of the two groups dynamically and result in optimizing the desired targets at the same time. Perceptron is one of the simple neural nets that are widely used to classify whether an input belongs to one of the two classes [3][7], while ADALINE and Back-propagation are generalizations of the Least Mean Square (LMS) algorithm [7]. We found that if we use the perceptron to classify applications belong to one of two groups. The first target, that is to support QoS for VoIP, can be achieved. However, the second target, that is the effective link usages, can not be achieved since a) the nature of the perceptron is not try to minimize an error between output and target like ADALINE [7]. b) the original perceptron is not developed for adaptive classification and for the two desired targets [7]. Therefore, the fuzzy controller [6] is used to train the neural net to allow the perceptron to be able to minimize the error and support the desired targets. Based on simulation results, the proposed APALB with neuron-fuzzy approach [4][5] can reduce the time-delay of VoIP packets, decrease the unequal usage of links, and increase the network throughput when compared to the existing load balancing tools.

2

Adaptive Per-application Load Balancing Algorithm

APALB classifies the certain application as either belonging to Group1(G1) or Group2(G2) by using the proposed application metric (AM) for evaluating the cost of each application. Then the application of G1 and G2 will be forwarding to the different links. Since the Internet traffic pattern is uncertain, the criterion for evaluating cost should be adjusted to support the targets. Low cost means that its current properties, e.g., packet size, is similar to VoIP and therefore it can be the member of G1. High cost means that its current properties is much more different from VoIP and therefore it has to be member of G2. APALB consists of two major parts: the initial weight state with the perceptron and the adaptive weight state with the neuron-fuzzy. In the initial state, G1 contains only VoIP application and G2 contains other applications. Therefore, this part is to train the perceptron to evaluate the AM formula in Eq. (1) called “weights for the initial state”. After training is done, the AM cost of all VoIP patterns are less than or equal to zero. AM cost for other applications are higher than zero and therefore they are belong to G2. This mean that, VoIP will obtain all network resources of G1 due to having only VoIP. In the adaptive state, the fuzzy controller will adjust the weight so that other applications can be in G1, therefore G1 may contain both VoIP and other applications as well. This process is called “weights for the adaptive state” which results in optimizing both the usage of network resources and QoS for VoIP.

Adaptive Per-application Load Balancing with Neuron-Fuzzy INPUT LAYER

535

OUTPUT LAYER /A

X1

Input: Application

W1 /B

X2

/C

W2

Z

Output: Classifying application whether belongs to one of two groups

W3 Threshold

X3

Fig. 1. A single-layer perceptron neural net

2.1

Initial Weight State with the Perceptron

The initial weight with the perceptron has performed and obtained the initial weights in advance. The results are displayed in the subsection 4.1. The formula of application metric (AM) uses the application's properties that including packet size, volume traffic, and RTP, to be in the form of the perceptron as AM = W1(X1/A) + W2(X2/B) + W3(X3/C)

(1)

where X1 is the average packet size (bytes) , X2 is the average traffic volume in packet per second (pps). If the application is the Real Time application Protocol (RTP), X3 is 1. Otherwise, X3 is 2. A is the packet size coefficient , B is the traffic volume coefficient, C is the RTP coefficient, W1 is the packet weight, W2 is the traffic volume weight, and W3 is the RTP weight. From the Table 4, the maximum average packet size and traffic volume of VoIP is 70 byte and 50 pps, respectively. Since the characteristic of VoIP is used as criteria for classifying application into two groups, the evaluation cost for each element of Eq. (1), such as X1, is compared to VoIP properties by assigned A = 70, B = 50, and C = 1. This approach allows us to evaluate the cost for each application based on the characteristic of VoIP. Then, the perceptron is trained with 600 different Internet application patterns to adjust the appropriate weights for classification VoIP application to G1 and other applications to G2. It should be noted that the system uses the traditional port number in IP packets to specify the type of application of packets. The single node perceptron as shown in Fig. 5 computes a weighted sum of the input elements, subtracts threshold and passes the result through a hard-limit function [3],[7]. Z_in is the net input to output unit Z (2) the transfer function is the hard-limit (fh)

(3)

536

Sanon Chimmanee et al.

Update weights: Wi(new) = Wi(old) + ∆Wi

(4)

Update threshold: θm (new) = θm (old) + ∆θm

(5)

where, θm is the threshold. 2.2

Adaptive Weight State with Neuron-Fuzzy

The neuron-fuzzy consists of 2 mechanisms: the perceptron and the fuzzy control for training the perceptron. After fuzzy control has trained the perceptron, the perceptron classifies applications belonging to G1 or G2 dynamically. Its diagram is shown in Fig.2. 2.2.1 Perceptron Neural Net. The criteria for the adaptive application classification can be expressed as (6) where Z_in is the net input to output unit Z The transfer function is the same as Eq. (3). The errors (that are unequal usage of links and high time-delay for VoIP) will be reduced most rapidly by adjusting the weights according to the following condition: Update weights: Wi(new) = Wi(old)+ WF

(7)

where WF is the weight update that is produced by training of fuzzy controller. The resulting weights in the initial state are used as the initial values of the weights for the adaptive and fix the threshold equal to 3. 2.2.2 Fuzzy Control for Training the Perceptron : The fuzzy controller contains three layers: input layer, hidden layer, and output layer. The error is determined from the input information and then the controller will produce the output that is the weight update (WF) to adjust the weight connection of perceptron in Eq. (7). A. Input Layer: the usage of links and the delay are the two main input parameters. A.1) Usage of links parameter consists of two elements. The first one is the Usage of links Error, ∆EUS(t),that is the difference between the usage of link1 and link2 , each is denoted by US1(t) and US2(t), respectively.

∆EUS(t) = US1(t) –US2(t)

(8)

The other is the Change-in- link usage error, ∆EUS(t)′, which can be defined as

∆EUS(t)′ = d(∆EUS(t))/dt

(9)

Adaptive Per-application Load Balancing with Neuron-Fuzzy Reference model or Target

Fuzzy Controller

Delay error

+ - Link Usage error +

Target for Delay (RD(t))

Target for Link Usage (RUS(t))

-

537

d/d(t)

Delay force (FD)

d/d(t)

Usage force (FUS)

FD(t) Weight force (WF ) FUS(t)

Weight update to train the perceptron INPUT X(t)

Perceptron Neural Net (Classified by AM)

Packet Forwarding (Plant)

Link usage output (YUS(t)) Delay output (YD(t))

Adjust Weights

Fig. 2. Block diagram of APALB contains two major parts: the perceptron and the fuzzy control known as the neuron-fuzzy. YUS is the link usage output, YD is the delay output, RUS is the link usage target or reference, RD is the delay target or reference, ∆US is the link usage error, ∆D is the delay error, FD is the delay force for maintaining the QoS for VoIP (first target), FUS is the link usage force for balancing the usage of link1 and link2 (second target), and WF is the weight update to train the perceptron, which results in optimizing both the end-to-end delay for VoIP and the effective usage of links

A.2) Delay parameter consists of two elements. The first one is the Delay Error, ∆ED(t), that is the difference between θDelay and, D(t), which can be defined as

∆ED(t) = D(t) -θDelay

(10)

where θDelay is a threshold for starting the adaptive control to the system. If the delay value is longer than 100 ms, the system must start to adjust itself with feedback control to maintain QoS for VoIP. As a result, θDelay is equal to 100 ms. D(t) is the current time-delay of VoIP. The other is the Change-in-error of the Delay Error, ∆ED(t)′, which can be defined as (11) ∆ED(t)′ = d(∆ED(t))/dt B. Hidden Layer : There are also two main elements. The first one is the usage of links force, FUS, that is a force sourced from ∆EUS(t) and ∆EUS(t)′. The other is the delay force, FD, that is a force sourced from ∆ED(t) and ∆ED(t)′. C. Output Layer: Weight force, WF, is a force sourced from FUS and FD. D. Rule-Base for Fuzzy Control: Table 1. Rule Table for the FUS “Link usage “Change-in-Link usage error” ∆EUS′ force”FUS -2 -1 0 1 2 -2 -2 -2 -2 1 0 -1 -2 -2 -1 0 1 “Error” 0 -2 -1 0 1 1 ∆EUS 1 -1 0 1 2 2 2 0 1 2 2 2

538

Sanon Chimmanee et al. Table 2. Rule Table for the FD “Delay force” FD “error” ∆E D

0 1 2

“Change-in-delay error” ∆ED ′ -2 -1 0 1 2 0 0 0 1 2 0 0 1 2 2 0 1 2 2 2

Table 3. Rule Table for the WF “Weight WF FD

force” 0 1 2

-2 -2 -1 2

-1 -1 0 2

FUS 0 0 1 2

1 1 2 2

2 2 2 2

E. Membership Functions:

Fig. 3. Membership function for ∆EUS(t), ∆EUS(t)′,FUS(t) , ∆ED(t), ∆ED(t)′, FD(t) , and WF (t)

3

Simulation Model

3.1

Traffic Load Model

We obtained a total of 600 different Internet application patterns based on actual traffics measured at our Intranet that is summarized in Table4. The Internet application patterns were recorded in the database, which are used for training

Adaptive Per-application Load Balancing with Neuron-Fuzzy

539

perceptron and simulator. We have built the simulator for generating the Internet traffic load by randomizing the traffic volume and packet sizes of application from the database. Table 4. The summary table of average value for the Interent application patterns based on TCP/IP layer 3 Application VoIP Telnet DNS SMTP Http FTP

3.2

Packet Size (bytes) Avg. Max Avg. Mean Avg. Min 70 66.6 66 154 109 79 180 126 102 668 446 375 670 503 164 919 752 597

Traffic Volume (pps) Avg. Max Avg. Mean Avg. Min 50 31.4 10 8.5 4.9 1 14 7.4 1.9 8.2 3.6 1.6 15 5.2 1 6 4.4 2.7

Network Configuration Models

Initial value of the network delay of each route is randomized within the range of 40 to 100 ms. In addition, initial value of the access bandwidth (Internet bandwidth) of each route is randomized within the range of 45 to 256 kbps. And then these value may be increased, remaining constant, and decreased after every 10 seconds by adding a random value within the range of –10 to 10 percentage of their initial values.

4

Experimental Results

4.1

Training the Perceptron for the Initial Weight State

In the initial state, we use the MatLab release 12 to be perceptron tool for finding out weights for initiate state that are W1,W2, W3, and Threshold. Given the initiate values for W1,W2,W3, and threshold with 0. In addition, the targets are given as follows: AM cost of VoIP is less than or equal to zero and AM cost of other applications are higher than zero. Therefore, After training (weights adjustment) with the total 600 different Internet application patterns is done, the appropriate values for W1,W2,W3, and threshold are 1.1271, -3.69, 1, and 3, respectively.

Fig. 4. Network Configuration Model for experimental simulation

540

Sanon Chimmanee et al.

4.2

Performance Evaluation

4.2.1 Performance Evaluation on Unequal Usages and Network Throughput: The result is that APALB, ARLB and per-destination introduce 22 %, 30 %, and 58 % of the unequal usage of links, respectively. In addition, APALB, ARLB and perdestination introduce 86 %, 74 %, and 66 % of the network throughput, respectively. We see that if the unequal link usage decrease, the throughput will increase. In other words, the APALB can increase the network throughput when compared to perdestination and ARLB by up to 20 % and 12%, respectively. However, the unequal link usages is higher and throughput is less than compared with per-packet since APALB has to optimize not only usage of link but also the QoS . 4.2.2 Performance Evaluation on QoS: It can be seen that APALB can reduce the number of packets that take more than 150 ms when compared to per-packet and perdestination by up to 80% and 72%, respectively. In addition, it decreases the average end-to-end delay when compared to per-packet and per-destination by up to 34% and 22%, respectively. This means that APALB can offer a better quality of voice. However, the QoS performances of APALB and ARLB are nearly the same.

5

Conclusion

The APALB uses the concept of classification and then try to optimize link usages with respect to maintain the QoS for VoIP. Unfortunately, the perceptron has only a capability of classification. Therefore, fuzzy control is introduced to enhance the capability of perceptron in order to support optimization for the desired targets. The APALB has a better performance than the existing load balancing since it is the novel adaptive per-application load balancing with the intelligent feedback control.

References [1] [2]

[3] [4]

S. Chimmanee, K. Wipusitwarakun and S. Runggeratigul : Load Balancing for Zone Routing Protocol to Support QoS in Ad Hoc Network, Proc. ITC-CSCC 2002, vol.3, pp.1685-1688, 16-19 July, 2002. S. Chimmanee, K. Wipusitwarakun, P. Termsinsuwan and Y. Gando : Application Routing Load Balancing (ARLB) to support QoS for VoIP application over VPN Environment, Proc. IEEE NCA 2001, pp.94-99, 11-13 Feb, 2002. A. Ali and R. Tervo : Traffic Identification Using Artificial Neural Network, Canadian Conference on Electrical and Computer Engineering 2001, Vol. 1, pp.667-671. Lazzerini, L. M. reyneri and M. Chiaberge : Neuron-Fuzzy Approach to Hybrid Intelligent Control, IEEE transactions on industry applications, Vol.35, No.2, March/April 1999.

Adaptive Per-application Load Balancing with Neuron-Fuzzy

[5] [6] [7]

541

L. C. Jain : Intelligent Adaptive Control Industrial Applications, 1999 by CRC Press LLC. K. M. Passino and S. Yurkovich : Fuzzy Control, Addison Wesley Longman, Inc.,1998. L. Fausett, “Fundamentals of Neural Networks Architectures, Algorithms, and Applications”, 1994 by Prentice-Hall, Inc., New Jersey.

Hybrid Intelligent Production Simulator by GA and Its Individual Expression Hidehiko Yamamoto and Etsuo Marui Faculty of Engineering, GifuUniversity,1-1, Yanagido, Gifu, Japan [email protected]

Abstract: This paper is focused on the problem for one-by-one products input into Flexible Transfer Line (FTL) and describes an algorithm for a hybrid intelligent off-line production simulator connected Genetic Algorithm (GA) system in order to search a better solution. New individual expression, crossover operations, mutation operations in GA and their application examples are also described. The developed hybrid simulator to have a wide solution search space can solve the problem for production levels.

1

Introduction

Because of users' taste variety, FMS (Flexible Manufacturing System) and FTL (Flexible Transfer Line) that a single production line manufactures a variety of products have been developing and working[1]. In the production line, it is important to decide which parts are input into the production line in order to fit the timing of users' needs and to increase productivity. Currently, the change from batch production to one-by-one production that inputs each parts into a production line is seen. For example, there are the one-by-one input method according to production ratio and the method with the constraints of appearance ratios such as “Assembly interval of a sun-roof car is every ten” which is from engineers' experience. Although these methods realize one-by-one production, they include the problem that they don't search better solutions. This paper is focused on the problem for one-by-one parts input into FTL and describes the research of a hybrid intelligent off-line production simulator connected Genetic Algorithm (GA) system in order to search a better solution. This paper also describes the methods to input new individuals and to keep the diversity in GA

2

One-by-One Production System

FTL that a single production line manufactures varieties of parts according to a decided production ratio is expressed as an automatic production line including some V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 542-548, 2003.  Springer-Verlag Berlin Heidelberg 2003

Hybrid Intelligent Production Simulator by GA and Its Individual Expression

543

machine tools and some automatic conveyers connected with machine tools. One-byone production means the production style to input each variety of parts one-by-one under the constraint to keep the production ratio. In order to satisfy the constraint, it is necessary for the production style to have the production equality not to arise the input number difference between each variety of parts in a limited time. The methods of the production levels have been developed[2]. In the methods, however, the problem that the methods ignore other many feasible solutions for input sequence is remaining.

3

Hybrid Intelligent Simulator and Its Recurring Individual

3.1

Recurring Individual

In order to solve the problem that the conventional production equality methods have, this research takes a wide solution search space by connecting a production simulator with GA system or hybrid intelligent production simulator. New individual expression for the sake of keeping the production ratio of the simulator is proposed in this paper. The individuals generated in the algorithm of the production simulator correspond to input information of a production simulator. For example, in a case where the variety of parts are A, B, and C, the sequence A,B,C,A,B,C,… or A,A,C,B,A,C,B,… corresponds to the individual. The conventional method to generate an individual is randomly to select a gene and to assign the gene into a locus[3]. Because one-by-one production has the constraint to keep the goal production ratio, the input parts ratio must not be ignored in generating individuals. In order to give the equality to a GA problem, it is important that the parts percentage assigned as the gene expressing input sequence is not much different from the goal production ratio. The conventional method to generate individuals does not consider the ratio of each gene in an individual. There is another problem. If the simulation time is tremendous, input parts number also becomes tremendous. If the conventional method of an individual expression that is a sort of a belt-type chromosome[4], the individual length will become thousands or millions. In order to solve the problems, because of using the characteristic of the production equality, the new individual expression to keep a gene ratio and to generate a short individual length is used. If a variety of product is uniform during a limited time, it is regarded that the production equality can be performing during the time. If a constant individual length is programmed and the equality is performing between individual length, the individual is regarded as a constant percentage of individual length. The production equality is realized as the following algorithm. [Primitive Individuals Generative Algorithm] STEP1: Based on the gene ratio for v kinds of genes, r(1) : r(2) : … : r(v), the possibility, P(v), to assign on a locus for each of genes is calculated as the following ;

544

Hidehiko Yamamoto and Etsuo Marui r (v)

P(V) = v

…

(1)

∑ r (v ) v =1

STEP2: By using the assign possibility P(v), select a gene from among v kinds of genes with roulette strategy. The selected gene is expressed as “ i ”. STEP3: Randomly select an integer constant “a”, and decide the value q(v) corresponding to a gene quantity in an individual and the individual length δ with the following equations; q(v) = a × r ( v ) …

v

δ = ∑ q (v ) …

(2)

v =1

(3)

STEP4: Regard the cumulative value of assigned genes as Sum(v) and give the primitive value the following; Sum(v) = 0…

(4)

STEP5: Carry out the following rule to the accumulative value Sum(i) for gene “ i “ and this gene quantity q(i); [if] Sum ( i ) ≠ q ( i ) [then] Continue to STEP6 [else] Regard the gene i assignment as bad and go to STEP9 in order to proceed another gene assignment. STEP6: Judge the gene “i” as the assignment gene and assign the gene to the vacant left locus. STEP7: Renew the accumulative value of gene “i” with the following equations; newSum(i)

= Sum(i) + 1 …

(5)

Sum(i) = newSum(i) …

(6)

STEP8: By comparing an individual length δ with the current accumulative values Sum(v) for each gene, carry out this algorithm's finish judgment with the following rule. v

[if] δ =

∑ Sum( v ) v =1

[then] Regard acquired chromosome as an individual and finish this algorithm. [else] Continue to STEP9 STEP9: By using a roulette strategy with the assign probability P, select a gene from among v kinds of genes, regard the gene as “i” and go back to STEP5.

Hybrid Intelligent Production Simulator by GA and Its Individual Expression

545

Fig. 1. Recurring individual example

In this way, it is possible for the algorithm not to change the containing ratio and to generate an adequate length of an individual. The individual generated in the algorithm is expressed as a ring-type individual shown in Figure 1. The ring-type individual is considered as the first cycle that regards the locus to which the gene selected in STEP2 of the algorithm is assigned as a tentative locus number “1“ and the tentative locus number “δ” which is the last result by continuing to add 1 to the right direction of the ring. The tentative locus number 1 is regarded as the tentative locus number “δ+ 1” corresponding to the start of the second cycle. In the same manner, by recurring the above operation, a single individual is expressed. This individual is called recurring individual. 3.2

Production Simulator and New Individuals Inputs

The hybrid simulator's relation between production simulator and GA system is connected by the idea that GA system generates one of the input information to operate a production simulator[5]. The relation is realized by the following algorithm. [Algorithm Connected Production Simulator and GA] STEP1: Generate a population that have n pieces of individuals corresponding to randomly selected genes. STEP2: Carry out the production simulations with input information corresponding to each individual and calculate each individual's fitness by using the simulation results. STEP3: Carry out the elitist strategy to send e pieces individuals that have high ranking fitness to the next generation. STEP4: Select a pair of individuals by a probability based on fitness and give them crossover operation. STEP5: Select an individual with the mutation probability (Pm) and the selected individual participates in mutation operation. STEP6: If the individuals' number generated in STEP4 and STEP5 are equal to (n-e), continue to STEP7. If not, return to STEP4.

546

Hidehiko Yamamoto and Etsuo Marui

STEP7: Regard n pieces of individuals (sum up e acquired in STEP3 and (n-e) acquired in STEP6) as the next generation's population and finish one cycle of GA operation. STEP8: If the finish condition is not satisfied, some (f) new individuals are generated, f individuals whose fitness corresponds to smallest in the n pieces individuals of a population set are deleted, put the new f pieces individuals into the population set and return to STEP2. If satisfied, the algorithm is finished. In STEP 8 of the above algorithm, new individuals are sometimes input into the population and, because of it, the diversity can be kept.

4

Application Examples

4.1

Simulations for Productivity

The developed algorithm is applied for FTL example. The FTL has 8 stations ( st1, st2, …… , st8 ) assembly line and assembles 10 products. Products are input into the station st1 one-by-one. Assembling times for each variety of products in each station are not same. The goal cycle time of this line is 12 seconds. The production ratio for 10 products is P1 : P2 : …… : P10 = 9 : 6 : 7 : 6 : 8 : 7 : 3 : 2 : 4 : 1 . FTL has glitches and they randomly happen every 120 seconds. Their fix time is constant, 20 seconds. The conditions of GA system are the following; population size = 50, mutation probability = 1%, elite individual = 2, integer constant (a) = 1, individual length (δ) = 53, optional constant (ε) = 865, optional integers S1 = 5 and S2 = 5. As the finishing condition of STEP8 in Section3 algorithm, the time when generations = 60 is adopted.

Fig. 2. Fitness curves

Hybrid Intelligent Production Simulator by GA and Its Individual Expression

547

Several simulations by changing a series of a random number were carried out. Figure 2 shows one of the results. In the figure, a thin fine curve shows an average fitness among 50 individuals and a bold fine curve shows a maximum fitness among the 50. Fitness converge appears from about generation 35. This kind of results is shown in other examples of a series of a random number. The resulted production ratio was P1 : P2 : …… : P10 = 8.505 : 5.902 : 6.854 : 5.902 : 7.829 : 6.854 : 2.927 : 1.951 : 3.902 : 1 which is very similar to a goal production ratio. When the conventional belt-type chromosome was used, the program was stopped because the crossover to satisfy the production ratio was not carried out. 4.2

Simulations for Diversity

In general, GA avoids to fall in the local solutions with expanding the solutions space by mutations. The research carries out the simulations by the production simulator connected with GA. That means not only the calculations of many evolutions of GA but the calculations of production simulations are needed. Especially, production simulations for each individual need much time if we adopt a big population size, tremendous amount of a whole simulation time are needed. It is necessary to adopt the small population size and we adopt 50 individuals in a generation. If the population size is small, it occurs that the diversity disappears. In order to solve the problem, we adopt the method that new individuals are sometimes input into the population set. Every 5 generation, 10 individuals are input as new individuals to keep the diversity. Figure 2 also shows the fitness curves of the simulations results. The bold dotted curve shows the maximum fitness without diversity idea. Around the generation 38, the curve is saturated. Checking up the individuals among a population, it is found that each sequence of the chromosome is very close, the diversity disappears and the simulations go straight to the one local solution. The bold solid curves show the maximum fitness with diversity idea or to input new individuals. The thin solid curve shows the average fitness curve. It can be seen that the average fitness curve sometimes drops because of inputting 5 new individuals every 5 generation. Because of it, the solution space is always kept wide and the best fitness curve (bold solid curve) saturated at the higher position.

5

Conclusions

This paper describes the development of a hybrid intelligent off-line production simulator connected GA system in order to realize one-by-one production for a variety of products. The production simulator includes recurring individuals, their crossover operations and mutation operations not to change the results of a production ratio for each variety of products. The developed production simulator was applied for a FTL model that assembles a variety of products. As a result, acquired production ratio was very similar to a goal production ratio. The developed production simulator can be used in starting a production plan of FTL that manufactures a variety of products. Because of inputting new individuals to keep the diversity, it is also ascertained that the better solutions are acquired.

548

Hidehiko Yamamoto and Etsuo Marui

References [1] [2] [3] [4] [5]

Monden, Y.,Toyota Production System, Institute of Industrial Engineers, (1983), Atlanta, GA. Hitomi, K, Production Management, Corona Pub. (in Japanese). Goldberg, Genetic Algorithm in Search, Optimization and Machine Learning, (1989), Addison-Wesley. Holland, Adaption in Natural and Artificial Systems, (1975)Univ. Michigan Press. Yamamoto, H., Simulator of Flexible Transfer Line including Genetic Algorithm and its Applications to Buffer Capacity Decision, Proc. of the 3rd IFAC Workshop on Intelligent Manufacturing Systems, (1995), pp127-132.

On the Design of an Artificial Life Simulator Dara Curran and Colm O’Riordan Dept. of Information Technology National University of Ireland, Galway, Ireland Abstract. This paper describes the design of an artificial life simulator. The simulator uses a genetic algorithm to evolve a population of neural networks to solve a presented set of problems. The simulator has been designed to facilitate experimentation in combining different forms of learning (evolutionary algorithms and neural networks). We present results obtained in simulations examining the effect of individual life-time learning on the population’s performance as a whole.

1

Introduction

Genetic algorithms have long been used as an efficient approach to solution optimisation [1]. Based on a Darwinian evolutionary scheme, potential solutions are mapped to genetic codes which, in the case of the canonical genetic algorithm, are represented by bit patterns. Each of these solutions is tested for validity and a portion are selected to be combined to create the next generation of solutions. Using this mechanism in an iterative manner, the approach has been shown to solve a variety of problems [2]. Artificial neural networks are a method of machine learning based on the biological structure of the nervous systems of living organisms. A neural network is composed of nodes and interconnecting links with associated adjustable weights which are modified to alter a network’s response to outside stimuli. Through a process of training, a neural network’s error can be iteratively reduced to improve the network’s accuracy. The focus of this paper is on the design and development of an artificial life simulator which combines both genetic algorithm and neural network techniques. An initial set of experiments is also presented, which examines the relationship between life-time learning and increased population fitness. The next section discusses in more detail some limitations of genetic algorithms and neural networks when used in isolation and presents some of the successful work which has used a combination of the two approaches. Section 3 presents the simulator’s architecture, including the encoding mechanism employed and Section 4 outlines the initial experiments undertaken with the simulator.

2

Related Work

While the individual use of genetic algorithms and neural networks has been shown to be successful in the past, there are limitations associated with both V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 549–555, 2003. c Springer-Verlag Berlin Heidelberg 2003


550

Dara Curran and Colm O’Riordan

Fig. 1. Simulator Architecture approaches. The learning algorithm employed by neural network implementations is frequently based on back propagation or a similar gradient descent technique employed to alter weighting values. These algorithms are liable to become trapped in local maxima and, in addition, it is difficult to foresee an appropriate neural network architecture to solve a given problem [3], [4]. Genetic algorithms may also become trapped in local maxima and, furthermore, they require that potential solutions to a given problem be mapped to genetic codes. Not all problems are readily converted to such a scheme [1]. The combination of neural networks and genetic algorithms originally stemmed from the desire to generate neural network architectures in an automated fashion using genetic algorithms [5], [6]. The advantage of this approach is that neural networks can be selected according to a variety of criteria such as number of nodes, links and overall accuracy. The neural network component, on the other hand, provides computational functionality at an individual level within the genetic algorithm’s population. The combination of genetic algorithms and neural networks has since been proven successful in a variety of problem domains ranging from the study of language evolution [7] to games [8].

3

Simulator Architecture

The architecture of the simulator is based on a hierarchical model (Fig. 1). Data propagates from the simulator’s interface down to the simulator’s lowest level. The neural network and genetic algorithm layers generate results which are then fed back up to the simulator’s highest level. 3.1

Command Interpreter Layer

The command interpreter is used to receive input from users in order to set all variables used by the simulator. The interpreter supports the use of scripts, allowing users to store parameter information for any number of experiments.

On the Design of an Artificial Life Simulator

3.2

551

Neural Network Layer

The neural network layer generates a number of networks using variables set by the command interpreter and initialises these in a random fashion. Once initialisation is complete, the network layer contains a number of neural networks ready to be trained or tested. Training. Several algorithms exist to alter the network’s response to input so as to arrive at the correct output. In this system, the back propagation algorithm is used. Error reduction is achieved by altering the value of the weights associated with links connecting the nodes of the network. Each exposure to input and subsequent weight altering is known as a training cycle. Testing. Testing allows the simulator to ascertain how well each network solves a given problem. The output for each network is used by the selection process in the genetic algorithm layer. 3.3

Encoding and Decoding Layers

To perform genetic algorithm tasks, the neural network structures must be converted into gene codes upon which the genetic algorithm will perform its operations. This conversion is carried out by the encoding layer. Once the genetic algorithm has generated the next generation, the decoding layer converts each gene code back to a neural network structure to be trained and tested. The encoding and decoding layers follow the scheme outlined in Section 3.5. 3.4

Genetic Algorithm Layer

The genetic algorithm layer is responsible for the creation of successive generations and employs three operators: selection, crossover and mutation. Selection. The selection process employed uses linear based fitness ranking to assign scores to each individual in the population and roulette wheel selection to generate the intermediate population. Crossover. As a result of the chosen encoding scheme, crossover does not operate at the bit level as this could result in the generation of invalid gene codes. Therefore, the crossover points are restricted to specific intervals—only genes indicating the end of a complete node or link value can be used as a crossover point. Two-point crossover is employed in this implementation. Once crossover points are selected, the gene portions are swapped. The connections within each portion remain intact, but it is necessary to adjust the connections on either side of the portion to successfully integrate it into the existing gene code. This

552

Dara Curran and Colm O’Riordan

is achieved by using node labels for each node in the network. These labels are used to identify individual nodes and to indicate the location of interconnections. Once the portion is inserted, all interconnecting links within the whole gene code are examined. If any links are now pointing to non-existing nodes, they are modified to point to the nearest labelled node. Mutation. The mutation operator introduces additional noise into the genetic algorithm process thereby allowing potentially useful and unexplored regions of the problem space to be probed. The mutation operator usually functions by making alterations on the gene code itself, most typically by altering specific values randomly selected from the entire gene code. In this implementation, weight mutation is employed. The operator takes a weight value and modifies it according to a random percentage in the range -200% to +200%. 3.5

Encoding and Decoding Schemes

Before the encoding and decoding layers can perform their respective tasks, it is necessary to arrive at a suitable encoding scheme. Many schemes were considered in preparation of these experiments. These included Connectionist Encoding [6], Node Based Encoding [9], Graph Based Encoding [10], Layer Based Encoding [11], Marker Based Encoding [8], Matrix Re-writing [12],[13], Cellular Encoding [14], Weight-based encoding [3],[4] and Architecture encoding [15]. In choosing a scheme, attention was paid to flexibiliy, efficiency and scalability. The scheme chosen is based on Marker Based Encoding which allows any number of nodes and interconnecting links for each network giving a large number of possible neural network permutations. Marker based encoding represents neural network elements (nodes and links) in a binary string. Each element is separated by a marker to allow the decoding mechanism to distinguish between the different types of element and therefore deduce interconnections[12], [13]. In this implementation, a marker is given for every node in a network. Following the node marker, the node’s details are stored in sequential order on the bit string. This includes the node’s label and its threshold value. Immediately following the node’s details, is another marker which indicates the start of one or more node-weight pairs. Each of these pairs indicates a back connection from the node to other nodes in the network along with the connection’s weight value. Once the last connection has been encoded, the scheme places an end marker to indicate the end of the node’s encoding. The scheme has several advantages over others: - Nodes can be encoded in any particular order, as their role within the network is determined by their interconnecting links. - The network structures may grow without restriction—any number of nodes can be encoded along with their interconnections.

On the Design of an Artificial Life Simulator

553

- Links between nodes can cross layer boundaries. For instance, a node in the input layer may link directly to a node in the output layer, even if there are many layers between the two. - The system encodes individual weighting values as real numbers, which eliminates the ‘flattening’ of the learned weighting values which can occur when real number values are forced into fixed bit-size number values. The decoding mechanism must take the gene codes and generate neural network data structures ready to be trained or tested. Any decoding mechanism employed must be robust and tolerate imperfect gene codes. A number of anomalies may occur following crossover: - Data may occasionally appear between an end marker or a start marker. In such a circumstance the decoder ignores the data as it is no longer retrievable. - It is possible that extra start or end markers may be present within a node definition. In such a case, two choices are possible: either the new marker and its contents are ignored, or the previous section is ignored and the new one is taken as valid. The current implementation follows the latter approach. - A start marker may have no corresponding end marker or vice-versa. In such a situation, the decoder ignores the entire section of the gene code.

4

Experiments

The problem set employed for these experiments was 5-bit parity. Each network was exposed to 5-bit patterns and trained to determine the parity of each pattern. The number of training iterations was varied from 0, 10 and 100 iterations. The crossover rate was set to 75% and the mutation rate to 2%. Three experiments were carried out in total with 500 networks in each generation for 600 generations. The general aim of these experiments was twofold: to demonstrate the validity of the simulator and to ascertain how much the training or learning process affects each population’s fitness. The experimental results are shown in figure Fig. 2. No Training. When the genetic algorithm is used in isolation without the help of the learning process, the population’s fitness shows very little improvement in the first 200 generations. There is then a slight increase in fitness leading to further stagnation at around the 0.3 fitness level. The genetic algorithm alone is unable to generate a successful population for this problem set. 10 Training Iterations. The addition of training shows that even a modest increase in the population’s individual learning capability, enables the simulator to achieve very high levels of fitness. The fitness level ascends steadily to 0.85 before leveling out at nearly 0.9. At this level of fitness most individuals in the population are capable of solving all 32 solutions in the 5-bit parity problem.

554

Dara Curran and Colm O’Riordan 1

0.9

0.8

Fitness

0.7

0.6

0.5

0.4

0.3 No Training Training x 10 Training x 100 0.2 0

100

200

300 Generations

400

500

600

Fig. 2. Population Fitness For 0-100 Training cycles

100 Training Iterations. Once the networks receive more training, the advantages of the training process become obvious. The population’s fitness increases along a steep curve before jumping 0.3 points in 100 generations. The curve then levels out at around 0.95, the highest level of fitness attained in this experiment set.

5

Conclusion

The results achieved with the simulator indicates that population learning alone is not capable of solving problems of a certain difficulty. Once lifetime learning is introduced, the training process guides the population towards very high levels of fitness. The fact that the population is capable of achieving such high levels from little training (in the case of the 10 Training iterations experiment) shows that this approach should be capable of solving more complex problems.

6

Acknowledgements

This research is funded by the Irish Research Council for Science, Engineering and Technology.

References [1] J. H. Holland. Adaptation in Natural and Artificial Systems. Ann Arbor MI: The University of Michigan Press, 1975. 549, 550

On the Design of an Artificial Life Simulator

555

[2] D. E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Reading, MA, Addison-Wesley, 1989. 549 [3] R. S. Sutton. Two problems with backpropagation and other steepest-descent learning procedures for networks. In Proc. of 8th Annual Conf. of the Cognitive Science Society, pages 823–831, 1986. 550, 552 [4] J. F. Kolen and J. B. Pollack. Back propagation is sensitive to initial conditions. In Richard P. Lippmann, John E. Moody, and David S. Touretzky, editors, Advances in Neural Information Processing Systems, volume 3, pages 860–867. Morgan Kaufmann Publishers, Inc., 1991. 550, 552 [5] P. J. Angeline, G. M. Saunders, and J. P. Pollack. An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks, 5(1):54–65, January 1994. 550 [6] R. K. Belew, J. McInerney, and N. N. Schraudolph. Evolving networks: Using the genetic algorithm with connectionist learning. In Christopher G. Langton, Charles Taylor, J. Doyne Farmer, and Steen Rasmussen, editors, Artificial Life II, pages 511–547. Addison-Wesley, Redwood City, CA, 1992. 550, 552 [7] B. MacLennan. Synthetic ethology: An approach to the study of communication. In Artificial Life II: The Second Workshop on the Synthesis and Simulation of Living Systems, Santa Fe Institute Studies in the Sciences of Complexity, pages 631–635, 1992. 550 [8] D. Moriarty and R. Miikkulainen. Discovering complex othello strategies through evolutionary neural networks. Connection Science, 7(3–4):195–209, 1995. 550, 552 [9] D. W. White. GANNet: A genetic algorithm for searching topology and weight spaces in neural network design. PhD thesis, University of Maryland College Park, 1994. 552 [10] J. C. F. Pujol and R. Poli. Efficient evolution of asymmetric recurrent neural networks using a two-dimensional representation. In Proceedings of the First European Workshop on Genetic Programming (EUROGP),, pages 130–141, 1998. 552 [11] M. Mandischer. Representation and evolution of neural networks. In R. F. Albrecht, C. R. Reeves, and N. C. Steele, editors, Artificial Neural Nets and Genetic Algorithms Proceedings of the International Conference at Innsbruck, Austria, pages 643–649. Springer, Wien and New York, 1993. 552 [12] H. Kitano. Designing neural networks using genetic algorithm with graph generation system. In Complex Systems, 4, 461-476, 1990. 552 [13] P. M. Todd G. F. Miller and S. U. Hedge. Designing neural networks using genetic algorithms. In Proceedings of the Third International Conference onGenetic Algorithms and Their Applications, pages 379–384, 1989. 552 [14] F. Gruau. Neural Network Synthesis using Cellular Encoding and the Genetic Algorithm. PhD thesis, Centre d’etude nucleaire de Grenoble, Ecole Normale Superieure de Lyon, France, 1994. 552 [15] J. R. Koza and J. P. Rice. Genetic generation of both the weights and architecture for a neural network. In International Joint Conference on Neural Networks, IJCNN-91, volume II, pages 397–404, Washington State Convention and Trade Center, Seattle, WA, USA, 8-12 1991. IEEE Computer Society Press. 552

3-Dimensional Object Recognition by Evolutional RBF Network Hideki Matsuda1, Yasue Mitsukura2, Minoru Fukumi2, and Norio Akamatsu2 1 Graduate

School of Engineering, University of Tokushima, 2-1 Minami-josanjima, Tokushima, Japan [email protected] 2 Faculty of Engineering, University of Tokushima, 2-1 Minami-josanjima, Tokushima, Japan {mitsu,fukumi,akamatsu}@is.tokushima-u.ac.jp

Abstract. This paper tries to recognize 3-dimensional objects by using an evolutional RBF network. Our proposed RBF network has the structure of preparing four RBFs for each hidden layer unit, selecting based on the Euclid distance between an input image and RBF. This structure can be invariant to 2- dimensional rotation by 90 degree. The other rotational invariance can be achieved by the RBF network. In hidden layer units, the number of RBFs, form, and arrangement are determined using real-coded GA. Computer simulations show object recognition can be done using such a method.

1

Introduction

By development of computer technology, various robots, such as a humanoid robot and a pet type robot, have been developed. There is a robot with a visual system. Although the visual system of the robot for recognizing an object has not reached to pattern recognition with the same level as man yet. Man can recognize it as seeing from a fixed angle to a known object in the case of what the object is from every angle. In the case of a robot, it cannot recognize except a fixed angle. Conquest of this fault is indispensable to further improvement in a visual system. In order to conquer such a fault in recent years, research of 3-dimensional object recognition has been done. A technique of recognizing an object is achieved by restoration of the 3-dimensional object form by the multi-viewpoint picture in one of them. One object restoration takes huge calculation time, and the many training data is mostly needed for this technique. We think that it is unsuitable for object recognition to a picture photographed from a strange angle. The technique of not building 3dimensional object form but recognizing an object by the different method is desirable. Moreover, there is also report whether it can recognize by a technique of building relation of each training picture, by using a neural network, which is geometric structure. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 556-562, 2003.  Springer-Verlag Berlin Heidelberg 2003

3-Dimensional Object Recognition by Evolutional RBF Network

557

By this research from the above thing, we propose a technique of the 3dimensional object recognition using the Radial Basis Function (RBF) network that introduced rotation invariant structure. It is known that RBF network has the advantage that an input image can be localized, and the performance of interpolation is excellent. Rotation invariant structure prepares four RBF for each hidden layer unit of RBF network, inputs them, calculating the Euclid distance between an input image and the center position of RBF, and uses the RBF unit with the nearest Euclid distance. With such a structure, rotated objects can also be trained at the same time. Therefore, it is not necessary to newly add another hidden layer unit and curtailment of the number of hidden layer units is considered to influence improvement in the speed of training. Calculation time and capability are influenced by arrangement of RBFs, form, and input area. It can be trained completely, if the number of training data and the same number of RBF are prepared. But when the number of training data becomes huge, the number of RBFs increases, calculation time becomes long, and it becomes less practical. Therefore in this paper Genetic Algorithms (GA) tries to find the number of RBFs, form, and an arrangement.

2

RBF Network

An output function in RBF network is given by

 ( x − a j )T ( x − a j )  y j = exp − , 2σ 2j   T

z k = wk y

(1) (2)

where a variable x is an input vector in an input layer, a value zk(k=1,2, …, n) is an output value of each output layer unit, and a vector aj(j=1,2, …, m) is the center position information of RBF. A value σ j (j=1,2, …, m) is the range of RBF. A vector wk(k=1,2, …, n) is weights of combination between each hidden layer and output layer. In this paper, a Gaussian Function is used as RBF.

3

Recognition System

3.1

Network Form

The system proposed in this paper has a network structure as shown in Fig.1. The brightness value in an input image is an input signal to an input layer. The output value of each output layer unit shows the possibility of whether to be each object. Four RBFs are prepared for each hidden layer unit, and as shown in Fig.2, which has position information. Each position information is constituted on the basis of the position information expressed with one arrangement, and is made by rotating by 0 degree, 90 degrees, 180 degrees, 270 degrees. For every input, each hidden layer unit

558

Hideki Matsuda et al.

finds the Euclid distance between the center position of four RBFs, and an input image, and uses only RBF with the nearest distance. With such a structure, the picture of Fig.3 (b), (c), (d) can also be trained by training the picture of Fig.3 (a) at the same time. Because the output layer of the RBF network is a linear filter model, it makes correction for the error of each unit using the Normalized LMS (3).

Fig. 1. Network structure

The normalized LMS algorithm is given by

wk +1 = wk + µ

ε k yk ,

|| y k ||

(3)

where µ is a step size, ε k (k=1,2, …, n) is the error between the output signal of each output layer unit and a teacher signal.

Fig. 2. The center position of RBF structure

3-Dimensional Object Recognition by Evolutional RBF Network

559

Fig. 3. The picture of input image

3.2

Input Data

A single viewpoint picture which is taken from only one direction is used as an input image to a subject object. As photography conditions, a background is made uniform and fixes the distance between a subject object and a camera in all subject objects. We take the photography from one direction to the subject object. A viewpoint is in a state as it is, and performs fixed angle rotation for a subject object in the position and we take it. Such an operation is repeated. These pictures are changed into a gray scale picture. The objective center of gravity is moved to the center of a picture in all pictures. Let brightness for every coordinates of a picture be an input value. 3.3

GA

As above-mentioned, the position and form of RBFs, and the number of hidden layer units, are important in a RBF network. These figures are determined using real-coded GA. The chromosome structure is shown in Fig.4.

Fig. 4. The chromosome structure

where a value N is the number of hidden layer units used in the RBF network. A vector a is the center position information of RBF. A value σ is the range of RBF.The following operations are performed using such a chromosome structure. • • • • • •

STEP 1: Individuals are generated using a random numbers. STEP 2: An each individual is applied to a RBF network and training cycles is performed based on training data. After training, let the acquired least square error be the degree of adaptation of an each individual. STEP 3: One individual with the best degree of adaptation is selected out of the individuals, and it is saved as the elite. STEP 4: GA search stops if the degree of adaptation of the elite's individual exceeds a fixed value. If conditions are not fulfilled, proceed to STEP 5. STEP 5: Individual generation of the next generation, crossover, and operation of mutation are performed. STEP 6: When the number of generations does not exceed a fixed value, it proceeds to STEP1, with the elite saved.

560

Hideki Matsuda et al.

i ) KENDAMA

ii ) stapler

iii ) cup

Fig. 5. The subject objects

a)

b)

c)

d)

Fig. 6. The examples of teacher image

3.4

e)

f)

g)

Fig. 7. The examples of test image

Computer Simulations

Experiments using the RBF network that introduced the rotation invariant structure are carried out. As a comparison experiment, the experiment using the usual RBF network that has one RBF in the each hidden layer unit, is conducted. As a subject object, three objects as shown in Fig.5 are used for the experiment. Pictures of the object rotated by 45 degree as shown in Fig.6, are used in training. Pictures of the object rotated by 22.5 degree as shown in Fig.7, are used for testing. For study of the proposed network, 24 teacher data is used. For the usual RBF network, the pictures which rotated dimensional every 90 degree are added to teacher data. Therefore, it learns using 96 teacher data. Both networks use the picture of 24 sheets as test data. Picture size is set to 32 × 32 pixel. The parameter of a setup of both networks is shown in Table 1. By GA, the number of the hidden layer units determines both networks. The number of the hidden layer units, form and center position of RBF are determined using GA. Table 1. The parameter of a setup of networks

Input layer unit Hidden layer unit Output layer unit Times of the maximum study Step size of study

The proposed network 1024 24 3 1000 0.1

The usual RBF network 1024 96 3 1000 0.1

3-Dimensional Object Recognition by Evolutional RBF Network

3.5

561

Simulation Results

Fig. 8. The transition of the proposed network

Fig. 9. The transition of RBF network

The result experimental, in processing by GA, transition of the degree of adaptation became to 500 generations as it is shown in Fig. 8. 9. The proposed technique was able to find an optimal parameter more quickly than technique that used the usual RBF network. The rate of recognition and the number of hidden layer units became as it is shown in Table 2. Moreover, the number of hidden layer units could also be learned with the number fewer than it. In accuracy of recognition, both of techniques recognized the object correctly about 80 percent on average. As a result, the present approach requires a fewer number of RBF units than a conventional RBF network.

4

Conclusions

In this paper, an object which rotated by the fixed angle, has been recognized by training based on the Euclid distance using a RBF network. Moreover, the present method can carry out reduce the number of hidden layer units and high-speed training rather than the usual RBF network structure was brought by introducing rotation invariant structure into RBF network. At this time, this structure accepts only rotation of a fixed direction which corresponds to the object. Therefore, it is necessary to also conduct the experiment of rotation of any different direction. Moreover, since GA determines values about RBF, there is a problem that long training time is needed. Correspondence of all rotations of an object and our high-speed search of values about RBF are done as future work. Table 2. The result of experiments of each technique The proposed network The usual RBF network

Correct answer 21/24 20/24

Hidden layer units 12 79

562

Hideki Matsuda et al.

References [1] [2] [3] [4] [5] [6]

S. Itoh, J. Murata and K. Hirasawa, "Size-Reducing RBF network and Its Application to Control Systems", IEEJ Trans. EIS, vol. 123, no. 2, pp.338-344, 2003, in Japanese. R. Katayama, Y. Kajitani, K. Kuwata, and Y. Nishida, "Self Generating Radial Basis Function as Neuro-Fuzzy Model and its Application to Nonlinear Prediction of Chaotic Time Series", FUZZ-IEEE, no. 2, pp.407-414, 1993. N. B. Karayiannis and G. W. Mi, "Growing Radial Basis Neural Network, Merging Supervised and Unsupervised Learning with Network Growth Techniques", IEEE Trans. Neural Networks, 8, 6, pp.1492-1506, 1997. S. A. Billings and G. Zheng, "Radial Basis Function Network Configuration Using Genetic Algorithms", Neural Networks, 8, pp.877-890. M. Maruyama, "Networks for Learning Based on Radial Basis Functions", J. ISCIE, vol. 36, no. 5, pp.322-329, 1992, in Japanese. N. Sato and M. Hagiwara, "Parallel-Hierachical Neural Network for 3D Object Recognition", IEICE Transactions, vol. J86- D-II, no 4, pp.553-562, 2003.

Soft Computing-Based Design and Control for Vehicle Health Monitoring Preeti Bajaj1 and Avinash Keskar2 1

Electronics Department G. H. Raisoni College of Engineering CRPF Gate 3, Digdoh Hills, Hingna Road, Nagpur, India- 440016 Phone (R: 91-712-2562782, M: 91-9822237613) [email protected] 2 Department of Electronics and Computer Sciences Visvesvarya National Institute of Technology Nagpur-440011, Phone (R: 91-712-2225060, M: 91-9823047697) [email protected]

Abstract. The study of Vehicle Health monitoring plays very important role in deciding the possibility of completing journey successfully. Already work has been done on Pre Trip plan assistance with vehicle Health Monitoring with simple and Hybrid Fuzzy controllers. The conventional Fuzzy logic controllers are knowledge-based systems, incorporating human knowledge into their knowledge base through fuzzy rules and fuzzy membership functions. They are limited in application, as its logic rules and membership functions have to be preset with expert knowledge and have a great influence over the performance of the fuzzy logic controller. In the present work, a Genetic Fuzzy system is proposed to promote the learning performance of logic rules. The resultant hybrid system seems to be highly adaptive and trained through a proper performance, hence is much more sophisticated and has a higher degree of adaptive parameters. The work proposes a benefit of Methodology by comparison of calculating safest distance by evaluating Vehicle Health Monitoring using Hybrid Fuzzy, and genetic fuzzy controller.

1

Introduction

In recent years, increased efforts have been centered on developing intelligent vehicles those can provide in vehicle information to the motorists. These include the development of the systems to increase comfort to the driver. In the field of transport studies, FLC can be applied to transportation planning including trip generation, distribution, model split, and route choice, Vehicle Health Monitoring and many others. Genetic programming has recently been demonstrated to be a vital approach to V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 563-569, 2003.  Springer-Verlag Berlin Heidelberg 2003

564

Preeti Bajaj and Avinash Keskar

learning fuzzy logic rules for transportation. The study of Vehicle Health monitoring plays very important role in deciding the possibility of completing journey successfully. Already work has been done on Pre Trip plan assistance with simple Fuzzy and Hybrid Fuzzy systems [1,2]. In the present work, a GA-Fuzzy system is proposed; to promote the learning performance of logic rules that can extract information from multiple fuzzy knowledge source and put into a single knowledge base. The work elaborates a fuzzy system, in which the membership function shapes, types and fuzzy rule set including number of rules inside it are evolved using a Genetic Algorithm. The genetic operators are adapted by a fuzzy system. The resultant hybrid system seems to be highly adaptive and trained through a proper performance, hence is much more sophisticated and has a higher degree of adaptive parameters. The benefits of the methodology are illustrated in the process of calculating safest Distance Traveled by a vehicle depending upon status of various inputs.

2

Vehicle Health and Vehicle Maintenance

The success of completing a desired journey depends on various factors, which can be categorized as journey classification, journey type, journey condition, vehicle condition, vehicle maintenance, vehicle characteristics and the proficiency of the driver. The vehicle health is the current condition of the vehicle based on sensor output about the Air pressure in tyres, Coolant level, Engine oil level, Break oil level, Battery condition, Fuel condition, Tyre age, Last servicing report, part replacement, age of the car etc. The detail history of servicing and life of different parts of vehicle is maintained as per the characteristics of the vehicle. Since all these factors are measured by sensors and vary in specifying the minimum requirement with the make, type of journey, load etc, Fuzzy controller is chosen to judge the Vehicle Health and to calculate the safest distance to be traveled by a vehicle with current sensors output. For simplicity the adaptive ness of the Controller is tested with four dominating inputs as coolant, engine oil, break oil and fuel.

3

Conventional Fuzzy Logic Controller

Like a typical PID system, a conventional Fuzzy system usually takes the form of an iteratively adjusting a model. In such a system, input values are normalized and converted to fuzzy representations, the model's rule base is executed to produce a consequent fuzzy region for each solution variable and the consequent regions are defuzzified to find expected value of each solution variable using centroid method. An open loop Fuzzy logic controller is designed. The Membership functions assigned to the inputs and outputs for Input Mamdani Fuzzy inference system is as shown in Table 1. The sample of input and output membership function for Hybrid fuzzy is as shown in Figure 1 & 2 respectively. These inputs are fuzzified and applied to the fuzzifier. The Inference engine fires appropriate rules and gives defuzzified output for safest distance to be traveled.

Soft Computing-Based Design and Control for Vehicle Health Monitoring

565

Table 1. Input & Output membership mapping in fuzzy

In/Out labels

Membership function and levels

Coolant

Trapezoidal, Dangerous, safe, almost full

0, 10

Engine oil

Triangular, Dangerous, safe, almost full

0, 10

Break oil

Triangular, Dangerous, safe, almost full

0,10

Fuel

Trapezoidal, Less, Moderate, Full

0, 35

Safest traveling distance -Km

Trapezoidal, Short, moderate, large

In range

0, 300km

Fig. 1. Hybrid Membership Function for Fuel

4

Design of Genetic Fuzzy Controller (GFLC)

In current problem, a genetic algorithm is applied to the Fuzzy controllers, which adapts the car simulator (real vehicle) and generates rule base of the fuzzy logic controller while maintaining membership function unchanged. The GFLC is implemented in Matlab using ‘Fuzzy' and ‘Geatbx' toolboxes. It has been implemented using the functions as given below. Gene length is taken as

566

Preeti Bajaj and Avinash Keskar

4*4*4*4=256. Assumed the constant mileage of the vehicle as 19.0 Km/lt. It took 41 generations to give best output. GA tries to minimize ObjVal in each iteration, and thus trying to minimize error between outputs of simulator and FLC [3-4]. Authors have used Pittsburgh approach for tuning the rule base [5] in which each Chromosome represents entire rule base or knowledge base.

Fig. 2. Membership function for Safe Traveling Distance

Car Simulator. Car simulator function simulates a vehicle on which Fuzzy logic controller is to be adapted. A sim_datagen.m is a function, which saves computation time for output of car simulator. It generates sim.mat that contains output for various inputs, and gives output for simulator without running carsim and hence saving time Initialize Geatbx Toolbox. A function geamain.m is the main driver for the GEA Toolbox where all the parameters are resolved and the appropriate functions are called. Promain.m is main program [6-7] and a front-end module used to start genetic optimization use to initialize various parameters migration rate, and crossover rate. Initialize Population. The process of initializing starts with random values in the population concerned. Chrome_encode.m is a program that encodes the chrome. It inputs rule set from input mamdani fuzzy inference system and converts rule set into 256 gene initial chromosome. Calculate Fitness for Each Chromosome in Population (fitness.m). This file implements fitness function. It is proportional to the performance measurement of the function being optimized. Usually the value can range from 0 to 1. The rules generated by genetics are evaluated with random inputs and is compared with output of car simulator. GEATBx tries to minimize the value of fitness function, objval. Objval = abs (distance recommended by car simulator distance recommended by fuzzy system). Reproduction. It comprises forming new population, usually with same number of chromosomes by selecting from number of current population on the basis of fitness values. The method used is a ‘Roulette wheel' procedure that assigns a portion in the wheel is proportional to the fitness value. The members in the new population will certainly be from having higher fitness value [8-9].

Soft Computing-Based Design and Control for Vehicle Health Monitoring

567

Crossover. It is the process of exchanging portions of the strings of the two ‘Parent' chromosomes. The probability of crossover is often in the range of 0.65-0.80. Mutation. It consists of changing an element's value at random, often with a constant probability for each element in the population. Rule Generation. A function Rulegen.m decodes chromosome to generate a rule set. It takes chromosome generated by genetic toolbox and outputs a new fuzzy system. It generates rule set from chromosome.

5

Results

The Input mamdani model is provided knowledge base of only 13 rules. Sample of the rules are: 1.

If (engine-Oil is mf1) then (safest-distance is mf1) (1)

2.

If (brake-oil is mf1) then (safest-distance is mf1) (1)

3.

If (Fuel is less) then (safest-distance is mf1) (1)

And the sample of 29 rules generated by adaptive GA-fuzzy controller is as under. 1.

If (Fuel is moderate) then (safest-distance is mf1) (1)

2.

If (Fuel is high) then (safest-distance is mf1) (1)

3.

If (brake-oil is mf1) and (Fuel is less) then (safest-distance is mf1) (1)

4.

If (brake-oil is mf1) and (Fuel is high) then (safest-distance is mf1) (1)

5.

If (engine-Oil is mf1) and (Fuel is less) then (safest-distance is mf1) (1)

6.

If (coolant is mf1) and (brake-oil is mf3) and (Fuel is high) then (safest-distance is mf1) (1)

7.

If (coolant is mf1) and (engine-Oil is mf1) and (brake-oil is mf2) then (safestdistance is mf1) (1)

8.

If (coolant is mf2) and (engine-Oil is mf3) and (brake-oil is mf2) and (Fuel is high) then (safest-distance is mf2) (1)

9.

If (coolant is mf2) and (engine-Oil is mf3) and (brake-oil is mf3) and (Fuel is high) then (safest-distance is mf3) (1)

10. If (coolant is mf3) and (Fuel is less) then (safest-distance is mf1) (1) 11. If (coolant is mf3) and (engine-Oil is mf2) and (brake-oil is mf3) and (Fuel is less) then (safest-distance is mf3) (1) The sample of adaption output stored in text file gives following information. Number of variables: 256 Boundaries of variables: 0 3 Evolutionary algorithm parameters: subpopulations =5

568

Preeti Bajaj and Avinash Keskar

Individuals=50 30 20 20 10 (at start per subpopulation) Termination 1:max. Generations =100; 4:running mean= 0; Variable format 2(integer values-phenotype == genotype) Output Results on screen every 1 generation Graphical display of results every 5 generations Method = 111111, Style = 614143 File name = Rule_Adapt.txt End of optimization: running mean (55 generations; 38.43 cpu minutes / 38.43 time minutes) Best Objective value: 3.1746 in Generation 41 Figure 3 shows performance characteristics of hybrid and Genetic fuzzy controller.

Fig. 3. Performance Characteristics for simulator, FLC, GFLC

6

Summary and Conclusion

This paper demonstrates an approach to Vehicle health monitoring. GA is applied to discover the fuzzy controllers capability of determining the safest distance allowed by the vehicle. The performance of the best evolved FLC was comparable to that of

Soft Computing-Based Design and Control for Vehicle Health Monitoring

569

conventional FLC and of simulator. GA evolved a good and reasonable set of rules for an FLC that demonstrated satisfactory responsiveness to various initial conditions while utilizing minimal human interface. The evolved FLC shown to be robust to permutations of sensor noise and change in membership functions. Such systems strike a balance between learned responsiveness and explicit human knowledge makes the system very robust, extensible and suitable for solving a variety of problems.

References [1] [2] [3] [4] [5] [6] [7] [8] [9]

Ms Bajaj, Dr. A.G.Keskar,” Smart Pre trip Plan assistance system using Vehicle health monitoring”, ITSC-02, fifth international conference on Intelligent transportation systems, IEEE, 3-6th sept 2002, Singapore Ms Bajaj and Dr. Keskar,” Smart Pre Trip Plan Assistance System With hybrid Fuzzy Algorithm”, Hong Kong Transportation Association Luis Magdalena, “Adapting the gain of FLC with Genetic Algorithm” ETS Ingenieros De Telecommunication, Universidad, Spain G P Bansal and et al, “Genetic algorithm based Fuzzy expert system”, paper no 224-B, IETE journal, pp 111-118, India Yu-Chiun and et al,” Genetic Fuzzy Controllers” ITSC-02, fifth international conference on Intelligent transportation systems, IEEE, 3-6th Sept 2002, Singapore www.geatbx.com www.mathtools.net/MATLAB/Genetic_algorithms www.genetic-programming.org www.geatbx.com/links/ea_java.html

An Adaptive Novelty Detection Approach to Low Level Analysis of Images Corrupted by Mixed Noise Alexander N. Dolia1, Martin Lages1, and Ata Kaban2 1 Department of Psychology, University of Glasgow, 58 Hillhead Street, Glasgow G12 8QB, United Kingdom {a.dolia,m.lages}@psy.gla.ac.uk 2 Computer Science Department, University of Birmingham, Edgbaston Birmingham B15 2TT, United Kingdom [email protected] http://www.cs.bham.ac.uk/~axk/

Abstract. We propose a new adaptive novelty detection based algorithm for the primary local recognition of images corrupted by multiplicative/additive and impulse noise. The purpose of primary local recognition or low level analysis such as segmentation, small object and outlier detection is to provide a representation which could be potentially used e.g. in context based classification or nonlinear denoising techniques. The method is based on the estimation of mixing parameters (priors) of probabilistic mixture models along a small sliding window. A novelty score is defined by the mixing parameters and this is utilized by the procedure for determining the corresponding class of image patch with the aid of a lookup f. Numerical simulations demonstrate that the proposed method is able to improve upon previously employed techniques for the same task. In addition, the computational demand required by the proposed method is clearly inferior to some of the recently applied techniques as expert systems or neural networks.

1

Introduction

A number of methods of novelty detection have been proposed using nonparametric [1,2], semi-parametric and parametric [2,3] statistical approaches, support vector techniques [4,5] and neural networks [6]. The main idea behind these approaches is to estimate either the unconditional probability density function or directly the support of the data distribution from a training set. Then based on this information it is tested to what extent the new data do fit to the model by calculating some measure or score of novelty. If the score exceeds

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 570-577, 2003.  Springer-Verlag Berlin Heidelberg 2003

An Adaptive Novelty Detection Approach to Low Level Analysis of Images

571

some previously set threshold then the corresponding test samples are considered as outliers or novelty [1-6]. Note however that in practice data could be inhomogeneous and so the location of the support of the data distribution may change. In these cases the support of the data is rather a local feature as the global support becomes meaningless. The approach proposed in this paper provides a solution by adapting the parameters of the model to small space-time domains. This is achieved by local analyses of the image with small sliding windows and by updating a parameter of the novelty model itself. We also show real-world examples where the proposed method is useful for low level analysis of images such as segmentation, small object and outlier detection. The paper is organized as follows. The image models are described in Section 2. A new adaptive novelty detection method is proposed in Section 3. Simulation results are presented in Section 4. And conclusions are provided in Section 5.

2

Models of Noisy Images

Two main noise models have been considered in the image processing literature: a) an image obtained by Ka-band side look aperture radar (SLAR) is corrupted by a mixture of Gaussian multiplicative noise and outliers [7]; b) an image is influenced by a mixture of Gaussian additive noise and outliers. The main difference between these two models is that in the first case the noise is multiplicative or data-dependent while in the second case the noise is additive or signal-independent [7,8]. Let index n denote both the index of the pixel under consideration and the location of the central pixel in the scanning window. With the probability Pim of the occurrences of outliers A n the model of SLAR image x n corrupted by outliers can be written as x n = A n and in the other cases x n is corrupted by signal dependent noise, x n = I n (1 + σξ n ) , [7,8,9] where I n ¸{µ 1 , µ 1 ,..., µ J } , I n denotes the value of the n-th pixel of the image without noise. Further, ξ n denotes standard Gaussian noise, σ 2 is the relative variance of the multiplicative noise with mean equal to 1, µ1 , µ 2 ,...µ J denote the different intensity levels, J is the number of different intensity levels in the noise-free image. The multiplicative noise can be easily transformed to additive noise using an appropriate homomorphic transformation, e.g., the natural logarithm [8,9]. Then the transformed image model can be written as y n = sI hn + σ s ξ n , where σ s2 = s 2 σ 2 , I hn

¸{µ1h , µ h2 ,..., µ hJ } and µ hj = ln µ j . Further, y n , I hn are the n-th pixels of

x n , I hn after homomorphic transform, respectively; s is a scaling parameter - in our

experiments we set s = 46 for our convenience (in this case x n , y n ¸[0,255] ). In the above approximation we have made use of the Taylor expansion of the natural loga-

572

Alexander N. Dolia et al. ‡

xk when x < 1 , by truncating it to the k k =1 linear term. Observe that after the homomorphic transform the probability density function p( y n ) of the image y n can be approximated by the mixture of Gaussians

rithm function i.e., ln (1 + x ) = ‡”(- 1) k +1

with

shared

variance

J

σ s2 ,

mixing

parameters

p( y n ) = ‡”π j N( y n , µ hj , σ S2 ) where 0 < π j ¡Ü1 and j=1

πj

and

means

µ hj ,

J

‡”π

j

= 1 [10]. Thus we now

j=1

have a model with 2J + 1 unknown parameters ( π j , µ hj and σ s2 ) which could be estimated using a gradient-like method or Expectation-Maximization (EM) algorithm [10]. However, this still does not provide any information about the local changes of intensity (what Marr has called as ‘primal sketch' [11]). The next section is concerned with developing a method for this purpose.

3

Locally Adaptive Novelty Detection Method

Our algorithm is based on the following intuitive idea. If an observer is trained to see just one homogeneous region at a time with different light intensity or texture then it is unusual for him to see: 1) two large homogeneous regions (edge) at a time; or 2) small objects or outliers with a homogeneous region in the background. Similar to [9,12,13], we consider six classes for primary local recognition: 1) homogeneous region (H); 2) edge neighborhood between large objects or two homogeneous regions (E); 3) neighborhood of a spike (outlier) - by this we mean that there is one or no more than 3 spikes elsewhere than in the central pixel (NS); 4) spike in the central pixel of the scanning window (S); 5) the central pixel belongs to a small sized object (a small sized object is characterized by compactness of pixels belonging to it as well as by homogeneity of the pixel values) (O); 6) neighborhood of a small sized object (NO). Because the scanning window should be small enough to ensure locality of the analysis (e.g., 5x5 pixels) and also for keeping our model as simple as possible [4] then it is reasonable to assume that there are not more than two Gaussian components appearing in the scanning window. Therefore, we will model the image fragment in each small scanning window as a mixture of two Gaussians with the mixing parameters π in and π nj ( π in + π nj = 1 , 0 ¡Üπ in , π nj ¡Ü 1 ) for the i-th and j-th component of the mixture distribution p( y n ) = π in N( y n , µ ih , σ s2 ) + π nj N( y n , µ hj , σ s2 ) , i

‚j , i, j ¸{1,2,..., J} .

(1)

The model (1) has only 5 free parameters but it is necessary to estimate these parameters for each location of the scanning window. Therefore, for the entire image we end up having as many models as pixels in the image. Because we employ small

An Adaptive Novelty Detection Approach to Low Level Analysis of Images

573

scanning windows and because the image is contaminated by outliers, it would be inefficient to apply a statistical estimation such as an EM algorithm, for example. Obviously, for i ‚j knowing one of mixing parameters we can find another one ( π in = 1 - π nj ) and a small reduction in the number of free parameters is possible. Note that there are three distinct possibilities: 1) π in < π nj ; 2) π in > π nj ; 3) π in = π nj . Therefore, we can find a Gaussian component that has a mixing parameter that is not less than the other one. All we need to do then is to estimate π nmax = max{π in , π nj } for each location of the scanning window. In order to estimate π nmax we will employ a well-known technique, the so-called 3sigma rule, which states that with probability 0.997 the random value y will lie within the interval [µ - 3σ s , µ + 3σ s ] if it belongs to the Gaussian distribution with mean µ and standard deviation σ s , P( | y - µ | ¡Ý 3σ s ) ¡Ü 0.003 [14]. We introduce a new notation here, let N µn denote the number of pixels of the scanning window in the neighbourhood of y n which are inside the interval [µ - 3σ s , µ + 3σ s ] . For grey-scaled images with 255 grey levels µ ¸[0,255] . In summary, the proposed algorithm will consist of two steps: 1) calculate π nmax where the estimate of π nmax is equal to the maximal value of N µn given the location of the scanning window and an estimate of σ s2 divided by the number of points in the scanning window L (e.g., if the size of the scanning window is 5x5 then L=25 and, π nmax = max N µn / 25 ); 2) based on the estimate of π nmax (or max N µn or novelty µ

score N score = 1 - π nmax ) and on the information about either the central pixel being inside of the selected interval [µ - 3σ s , µ + 3σ s ] corresponding to max N µn look up the class from the table (e.g., see Table 1). The method is locally adaptive because an estimate of the dominant mixing parameter of the mixture model π nmax is calculated for each position of the small scanning window but not for the entire image. For 5x5 scanning window when L = 25 , e.g., the first row of Table 1 corresponds to max N µn = 25 , the second (third) one relates to max N µn = {23,24} ( max N µn = 22 ). Similar tables were proposed for the classification of noise-free images in order to obtain a target image using a genetic algorithm to correct misclassifications (see Table 1,2 in [13]). Thus, if the image fragment in the scanning window is not novel, then we get a novelty score N score ¡Ö0 . If the image fragment is not just one homogeneous region or is unusual to the observer then N score > 0 . In practice N score is never exactly equal to one because even if pixels are uniformly distributed in [0,255] then max N µn is equal at least 1 ( 1 ¡Ümax N µn ¡ÜL and 0 ¡ÜN score ¡Ü1 - 1 / L ).

574

Alexander N. Dolia et al.

a)

b)

c)

d)

Fig. 1. Illustrations of primary image recognition for the artificial image: a) the artificial Kaband SLAR image; b) novelty score mapping; c) central pixel mapping; d) classification mapping

4

Experiments

In this section, we demonstrate the performance of the proposed algorithm for both an artificial image corrupted by Gaussian multiplicative noise (Fig.1,a) and a real Kaband SLAR image (see Fig.2,a). We start with a noise-free image containing small and large objects with intensity levels from the set {10,15,20,80,120,160} i.e., six different positive and negative contrasts and a background of 40. This image was then corrupted by multiplicative (data-dependent) noise with a mean equal to 1 and relative variance σ 2 = 0.003 (see Fig.1,a). Table 1. Example of novelty score table to select class based on N score

N score

A central pixel of scanning window belongs to the interval [µ - 3σ s , µ + 3σ s ]

A central pixel of scanning window does not belong to the interval [µ - 3σ s , µ + 3σ s ]

0..0.04

H

H

0.04..0.08

NS

S

0.08..0.16

NS

O

0.16..0.32

NO

O

0.32..0.64

E

E

0.64..1.0

O

NO

After applying the homomorphic transform to the noisy test image and setting the scaling factor ( s = 46 ) the variance of the additive noise be-

An Adaptive Novelty Detection Approach to Low Level Analysis of Images

575

comes σ s2 ¡Ös 2 σ 2 = 6.348 . We use this setting to compare three different methods: 1) the method of supervised primary local recognition based on radial basis neural network (NN) with 10 inputs (features are the bins of modified histograms of the image in the small scanning window, see [8] for details), 50 nodes in the hidden layer, and 6 outputs (6 classes). We will refer to this method as NN-Hi; 2) the second method uses a similar NN except that the number of inputs has been reduced to 6 local statistical parameters which have been calculated for each position of the scanning window (see [8,9]). This method will be referred to as NN-SP; 3) finally we will refer to our new method as ND. An equal window size of 5x5 has been chosen for all three methods in this comparison. Results are shown in Table 2. The analysis of this table highlights that the proposed method produced a superior recognition performance for all the main classes (H,E,O,NO) when compared to NN-Hi and NN-SP. In addition, our simulations demonstrate that the superiority of the proposed method holds also for images with different contrasts than those utilized in the training process for NN-Hi and NN-SP. Fig.1,b shows the novelty scores N score for noisy test image pixels: pixels in black color belong to homogeneous regions (not novel) while the white pixels are edges or highly novel. In Fig.1,c we can see that if the central pixel belongs to the interval [µ - 3σ s , µ + 3σ s ] this information is useful in classifying patches like S and O. Fig. 1,d shows how the information presented in Fig.1,b,c can be combined to recognize all classes (we use different colours to represent different classes). Fig.2,a depicts results for a real Ka-band SLAR image where Fig.2,b,c,d correspond to Fig 1,b,c,d, respectively. The size of the scanning windows was 5x5 and the relative variance was equal to 0.004. These values have been chosen according to a human expert analysing the homogeneous regions of the real image (see Fig. 2,a). Visual inspection also demonstrates that the proposed method presents encouraging performance and is able to distinguish between important components of the image such as edges, small objects, outliers and homogeneous regions.

Table 2. Results of correct image classifications by NN-Hi, NN-SP and ND

Recognized classes Methods

H

E

O

NO

ND

99.94 %

99.99 %

99.99 %

99.99 %

NN-Hi

99.5 %

93.6 %

99.6 %

90.6 %

NN-SP

99.1 %

91.8 %

94.7 %

85.8 %

576

Alexander N. Dolia et al.

a)

b)

c)

d)

Fig. 2. Real radar image processing: a) real Ka-band SLAR image; b) novelty score mapping; c) central pixel mapping; d) classification mapping

5

Conclusions

In this paper we have proposed a locally-adaptive novelty detection method for the primary local analysis in image data corrupted by mixed noise. The recognition results have outperformed those obtained with an RBF classifier on both artificial images and real Ka-band SLAR images.

References [1] [2] [3] [4] [5] [6] [7]

Bishop, C. Novelty Detection and Neural Network Validation. Proceedings, IEE Conference on Vision and Image Signal Processing (1994) 217--222 Nairac, A., Townsend, N., Carr, R., King, S., Cowley, P., Tarassenko L. A System for the Analysis of Jet Engine Vibration Data. Integrated Computer Aided Engineering 6 (1999) 53-65 Roberts, S.J. Novelty Detection Using Extreme Value Statistics. IEE Proceedings on Vision, Image and Signal Processing 146(3) (1999) 124—129 Schölkopf, B., Williamson, R., Smola, A., Taylor, J.S., Platt, J. Support Vector Method for Novelty Detection. In: Solla, S.A., Leen, T.K., Muller, K.R. (eds.) Neural Information Processing Systems (2000) 582--588 Campbell, C., Bennett, K.P. A Linear Programming Approach to Novelty Detection. Advances in Neural Information Processing Systems, 14 MIT Press, Cambridge, MA (2001) Ypma, A., Duin, R.P.W. Novelty Detection Using Self-organising Maps.: Progress in Connectionist Based Information Systems, 2 (1998) 1322--1325 Melnik, V.P. Nonlinear Locally Adaptive Techniques for Image Filtering and Restoration in Mixed Noise Environments: Thesis for the degree of Doctor of Technology, Tampere University of Technology, Tampere, Finland (2000)

An Adaptive Novelty Detection Approach to Low Level Analysis of Images

[8]

[9]

[10] [11] [12]

[13]

[14]

577

Dolia, A.N. Lukin, V.V., Zelensky, A.A., Astola, J. T., Anagnostopoulos, C. Neural Networks for Local Recognition of Images with Mixed Noise. In Applications of Artificial Neural Networks in Image Processing VI, N.M. Nasrabadi, A.K. Katsaggelos (eds.), Proc.SPIE 4305, (2001) 108-118 Dolia, A.N., Burian, A., Lukin, V.V., Rusu, C., Kurekin, A.A., Zelensky, A.A. Neural Network Application to Primary Local Recognition and Nonlinear Adaptive Filtering of Images. Proceedings of the 6-th IEEE International Conference on Electronics, Circuits and Systems, Pafos, Cyprus 2 (1999) 847-850 Bishop, C. Neural Network for Pattern Recognition. Oxford: Oxford University Press (1995) Marr, D., Vision: A Computational Invistigation into the Human Representation and Processing of Visual Information. San Francisco: Freeman (1982) Lukin, V.V., Ponomarenko, N.N., Astola, J.T., Saarinen, K. Algorithms of Image Nonlinear Adaptive Filtering using Fragment Recognition by Expert System. Proceeddings of IS@T/SPIE Symp. on Electronic Imaging: Science and Technology, San Jose (California, USA) 2318 (1996) 114-125 Niemistö, A. , Lukin, V., I. Shmulevich, I., Yli-Harja, O., Dolia, A. A Training-based Optimization Framework for Misclassification Correction. Proceedings of 12th Scandinavian Conference on Image Analysis, Bergen, Norway (2001) 691-698 Bendat, A., Pierson, J.S. Random Data: analysis and Measurement Procedures. Wiley series in Probability and Statistics, New York: Chichester: John Wiley&Sons, Inc., third edn. (2000)

The Image Recognition System by Using the FA and SNN Seiji Ito1 , Yasue Mitsukura2 , Minoru Fukumi3 , Norio Akamatsu3 , and Sigeru Omatu1 1

Osaka Prefecture University 1-1 Gakuen-chou, Sakai, Osaka 599-8531, Japan [email protected] [email protected] 2 Okayama University 3-1 Tsushima Okayama 700-8530 Japan [email protected] 3 The University of Tokushima 2-1 Minami-Josanjima Tokushima, 770-8506 Japan {fukumi,akamatsu}@is.tokushima-u.ac.jp

Abstract. It is difficult to obtain images only we want on the web. Because, enormous data exist in the web. A present detection system of images are keyword detection which is added the name of keyword for images. Therefore, it is very important and difficult to add the keyword for images. In this paper, keywords in the image are analized by using the factor analysis and the sandglass-type neural network (SNN) for image searching. As images preprocessing, objective images are segmented by the maximin-distance algorithm. Small regions are integrated into a near region. Thus, objective images are segmented into some region. After this images preprocessing, keywords in images are analyzed by using factor analysis and a sandglass-type neural network (SNN) for image searching in this paper. Images data are corresponded to 2-dimensional data by using these two methods. 2-dimensional data are plotted on a graph. Images are recognized by using this graph.

1

Introduction

Recently, enormous image data exist on the web by the rapid development of the Internet. Therefore, it is difficult to obtain images only we want on the web. In order to prevent these problems, there are image search systems, for example Google searching system [1]. However, by using this searching systems, the images only we want are not obtained. Because these systems consist of file name searching basically. Image searching systems are divided into 3 kinds; keyword search, similarity-based image retrieval, and browsing search [2]. However, there are many problems in them. The keyword searching method is used by using database. In similarity-based image retrieval method, we obtain images V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 578–584, 2003. c Springer-Verlag Berlin Heidelberg 2003


The Image Recognition System by Using the FA and SNN

579

Fig. 1. Process of preprocessing an image

by drawing images we want. However, this method is taken many tasks. The browsing search method means interactive method. In this method, however, it takes time to search images. Therefore, we focus on the keyword searching, This method is very simple and high speed system. Though, this method is difficult to get keywords in images. In this paper, it is proposed that images are recognized automatically. By recognition of images, we can expect to add keywords automatically. First, as images preprocessing, objective images are segmented by maximindistance algorithm. Small regions are integrated into a near region. Therefore, the objective images are segmented into some region. After this images preprocessing, keywords in images are analyzed by using factor analysis and a sandglasstype neural network (SNN) for image searching in this paper. Images data are correspond to 2-dimensional data by using these two methods. 2-dimensional data are plotted on a graph. After keywords are analyzed in detail, images are recognized by using this graph.

2

Image Preprocessing

In this section, the image preprocessing for image recognition system are introduced. Fig.1 shows the process of preprocessing in the proposed method. Images can be segmented into some region roughly by this preprocessing. This preprocessing is explained to below in detail. 2.1

Median Filtering

In the first procedure, impulse noises can be removed by using median filtering. By removing noises, images can be divided more roughly than original images. Median filtering is median density when image densities in the local region are ranked in the order of high (low) value.

580

2.2

Seiji Ito et al.

Maximin-Distance Algorithm

Image pixels with similar values after preprocessing of median filtering are classified into the same cluster by using a maximin-distance algorithm (MDA) [5]. There are another cluster methods, for example k-means. However, this method must be decided to the number of clusters k. The value of k changes with pictures. Therefore, it is difficult to decide the value of k automatically. The maximindistance algorithm is as it were the longest point is searched from existing cluster. If this distance is greater than a threshold, this point is set as the center of a new cluster. The detailed algorithm is as follows: 1. The cluster which sets x1 as a cluster center is made. 2. Di = min d(xi , Z¯j ) is calculated to each xi . However, Z¯i is set as the center of j

cluster Zi , Di (X, Y ) (Distance between a pixel X and a pixel Y) is computed as follows:
Di = (Xr − Yr )2 + (Xg − Yg )2 + (Xb − Yb )2 . (1) Where, Xr , Xg , Xb means RGB of a pixel X, Yr , Yg , Yb means RGB of a pixel Y. 3. l = max Di is calculated. Then, i is set as k. i

4. If l/M AX > r , then xk is set as the center of a new cluster, and back to step2, where, M AX is given by M AX = max d(Z¯i , Z¯j ) and r is parameter. ij

5. If l/M AX ≤ r, then this algorithm is finished. 2.3

Integration of Small Regions

There are a lot of small-sized regions by using MDA. In this case, the image is divided into big regions by integration of small regions [6]. In this study, a small region is defined as 0.5% or less area of a picture. The integration algorithm of small regions is summarized as follows: 1. Objective pixel is defined by the maximum outline pixel of small region. 2. In the 8-neighbor distance pixel of objective distance, regional information of another region is obtained. 3. In the obtained regional information, the most region information is selected. 4. Objective regions are integrated into selected region. 5. This algorithm is continued in all of small region. An original image is shown in Fig.2(a). The original image after this preprocessing is shown in Fig.2(b). Therefore, we can obtain the good regions by using this process.

The Image Recognition System by Using the FA and SNN

(a) An original image

581

(b) The result of preprocessing image

Fig. 2. The preprocessing image example

3

Factor Analysis

After regions are obtained by image preprocessing, feature parameters correlations are extracted by using factor analysis [4] in every keywords. In this section, we show the factor analysis. The equation of factor analysis is as follows:


Z = FA + UD

.

(2)

Where, Z is called data matrix. this matrix is measured data. F or U are not measured data, but assumed data. F is called common factor score (or factor score) matrix. U is called unique factor score. A is called factor loading. Factor analysis means assuming A, because A cannot be obtained from equation 2 directly. Factor analysis procedure is as follows: 1. 2. 3. 4. 5.

Generation the data matrix Z. Calculation of the correlation matrix R. Calculation of the matrix R∗ . Extraction of factors. Rotation of factors.

In this paper, the calculation method of R∗ is estimation of diagonal components. The decision method of diagonal components are maximum of correlation coefficient. The extraction of factors method are centroid method. The rotation of factors method are varimax method.

4

Sandglass-Type Neural Network (SNN)

2-dimensional data which are compressed these regions data are analized how keyword is distributed in the 2-dimensional graph. By compressing two or more information into two dimensions, compressing data can be plotted to 2-dimensional graph, and can be analized visually. The SNN [7] is used for compressing data. The SNN is one of the hierarchical Neural Networks. When input and output patterns are the same, compression data can be acquired on the hidden layers. The SNN structure in this paper is shown in Fig.3.

582

Seiji Ito et al.

Fig. 3. The structure of SNN

5

Computer Simulation

40 images are used for learning data. In 40 images, 140 regions are obtained by using image preprocessing with handwork. Important features are found by factor analysis. This result of found features is shown in Table 1. Showing this table, it is found that brightness (I) (included in HSI color system) is the most important feature parameter. Generally, the brightness (I) (included in HSI color system) is the most important parameter in image researches. Therefore, from these results, it was confirmed that the brightness (I) is the most important information in the various scene images. These selected features are compressed to 2-dimensional data by using SNN. Compressed data are plotted into 2-dimensional coordinate. This 2-dimensional coordinate are shown in Fig.4. Showing this Fig.4, it is found that all keywords are separated in this coordinate. By using Fig.4, images are recognized. This recognition method is as follows: 1. 2. 3. 4. 5.

Center points in every keywords are decided. Regions of unknown images are compressed by SNN. Distances between center and compressed points are calculated. The shortest distance is found in calculated distances. Regions are recognized as keywords with the shortest distance.

The distance calculation method is Euclidean distance. The distance between center point (xc , yc ) and a point (x, y) is as follows:  d = (xc − x)2 + (yc − y)2 . (3) Thus, regions are recognized. The recognition results are as follows. 74.4% accuracy can be obtained by the 7 keywords (sky blue, cloud, sandy beach, mountain, sun, tree and snow). 62.7% accuracy can be obtained by using the 8 keywords (addition rock). 42.9% accuracy can be obtained by using the 9 keywords (addition glassland), and 38.3% accuracy can be obtained by using the 10 keywords (addition sea). Therefore, this proposed method is effective,

The Image Recognition System by Using the FA and SNN

583

Table 1. The result of factor analysis Average RGBHS Sky blue * * * * Rock * Cloud * * * Sandy beach * * * * Sea Mountain * * * Glassland * * * Sun * * * * * Tree * * * Snow * * *

Variance IRGBHSI * * * * * * * * * * * * * * * * * * * * * * * *

because 7 keywords recognition accuracy is 74.4%. Generally, the separation of the keyword “sea” and “sky blue” is difficlut to do. In this paper, there is nothing to mistake sea for skyblue. There are 3 miss-recognitions which mistake sky blue for sea. Therefore, relatively good results can be obtained. The recognition accuracy by using the 10 keywords is not so high. The reason is thought as follows: In the keywords distributions in Fig.4, some is wide and some is narrow. Narrow distributions are converged more than wide distributions. Therefore, it is found that error recognitions are obtained by difference of distributions areas in every keywords. As this solution method, the numbers of sample data are the same in every keywords. It is roughly said that the more recognition objects I use, the lower recognition accuracy can be obtained. In the future works, many keywords separations will be achieved.

Fig. 4. The distribution of compressed data in 2-dimendional coordinate

584

6

Seiji Ito et al.

Conclusions

In this paper, we analized scene images for image search system. First of all, we do image preprocessing (median filtering, maximin-distance algorithm, and integration of small regions) for image segmentation. Next, we use factor analysis for extraction of important feature parameters. Finally, we use SNN for compression these parameters to 2-dimensional data, and 2-dimensional data are plotted. Images are recognised by using this graph. 74.4% recognition accuracy can be obtained by using the 7 keywords. Therefore, this proposed method is effectiveness. In the future works, a high recognition accuracy will be obtained by the improved system. Then, new feature parameters will be extracted, for example shape and so on. However, by using this preprocessing of image, accurate regional segmentation are not done so much. Therefore, accurate regional segmentation methods are necessary. Furthermore, the system of keyword addition automatically will be constructed.

References [1] http://www.google.com/ 578 [2] M.Mukunoki, M. Minoh, and K. Ikeda ”A Retrieval Method of Outdoor Scenes Using Object Sketch and an Automatic Index Generation Method”, IEICE D-II, Vol. J79-D-II No. 6, pp. 1025-1033 (1996) In Japanese 578 [3] John Hertz, Anders Krogh and Richard G. Palmer ”Introduction to the Theory of Neural Computation”, Addison-Wesley Publishing Company (1991) [4] S Shiba ”Factor Analysis Method”, University of Tokyo Press (1972), in Japanese 581 [5] M. Nagao ”Multimedia Information Science”, Iwanami Shoten, in Japanese 580 [6] S. Sakaida, Y. Shishikui, Y. Tanaka, and I. Yuyama ”An Image Segmentation Method by the Region Integration Using the Initial Dependence of the K-Means Algorithm”, IEICE Japan, No 2, pp.311-322 (1998), in Japanese 580 [7] E. Watanabe and K. Mori ”A Compression Method for Color Images by Using Multi-Layered Neural Networks”, IEICE Japan, No 9, pp.2131-2139 (2001), in Japanese 581

License Plate Detection Using Hereditary Threshold Determine Method Seiki Yoshimori1 , Yasue Mitsukura2 , Minoru Fukumi1 , and Norio Akamatsu1 1

University of Tokushima, 770-8506 2-1 Minami-Josanjima Tokushima, Japan {minow,fukumi,akamatsu}@is.tokushima-u.ac.jp 2 Okayama University, 700-8530 3-1-1 Tsushima Okayama, Japan [email protected]

Abstract. License plate recognition is very important in an automobile society. Also in it, since plate detection has big influence on subsequent number recognition, it is very important. However, it is very difficult to do it, because a background and a body color of cars are similar to that of the license plate. In this paper, we propose a new thresholds determination method in the various background by using the real-coded genetic algorithm (RGA). By using RGA, the most likely plate colors are decided under various light conditions. First, the average brightness Y values of images are calculated. Next, relationship between the Y value and the most likely plate color thresholds (upper and lower bounds)are obtained by RGA to estimate thresholds function by using the recursive least squares (RLS) algorithm. Finally, in order to show the effectiveness of the proposed method, we show simulation examples by using real images. . . .

1

Introduction

Much work on number plate detection with a digital computer has been done actively, and applied to car counting or the recognition of number plate of cars. For these demand, there are methods based on the color histogram. However these methods cannot cope with a change of back ground color (light condition). It is very difficult, by using nothing but the color information to extract a specific object from an image. People can recognize color information using various color characteristics. Objects are normally not monochrome except those made intentionally. Good results can be obtained by threshold obtained from binarization for a given image. However the results are not so good when several images are used, because of the variation in lightings, orientation etc [1]-[3]. In this paper, the best thresholds for every image are found using RGA. It can be automatically decided using RGA. By this method, the thresholds (upper and lower bounds) of a number plate are determined using RGA out of the strange images from which brightness and a type of a car differ. Next, the relation between this thresholds (upper and V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 585–593, 2003. c Springer-Verlag Berlin Heidelberg 2003


586

Seiki Yoshimori et al.

lower bounds) and brightness of car domain are changed into a function using a least-squares method. In the case of plate detection, thresholds are calculated based on this function. For this reason, the time to determine the thresholds is saved. Therefore, the calculation to detection also decreases. Moreover, in order to determine thresholds based on the brightness of a car domain, also to change of lighting conditions that it is effective. In addition, the images used for the simulation of this paper are photoed with the digital camcorder.

2

Image Processing

In this paper, images that photoed car while running an outdoor road with the digital camcorder are used. Photography conditions are as having been shown in Table 1. 2.1

Car Area Extraction

The continuous image of ten per car is used for a simulation. First of all, difference is taken to the image of two sheets followed in ten sheets. Next, in this difference, change asks for the large portion and makes the value the graph of length and a transverse direction [4]. The portion that exceeds a threshold in length and a transverse direction in this graph is extracted as a car domain. This situation is shown in Fig. 1. In addition, the portion in which cars are not contained in the stage of extraction, and the portion with a small plate are removed beforehand. 2.2

Color Systems

To obtain optical features of color information from color images, each attribute in a color system is used. Examples of color systems are RGB, YCr Cb , YIQ, etc. In this paper, the Y, Cr , Cb color system is used, which reflects the most color perception property of human beings. In particular, the value Y is used for the purpose of understanding the light concentration. A transform formula with the RGB table color is shown in the following.

Table 1. Photography conditions Shutter speed Picture size

1 / 30 [sec] length 340 pixel width 240 pixel Picture form 24bit color Environment fine day (daytime) rainy day (night) Photography time from 14:00 to 15:00 (daytime) from 18:00 to 19:00 (night)

License Plate Detection Using Hereditary Threshold Determine Method

(a) The original image

(b)Difference result

587

(c)An extraction result

Fig. 1. Car domain extraction 

   Y 0.29900 0.58700 0.11400  Cr  =  0.50000 −0.41869 −0.08131  Cb −0.16874 −0.33126 0.50000 Where Y is brightness, and Cr , Cb are colors. Note that, only Y is used in this paper. The method using thresholds can recognize objects at high speed. 2.3

Thresholds

How to choose the thresholds for the plate detection is a difficult problem. It is because a threshold changes by the condition of the light. Therefore, detection accuracy changes significantly with varying threshold value. In this paper, appropriate thresholds of plate detection are found by using RGA under the condition of an unknown Y value, which is calculated by the above equation. The plate domain for samples is extracted from a car domain by viewing. The example of a car domain and the extracted plate domain is shown in Fig. 2. When calculate the thresholds, these images are used. 2.4

Genetic Algorithms

On the problem with which it deals in this paper, we have to change the thresholds for plate detection for each image. The images from which brightness differs are also stabilized and a plate position must be detected. However, when a thresholds (upper and lower bounds) changes, the results after processing may differ greatly. In this paper, the genetic algorithm that used real numerical value

Fig. 2. A sample of a plate extraction result

588

Seiki Yoshimori et al.

coding (RGA) is adopted as optimization algorithm. When design variable value is the continuous function optimization problem, which is the real number, RGA can be efficiently taken over to a parent individual’s feature. Therefore, as compared with GA, which used the bit string for the chromosome, efficient search is possible [5]. Chromosomes In this paper, since a parameter was a thresholds (upper and lower bounds), it has set as the form that shows the form of a chromosome in Fig. 3. Fitness Functions In this paper, the degree of adaptation was set up as follows.  ✏ RECOGP LAT E < 0.20

if else

f itness =

RECOGP LAT E RECOGCAR

f itness = RECOGP LAT E ∗ BRESULT ∗GRESULT ∗ RRESULT

if (BMAX − BMIN ) , (GMAX − GMIN ),

(RMAX − RMIN ) > 80.0

(< 80.0) BRESULT ,GRESULT , RRESULT =

1.05 (MAX−MIN )0.01

else ({(BMIN , GMIN , RMIN ) + 1.0},

✒

{(BMAX ,GMAX ,RMAX ) − 3.0})BRESULT ,GRESULT ,RRESULT 1.05 = (MAX−MIN )0.01

✑

RECOGP LAT E : The rate of the number of pixels which exists in the threshold (upper and lower) bounds determined by GA to thetotal number of pixels in a plate domain RECOGCAR : The rate of the number of pixels which exists in the threshold (upper and lower bounds) determined by GA to all thepixels in a sample cars domain BMIN , GMIN , RMIN : The minimum of the threshold value of B, G, and R which were determined by GA BMAX , GMAX , RMAX : Maximum of the threshold value of B, G, and R which were determined by GA By using a thresholds function, the rate of recognition is high and what has as small the range of a threshold value as possible is chosen. Therefore, it is thought that it is expectable to obtain a desirable result to the method proposed in this paper.

Fig. 3. A sample of chromosome

License Plate Detection Using Hereditary Threshold Determine Method

589

Fig. 4. The estimated threshold line by using RLS algorithm(B, upper and lower bounds)

2.5

Thresholds Function

The relation between the thresholds (upper and lower bounds) and the average of the brightness in a car domain are used to calculate of the thresholds function to (upper and lower bounds). The thresholds function of the upper and lower bounds are shown in Fig. 4. The value calculated from this function is effective to the difference of light conditions or car color. Because, it has calculated based on the brightness of the whole car domain. Moreover, the thresholds of the plate detection in a strange image are determined based on the calculated function. That is, the threshold value, which detects the plate position in an image, is determined only calculating the brightness of an image. Therefore, the amount of calculation, which is needed for detection, can also be reduced. 2.6

Symmetric Property

If a domain is extracted only with a thresholds function, a domain, which has similar thresholds of a plate, may also be chosen. In this case, which domain finally is determined as a plate domain poses a problem. The method of determining a domain with many pixels, which exist within the limits of thresholds as a plate domain, is considered as one. However, by this method, a possibility of causing incorrect recognition in a plate domain, which has similar thresholds, becomes high. Then, in this paper, the degree of symmetry is proposed and it is used as the domain specification method. The degree of symmetry shows as the following equations. Sim Symmetry = P ixel∗2 Sim : The degree of similar of two points which is applicable (difference of pixel value) P ixel : The number of pixels of the half of a chosen domain

590

Seiki Yoshimori et al.

By using this method, the area chosen as a plate domain is equally divided into two first at length or a transverse direction. Next, the concentration value of the pixel, which becomes symmetry to a division line in two domains, is compared. Finally, the symmetry of a domain is measured by breaking the total of the value by the value of the half of the number of pixels of the whole domain.

3

Flow of Processing

The process in the case of actual plate detection is shown below. 3.1

Preprocessing

First of all, the cars domain that has brightness and cars color differ is prepared, and these image are formed into 2 values images using difference between frames. In quest of length and the number of pixels of a transverse direction, it graphizes using this 2-value image. In these graphs, determine the domain beyond a threshold value as a cars domain. Next, a plate domain is started from the extracted cars domain. The upper and lower bounds of thresholds for each image calculates by the real numerical value GA using the plate domain image and cars domain image. Finally, the thresholds function calculates from the relation between the average value of the brightness of car domain and threshold value, which is calculated from RGA. 3.2

Detection Processing

First of all, an inputted image is formed into 2 values image using difference between frames. Graph of length and transverse direction pixels calculate from 2 value pictures. In both these graphs, the domain beyond thresholds is determined as a cars domain. Next, the average value of the brightness in the determined cars domain is calculated. Each R, G, and B thresholds (upper and lower bound) are determined from thresholds function using car domain average. The number of pixels in a cars domain, which exists within the limits of the thresholds, is calculated. Graphs (length and transverse direction) calculate based on the pixel, which exist within the car domain and limits thresholds. By using these graphs (length and transverse direction), determine the domain beyond a threshold value as a plate domain. When two or more selection domains exist in the case of plate domain determination, in addition to the rate of recognition, the degree of symmetry of a selection domain is also used, and, the thing most in character with a plate domain is determined. Finally an exact position is determined by performing template matching to the inside of the determined domain [6], [7].

4

Computer Simulations

In this section, in order to show the effectiveness of the proposed method, computer simulations are done by using the 100 car area images.

License Plate Detection Using Hereditary Threshold Determine Method

591

Table 2. The parameters used in GA The number of individuals The number of generations The number of elite preservation Occurrence probability of crossover The mutation rate

200 1000 20 0.60 0.07

Table 3. Simulation Result

Method1 Method2 Method3 Method4 Method4 (night)

Rate of detection Processing time (%) (sec) 57.0 0.23 82.0 0.16 85.0 0.17 91.0 0.18 70.2 0.18

CPU PentiumIII(500MHz) Method1 : Only template matching Method2 : Threshold value function Method3 : Template matching and threshold vlue function Method4 : Template matching, threshold value function and symmetric property

4.1

Simulation Conditions

When computer simulation has done, parts where dose not contain a car are deleted. The deleted parts are part upper of original image and outside centerline. The number of the individuals, the generation shift number of times, crossover probability, mutation percentage in RGA is shown in Table 2. 4.2

Simulation Results

When more than 70 % of plate areas made contained in the square of result, the detection is succeeding. As a results of the detection are shown in Table 3 and the example of this simulation results are shown in Fig. 5. The one, which succeeded in the detection, can extract a plate area from these results correctly. Moreover, we have confirmed that the detection to the plate which was difficult by the research until now because of the car has the similar color is possible, too. When examining about the one which failed in the detection, there has been detected a plate in the mistaking with the headlight. It is because the color of the plate domain and the color of the headlight are similar, so the thresholds are also similar and the size is also similar too. Moreover, since the cover was

592

Seiki Yoshimori et al.

Fig. 5. Simulation results

attached to the plate portion, the cars by which the brightness of a plate portion became extremely low had caused incorrect detection.

5

Conclusions

In this paper, we propose the new thresholds determination method in the various background by using the RGA. In the conventional thresholds detection method is not robust for the changing background. That is, in the conventional method, we determine the thresholds, if the background color is changed, the detection accuracy is getting worse. However in this paper, determined the thresholds are changed adaptive by using thresholds function. Therefore, if the background is changed (for example, rainy day thresholds and day time thresholds), we can obtain the suitable thresholds by using RGA. Then, we can obtain the good detection accuracy. In order to show the effectiveness of the proposed method, we show the simulation. In the simulation, we show the effectiveness the proposed a new thresholds determination method using RGA.

References [1] T. Naitou, T. Tukada, K. Yamada, and S. Yamamoto, “License Plate Recognition Method for Passing Vehicles whith Robust Sensing Device against Varied Illumination Condition,” IEICE Tech. Report(D-II), vol. J81, no. 9, pp. 2019–2026, 1998. 585 [2] H. Fujiyoshi, T. Umezeki, T. Imamura, and T. Kaneda, “Area Extraction of the Licence Plate Using Artificial Nerual Network,” IEICE Tech. Report(D-II), vol. J80, no. 6, pp. 1627–1634, 1997. [3] K. Tanabe, H. Kawashima, E. Marubayashi, T. Nakanishi, A. Shio, and S. Ohtsuka, “Car License Plate Extraction Based on Character Alignment Model,” IEICE Tech. Report(D-II), vol. J81, no. 10, pp. 2280–2287, 1998. 585

License Plate Detection Using Hereditary Threshold Determine Method

593

[4] T. Hada and T. Miyake, “Tracking of a Moving Object with Occlusion by Using Active Vision System,” IEICE Tech. Report(D-II), vol. 84, no.1, pp.93–101, 2001. 586 [5] M. Haseyama, M. Kumagai, and T. Miyamoto, “A Genetic Algorithm Based Picture Segmentation Method,” IEICE Tech. Report(D-II), vol. J82, no. 11, pp. 1903– 1911, 1999. 588 [6] S. Muramatsu, Y. Otsuka, Y. Kobayashi, E, Shimizu, “Strategy of High Speed Template Matching and Its Optimization by Using GA,” IEICE Tech. Report(DII), vol. J83, no. 6, pp. 1487–1497, 2000. 590 [7] M. Ikeda, S. Yoshida, K. Nakashima, N. Hamada and H. Yoda, “High Speed Template Matching by Monotonized Normalized Correlation” IEICE tech. Report(DII), vol. 83 no. 9 pp. 1861–1869, 2000. 590

Recognition from EMG Signals by an Evolutional Method and Non-negative Matrix Factorization Yuuki Yazama1 , Yasue Mitsukura2 , Minoru Fukumi1 , and Norio Akamatsu1 1

University of Tokushima 2-1 Minami-josanjima Tokushima 770-8506 Japan Telephone + 81 -88-656-7488 {rabbit,fukumi,akamatsu}@is.tokushima-u.ac.jp 2 Okayama University 3-1-1 Tsushima Okayama Japan [email protected]

Abstract. In this paper, we propose a method of the noise rejection from a signal acquired from many channels using the Electromyograph (EMG) signals. The EMG signals is acquired by the 4th electrodes. EMG signals of 4ch(es) is decomposed into two processions using Non-Negative Matrix Factorization(NMF). And noise rejection is performed by applying the filter obtained by GA to the decomposed matrix . After performing noise rejection, EGM signals is reconstructed and the acquired EMG signal is recognized. The EMG signals based on 7 operations at a wrist are measured. We show the effectiveness of this method by means of computer simulations.

1

Introduction

EMG is an electrical recording of muscle activity which is measured from the surface of skin. It is possible to perform a control of manipulator or prosthetic hand by utilizing EMG as an incoming signal [1]-[3]. A facial expression and gesture is used as a natural interface of man, and a machine and a computer. However systems using EMG which can be used as interface more easily have studied since 1960s. The myoelectric upper limb prosthesis using EMG has a significant role in the field of rehabilitation medicine or welfare as an equipment which realizes natural motions as man’s hand by engineering progress [4]. However, the ecology signal is the multiple electrode signals. With the result that the ecology signal has strong nonlinear nature, various neural networks (NN) which has nonlinear separation capability are used for recognition of the ecology signal [5]. Therefore, the ecology signal includes the signal emitted from various parts. Then, research of the noise rejection using ICA or various filters is advanced. It is possible to control an unnecessary spectrum by performing noise rejection and strengthen the feature for each of the signals. The noise rejection with good filter leads to V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 594–600, 2003. c Springer-Verlag Berlin Heidelberg 2003


Recognition from EMG Signals

595

accuracy improvement in a recognition system and clear the way for the feature analysis. On the one hand, NMF is one of techniques in the feature extraction by imitating a function in man’s vision [6][7]. It is thought that noise rejection can be performed leaving the visual feature using NMF. This work proposes a method of carrying out noise rejection to the EMG signals of 4ch using non-negative matrix factorization(NMF). NMF is the method of decomposing non-negative matrices into the product of non-negative matrix. By covering the mask generated by using GA, the element of matrix is chosen to reconstruct EMG signal. We carry out experiments with the reconstructed EMG signal and NN and show the effectiveness of the present approach.

2

Electromyograph

One of the purpose of the ecological signal processing make clear the advanced information processing mechanism which a living body has through the analysis of the ecological signal. Then, it is reflected to an artificial system. Moreover, it is also the one of purpose to objectify and quantify diagnosis depending on doctor experience by medical diagnosis. There are many ecological signals (electroencephalogram(EEG), electrocardiogram (ECG), and electromyogram (EMG)). The living body signal used in this paper is a signal (EMG signal) extracted by operation of a wrist, and uses what recorded the activity potential of skeletal muscle from the outside of a cell. 2.1

EMG Pattern and Preprocessing

We focuse on 7 operations which are considered as basic wrist motion types, neutral, top, bottom, left, right, outward rotation and inward rotation. The EMG patterns are recorded under the wrist motion. These operations are thought as standard motions and reflect a subject’s simple operation. Data extraction is preformed for two male subjects and 30 sets are measured, each of which is composed of 7 operations. The EMG signals are measured by a surface electrode. Moreover, the surface electrode of dry type (electrolysis cream was not used) is adopted because of a subject’s displeasure and a practical application as PDA operation machines. The EMG pattern which is amplified electrically the feeble signal emitted from a living body, and is obtained as time series data. 4 electrodes are placed around the wrist (see Figure 1). An example of EMG pattern is shown in Figure 2. The EMG patterns are measured under sampling frequencies of 20.48kHz and for 2.048 seconds for all electrodes. The raw EMG signals have to be preprocessed and be reduced to a small number of attributes in order to achieve real-time processing and reduce the learning cost. We apply the FFT with the Hamming windowing function with 256 points to the raw EMG pattern and change the raw EMG pattern into a frequency domain. It is considered that the EMG signal after the 10,000th point is stationary. The frequency components are produced by shifting every 128 points in FFT of 256 points after the 10,000th. The feature vector is computed as an average over ten times in frequency components.

596

Yuuki Yazama et al.

Fig. 1. Wrist with 4 electrodes

3

Non-negative Matrix Factorization

NMF is a technique of decomposing non-negative matrix V into non-negative matrices W and H. V ≈ W H.

(1)

The rank of the approximate matrix W H is generally choosed in the rank of n × (n + m) × r < n × m,

(2)

W H is considered as the matrices which compressed the original matrix V . V is linear combination of W which is weighted with the elements of H. However, NMF performs matrix decomposition on non-negative restrictions conditions as above the constraint. Therefore, the obtained decompound matrix can express the original matrix by linear combination of only addition without subtraction. This show that the whole matrix be expressed only by the specific element, and man’s intuition is reflected. 3.1

Updata Rule

NMF approximates the matrix V by the product of two matrices. As a measure of the approximation, the distance during two matrices and the difference during two matrices are used. Firstly, we describe the technique which updates W H by minimarizing the distance two matrix (V and W H). The approximate matrix W H is updated under the following rules :

H ij = Hij

(W T V )ij (W T W H)ij

(3)

W ij = Wij

(V H T )ij , (W HH T )ij

(4)

Recognition from EMG Signals

597

where H and W are the updated decompound matrices. In performing repetition operation, the update is applied as H to H and W to W . This updating rule uses the Euclid distance between two matrices shown in equation 5. Then the matrix W H which approximates the original matrix V is repeatedly obtained until it it completed by the purpose function.

F = (Vij − (W H)ij )2 . (5) i

j

Secondly, we describe the technique which updates W H by minimizing the difference between V and W H. This technique updates the approximate matrix W H using the updating rules : H ij = Hij




(V )kj (W H)kj

(6)


(V )ik Hjk (W H)ik

(7)

Wki

k

ˆ ij = Wij W

k

ˆ ij W W ij =  . ˆ k Wk j

(8)

It is applied repeatedly by updating using the distance. This updating rule obtains W H to apply repeatedly so that the criterion function shown in the formula (9) converges to the local maximum.

F = (Vij log((W H)ij )) − (W H)ij ). (9) i

j

This criterion function uses the kullback-leibler divergence as an approximate measure.

4

The Recognition System of the EMG Signal

EMG signal is reconstructured using NMF as pretreatment of the EMG recognition system. It is then recognized using reconstructured EMG signal. However, in order to reconstruct it, we have to select an element in the matrix. Then, GA determines the used element to reconstruct EMG signal. The used element is selected form matrix H which obtained by NMF. Matrix W is considered to be base showing the feature of data, and matrix H is considered to be a coefficient for restoring the original data using matrix W . Therefore, the noise is suppressed by selecting the required coefficient in the element of matrix H for reconstruction. Furthermore, it is thought that the feature is emphasized. 4.1

Genetic Algorithm

Genetic Algorithm(GA) is the method of having considered evolutionary process. It is one of optimization techniques which calculate the better solution, changing

598

Yuuki Yazama et al.

Fig. 2. Crossover operation

Fig. 3. Mutant operation

two or more solutions hereditarily. Research which grasps the correlation of two or more objects using a GA is done [8]. This research can use the method of pattern recognition on the Euclid space by the map to the Euclid space [8]. And the analysis of an object from a new starting point is attained. We proposes the method extracting the feature and characteristic using GA from many features acquired. The unnecessary element in the matrix H is included. Then, the required element in the matrix H is chosen by using GA. The chromsome used for GA is based on binary coding. The element corresponding to a code “0” is not used, but the element corresponding to a code “1” is uesd. The chromsom is twodimensional model. Mutation and crossover of the genetic operator are described as follows. First, crossover perform the process which specifies the range made to cross and replace specified portion(see Fig.2). Mutation also specifies the range which performs mutation, reverses the bit of the binary coding(see Fig.3). The portion with the deep color in the figure is the range of crossover or mutation. The fitness function to each chromsom is described. The fitness function is defined by the product of the distance toward the original signal, and the average value of the element to be used. x is original signal and the reconstructed signal is x ˆ. i is number of sensor, and j is number of data. The distance toward the original signal is defined :

 (xij − x ˆij )2 . (10) D= i

j

The average value of the element to be used is defined :  Hij , U= n

(11)

Recognition from EMG Signals

599

Table 1. Recognition acurracy Subject1 ∗ Subject2 ∗ Subject3 ∗

N 88.1% 85.3% 81.3% 75.5% 55.8% 58.0%

T 40.8% 64.8% 51.5% 72.3% 32.6% 56.0%

U 53.8% 65.8% 69.1% 89.8% 66.6% 85.0%

R 63.0% 72.8% 65.0% 81.1% 62.6% 75.5%

L 75.5% 92.3% 57.3% 75.3% 63.1% 81.6%

RI 57.3% 67.0% 47.8% 64.1% 60.8% 74.0%

RO 74.0% 86.0% 50.5% 53.1% 51.8% 49.6%

Total 64.6% 76.2% 60.4% 73.0% 56.2% 68.5%

where H described here is only the element to be used. The fitness function is defined : F = DU.

(12)

As the reason using the average value of the element to be used, it is considerd that there is little influence to the reconstructed signal by using the big value which the element of matrix H has. Thus, it is thought that the waveform is generable near the original signal, deleting the unnecessary element. Furthermore, the noise could be removable leaving the feature which EMG signal has. 4.2

Result

The pretreatment, which is noise rejection and emphasis of the feature, for the recognition system is performed by NMF and GA. The recognition experiment is then performed using the acquired signal. The computer simulations are perfomed using the data obtained from 3 subjects. 30 sets of three male subjects are measured, each of which is composed of 7 operations. The frequency band to be used is from 40Hz to 1000Hz. The recognition experiment is performed by NN. The number of input layer units is 100, the number of hidden layer units is 10 and the number of output layer units is 7. The result of simulation shows average recognition accuracy by 30 simulation of NN. In addition, the comparison simulation of NN using the original EMG signal. The average of the recognition accuracy obtained by the simulation is shown in Table 1. Recognition accuracy is improved about 10% in all the results of 3 subjects. It is thought that the feature which the EMG signal has is emphasized and the signal influenced by other parts is reduced that influence by operating H matrix decomposed using NMF.

5

Conclusion

NMF is one of the techniques of the feature extraction imitating the function in cerebral vision. The filter for reconstructing the signal by using NMF and GA is created. Moreover, the signal is generated using the filter and the recognition

600

Yuuki Yazama et al.

experiment is carried out using the reconstructed signal. We verify effectivity of the waveform shrapening using NMF by computer simulation. As feature work, this technique is proposed as noise rejection. However, we do not know the generated signal in detail. It is possible that the feature for generating the original waveform not only to W matrix used as a base but to the element of H matrix is appeared. Therefore, the process of the signal waveform by NMF and GA will be analyzed. Moreover, it is thought that the important frequency band will be identified from H matrix obtained by NMF and GA. We will verify to conduct recognition experiment using the few amount of the feature by specifying the important frequency band.

References [1] D.Nishikawa, W Yu, H.Yokoi and Y.Kakazu. n-Line Learning Method for EMG Prosthetic Hand Controlling. IEICE, D-II Vol J82, No.9, pp.1510-1519, September 1999, in japanese. 594 [2] D.Nishikawa, H.Yokoi, and Y.Kakazu. Design of motion-recognizer using electoromyogram. In Robotics and Mechatronics Division, June 1998, in japanese. [3] D.Nishikawa, W Yu, H.Yokoi, and Y.Kakazu. On-Line Supervising Mechanism for Learning Data in Surface Electromyogram Motion Classifiers. In IEICE, D-II Vol J84, No.12, pp.2634-2643, December 2001. 594 [4] Yuichiro KAWAMURA, Masako NOZAWA, Haruki IMAI, Yoshiyuki SANKAI. Stand-up Motion Control for Humanoid using Human’s Motion Pattern and EMG. Academic lecture in the 19th Robotics Society of Japan, 2001,in japanese. 594 [5] M.Vuskovic, and S.Du. Classification of Prehensile EMG Patterns With Simplified Fuzzy ARTMAP Networks. JIJCNN, Honolulu, Hawaii, May 2002. 594 [6] D.Lee and H.Seung. Algorithms for non-negative matrix factorization. NIPS 2000, 2000. 595 [7] D.Lee and H.Seung. Learing the parts of objects by non-negative matrix factorization. Nature, Vol.401, pp.788-791, 1999. 595 [8] Shin’ichiro OMACHI, Hiroko YOKOYAMA, and Hirotomo ASO. Subject Arrangement in the Euclidean Space Using Algorithm. IEICE, D-II, Vol.J82-D-II No.12 pp.2195-2202, 1999, in japanese. 598

A Feature Extraction Method for Personal Identification System Hironori Takimoto1 , Yasue Mitsukura2 , Minoru Fukumi1 , and Norio Akamatsu1 1

The University of Tokushima 2-1 Minami-Josanjima Tokushima 770-8506 Japan {taki,fukumi,akamatsu}@is.tokushima-u.ac.jp 2 Okayama University 3-1-1 Tsushima Okayama 700-8530lJapan [email protected]

Abstract. Recently in the world, many researches for individual identification method using biometrics are widely done. Especially, personal identification using faces are used because of needless for physical contact. However, when the number of registrant of a system increase, the recognition accuracy of system will get worse certainly. Therefore, in order to improve the recognition accuracy, it is necessary to extract the feature area effectively for getting the high recognition accuracy. In this paper, we analyze and examine about the individual feature in a face using the GA and the SPCA. Thus, by removing the area which is not valuable, we think that recognition accuracy becomes high. Then, in order to show the effectiveness of the proposed method, we show computer simulations by using real image.

1

Introduction

Consequently, many researches of personal individual identification technique using biometrics are widely studied[1, 2]. By the way, biometrics is not in need of memory or carrying of cards, and only registrant is accepted. Especially, the face is always opened to society, it has little psychological burden compared with other physical feature. There are some personal individual identification methods using the front face image[3, 4]. One is an extraction method of feature points using edge detection and others are template matching, and so on. In the extraction method of the feature points, the form of the portion that constitutes a face and an individual difference of the position are used for recognition. In the technique of the pattern matching, a face is regarded as a value pattern of the gray under arrangement of 2-dimensions. However, it becomes redundancy expression by the vector with the huge number of dimensions, and it is influenced by the change of the environment[4].

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 601–608, 2003. c Springer-Verlag Berlin Heidelberg 2003


602

Hironori Takimoto et al.

On the other hand, the principle component analysis (PCA) is used for yielding a feature vector to discernment[4, 5]. However in the scheme of PCA, it is not easy to compute eigenvectors with a large matrix when considering the cost of calculation to adapt for time-varying processing. Therefore, the simple principle component analysis (SPCA) method has been proposed in order for PCA to simulate at high-speed[6, 7]. Moreover, the technique of calculating for the feature vector of face pattern by SPCA is effective in recognition of front face pattern. By the way, the deterioration of the recognition accuracy by the number of registrant increase is a very important problem. Therefore, in order to improve accuracy, it is necessary to extract the feature area effective for recognition. In this paper, we analyze and examine about the individual feature in a face using the genetic algorithm (GA) and the SPCA. Conventionally, it is said that the feature region of face is in eyebrows, an eye, a nose and a mouth. However, it is thought that they always are not regular, and change with registrants of system. Thus, by removing the area which is not valuable, we think that recognition accuracy becomes high. Then, in order to show the effectiveness of the proposed method, we show computer simulations by using real image.

2

SPCA

The SPCA has been proposed in order for the PCA to chieve high-speed processing. It is confirmed that it is effective in information compression at the handwritten digits and dimensionality reduction at a vector space model[7]. The SPCA produces approximate solutions without calculating a variance-covariance matrix. The SPCA algorithm is given as follows: 1. Collect of an dimensional data set, V = {v1 , v2 , ..., vm } 2. X = {x1 , x2 , ..., xm }, which are obtained by subtracting which is the average value of form the center of gravity of set of the vectors, is used as input vectors. 3. The column vector is defined as connection weights between the inputs and output. The first weight, is used to approximate the first eigenvector. The output function has a value given by: y1 = αT1 xj

(1)

4. Using the following equation (2) and (3), repetitive calculation of arbitrary vector suitably given as an initial value is carried out. As a result, the vector can approach the same direction as.
j Φ1 (y1 , xj ) k+1 (2) a1 =
 j Φ1 (y1 , xj )  y1 = (ak1 )T xj

(3)

A Feature Extraction Method for Personal Identification System

where is a threshold function given as  +xj Φ1 (y1 , xj ) = −xj

y1 ≥ 0 otherwise

603

(4)

5. Using the following equation (5), we remove the first principal component vector from the dataset in order to find the next principal component. xi = xi − (aT1 xi )a1

(5)

We obtain principal components because we substitute with and with in equation (2), (3) and perform repetitive calculation once again.

3

Face Image Data

In this paper, we used 72 adult men’s, 28 adult women’s in total 100 people images for registrant, and they are 6-face images par person (total 600 face images). Moreover, we used 113 adult men’s, 139 adult women’s in total 252 people images for un-registrant, and they are a face image par person (total 252 face images). Although the subjects are various generation and race, does not have glass. The subject is looking at the lens of the camera by natural expression. Background is restricted, however lighting is not restricted. In this paper, it is need to normalize the face image to recognize the individual. We show normalization method of face image. The face image is normalized based on both eyes. The reason for having used the eye for normalization of face image is as following. The first, eye is having been easy to perform the normalization about a rotation and a size, compared with a lip, a nose, or an ear. Next, many researches of extracting the region of an eye are proposed[8, 9]. Therefore, to use eye for normalization of face image is efficient. First of all, an original image is changed into gray scale image (420×560 pixels, 24bit color; Fig. 1(a)), and median filter is performed in order to remove noise. Next, center positions of both eyes are extracted. Then, the line segment joining both eyes is rotated so that it matches the horizontal line. Furthermore, the distance between both eyes is made 40 pixels by scale change. Moreover, in order to diminish influence of hair and clothes, an image is cut out as shown in Fig. 1(c). Finally, in order to ease the influence by photometric property, gray scale transformation is performed. Therefore, the image of the Fig. 1(a) is normalized as the Fig. 1(b).

4

Individual Feature Extraction

In this paper, the region segment where the individual feature has shown up is extracted by using the GA and the SPCA. Moreover, it is used for recognition. Recognition is obtained the good accuracy by removing the region which is not valuable for it. Furthermore, it is considered that it can mitigate calculation cost required for recognition.

604

Hironori Takimoto et al.

(b)The normalized image (a) The original image (c)The outline of the normalization

Fig. 1. The normalization of a face image

4.1

Region Segment and Genetic Coding

As shown in Fig. 2(a), the region is segmented. One block size is a square (4×4 pixel). In the case of using the Fig. 2(b), when performing SPCA for fitness function of the GA. This figure is performed sampling by average gray-scale value in segmented regions, in order to perform the high-speed processing and dimensionality reduction. The chromosomes of GA are binary coding, and they are expressed in 2-dimensional arrangement. Each gene is one-to-one correspondence with the segmented region. When chromosome value is 1, the related region is used for recognition. 4.2

Fitness Function

The fitness function is shown in equation (6). The numerator means that the fitness value becomes high, if the difference of individual in the selected region becomes large. ”Sab” means total of variance in the selected regions. ”Dim” means total of rate of contribution to the 5th, when performing the SPCA to the selected regions. Then, This denominator means that the fitness value becomes high, if number of selected region becomes a few. By using this fitness function, we can obtain the small-sized inputs and high recognition accuracy.

(a)The region segment

(b)The mosaic pattern

Fig. 2. The region segment and mosaic pattern

A Feature Extraction Method for Personal Identification System

605

In this paper, the elite preservation method and the roulette selection method was adopted as the selection method of the individual. The rate of the individual who reserves as the elite is about 10 %. Moreover, creation of a new individual chooses 40 % with good fitness value from the whole, and is performed using the GA operations (selection, crossover, mutation). F itness =

4.3

 Sab · (1 − Dim)  Pix Pixmax

(6)

Feature Extraction Simulation

The individual feature extraction is performed to 100 registrants. The experimental conditions are shown in Table 2. In the result, average of ten individuals with good fitness value is shown in Fig. 3(b). This figure means that the more blackish region was chosen by many individuals. In this paper, because face image is normalized based on both eyes, the selected region has more eye’s region than mouth’s region. From these results, it was confirmed that the proposed method by using the GA gets characteristics of faces. Therefore, the GA is effective for the getting individual characteristics.

5

Personal Identification Simulation

In order to show the effectiveness of proposed method, we show computer simulations by using real image. Recognition process is as follows, and we show each details of process.    

step step step step

5.1

1 2 3 4



Normalization face image Dimensionality reduction of image pattern by SPCA Calculation of similarity (cosθ) between eigenface and face image Learning and recognition by NN

Neural Networks

In this paper, the NN is used for a classification and is a three-layered type. It is learned by using the back-propagation (BP) method. The number of input layer

Table 1. The experimental conditions of GA Generation Individual Bit of gene Mutation rate

500 50 400 0.05

606

Hironori Takimoto et al.

(a)The sample face image

(b)The result image

Fig. 3. The result of feature extraction

units is same as the number of similarity (between each eigenvectors from which the total of rate of contribution becomes 90 % or more). The number of output layer units is same as the registrant of the system. If the output of an output unit exceeds a threshold, then it accept as a registrant corresponding to the unit. If the output of two or more units exceeds a threshold or no output of a unit exceeds a threshold, then it is rejected. 5.2

Evaluation Simulation

In this paper, we perform the simulation of six patterns which is shown in Table 3. This table shows the number of used pixels, used rate, and calculated principal component by SPCA for simulation. The case 1 is used the whole face image which was normalized. The regions of the case 2-4 are determined by result of GA. In the result, False Rejection Rate (FRR) and False Acceptance Rate (FAR) are shown in Fig. 4. From the result of FRR, a difference of each result is not seen when the threshold is comparatively low. However, if the threshold is comparatively large, it is found that the case 2, 3 accuracys are better than the case 1. Therefore, the proposed method is effective in the condition where a threshold is large, and the situation that a un-registrant is not allowed. However, the case 4 accuracy is bad than the case 1. It suggests that there is a bad influence, when the amount of information is reduced too much. Moreover, equal error rate (EER) and threshold are shown in Table 3. EER is the error rate in the threshold that FAR and FRR cross, and it is used often for verification of recognition system effectiveness[10]. In the result of EER,

Table 2. The simulation pattern case case case case

1 2 3 4

Used pixels Used rate(%) Input 6400 100.0 50 4528 70.7 50 3552 55.5 49 2672 41.7 47

Table 3. The EER and threshold case case case case

1 2 3 4

EER(%) 2.15 1.33 1.55 2.75

θ0 0.57 0.58 0.52 0.54

A Feature Extraction Method for Personal Identification System

30

Recognition Rate(%)

25 Recognition Rate(%)

20

case 1 case 2 csae 3 case 4

20 15 10

607

case 1 csae 2 case 3 case 4

15

10

5

5 0 0.4

0.45

0.5

0.55

0.6 0.65 0.7 Threshold

0.75

0.8

(a)The false rejection rate

0.85

0.9

0 0.4

0.45

0.5

0.55

0.6 0.65 0.7 Threshold

0.75

0.8

0.85

0.9

(b)The false acceptance rate

Fig. 4. The simulation result of recognition the case 2 accuracy is the best in four patterns. Then, the threshold of case 2 is the highest. Thus, the proposed method is effective in the severe situation of security environment such as high threshold. In coclusions, the proposed method is effective, and obtained the individual characteristics for recognition.

6

Conclusions

In this paper, the feature extraction method for personal identification was proposed be using the SPCA and the GA. Furthermore, effectiveness of proposed method was checked, as compared with the case which was used the whole face image. In the result, we obtained EER is 1.33%.

References [1] Y.Yamazaki, N.Komatsu: ”A Feature Extraction Method for Personal Identification System Based on Individual Characteristics”. IEICE Trans. Vol.D-II, Vol.J79. No.5, pp.373-380, (1996) in Japanese 601 [2] A. K.Jain, L.Hong, and S.Pankanti: ”Biometric identification”. Commun. ACM, vol.43, no.2, pp.91-98, (2000) 601 [3] R.Chellappa, C. L.Wilson, and S.Sirohey: ”Human and machine recognition of faces: A survey”. Proc.IEEE, vol.83, no.5, pp705-740, (1995) 601 [4] S.Akamatsu: ”Computer Recognition of Human Face”. IEICE Trans. Vol.D-II, Vol.J80. No.8, pp.2031-2046, (1997) in Japanese 601, 602 [5] O.Delloye, M.Kaneko and H.Harashima : ”Handling of Facial Features Using Face Space”. IEICE Trans. vol-J80A, No.8, pp.1332-1336, (1997) in Japanese 602 [6] H.Takimoto, Y,Mitsukuta, M.Fukumi, and N.Akamatsu: ”A Design of Face Detection System Using the GA and the Simple PCA”. Proc. of ICONIP’02, Vol.4, pp.2069-2073, November (2002) 602 [7] M.Partridge and R.Calvo: ”Fast dimentionality reduction and simple PCA”, IDA, Vol.2(3), pp.292-298, (1989) 602 [8] S.Kawato, N.Tetsutani: ”Circle-Frequency Filter and its Application”, Proc. Int. Workshop on Advanced Image Technology, pp.217-222, Feb (2001) 603

608

Hironori Takimoto et al.

[9] T.Kawaguchi, D.Hikada, and M.Rizon: ”Detection of the eyes from human faces by hough transform and separability filter”, Proc. of ICIP 2000, pp.49-52, (2000) 603 [10] S. Furui: ”Digital Sound Processing”, TOKAI UNIVERSITY PRESS, Tokyo, (1985) 606

A Feature Extraction of the EEG Using the Factor Analysis and Neural Networks Shin-ichi Ito1 , Yasue Mitsukura2 , Minoru Fukumi1 , and Norio Akamatsu1 1

University of Tokushima 2-1, Minami-Josanjima, Tokushima, 770-8506, Japan {itwo,fukumi,akamatsu}@is.tokushima-u.ac.jp 2 Okayama University 3-1-1, Tsushima-naka, Okayama, 700-8530, Japan [email protected]

Abstract. It is often known that an EEG has the personal characteristic. However, there are no researches to achieve the considering of the personal characteristic. Then, the analyzed frequency components of the EEG have that the frequency components in which characteristics are contained significantly, and that not. Moreover, these combinations have the human equation. We think that these combinations are the personal characteristics frequency components of the EEG. In this paper, the EEG analysis method by using the GA, the FA, and the NN is proposed. The GA is used for selecting the personal characteristics frequency compnents. The FA is used for extracting the characteristics data of the EEG. The NN is used for estimating extracted the characteristics data of the EEG. Finally, in order to show the effectiveness of the proposed method, classifying the EEG pattern does computer simulations. The EEG pattern is 4 conditions, which are listening to Rock music, Schmaltzy Japanese ballad music, Healing music, and Classical music. The result, in the case of not using the personal characteristics frequency components, gave over 80 % accuracy. Then the result, in the case of using the personal characteristics frequency components, gave over 95 % accuracy. This result of our experiment shows the effectiveness of the proposed method.

1

Introduction

Recently in the world, the research of the electroencephalogram (EEG) interface is being done, because it has the possibility to realize an interface that can be operated without special knowledge and technology by using the EEG as a means of the interface. The EEG is activities of electric potential inside the brain recorded from the top of the scalp. The EEG is a time series signal to change by the internal factor, which is human’s thinking and conditions, or the outside stimulus those are the light and sound [1]-[4]. Moreover, the EEG is the time series signals that more than one factor was intricately intertwined, and the EEG is different by the measurement points of the EEG. Therefore, taking V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 609–616, 2003. c Springer-Verlag Berlin Heidelberg 2003


610

Shin-ichi Ito et al.

account of these problem, the EEG analysis method and the measurement point of the EEG must be taken into consideration. In this paper, taking account of the EEG interface, we propose the method, which is focused on four points of the following. First of all, the EEG is analyzed by the information of one measurement point because we assume that the information of some measurement points is unfitted in using the EEG interface [4],[5]. Second, the Genetic Algorithms (GA) is used for the EEG analysis. There are frequency components, which characteristics are contained significantly and/or much unnecessary information, of the EEG. Then those frequency components, which characteristics are contained significantly, have human equation. We see frequency components, which characteristics are contained significantly, as the personal characteristics frequency components. So, the GA is used for specifying the personal characteristics frequency components. Moreover, the feature choosing method using GA is already proposed, and the validity is also reported [6]-[8]. Third, the Factor Analysis (FA) is used for the EEG analysis. Because the EEG is the time series signals that more than one factor was intricately intertwined and the EEG contains much noise, the EEG has the information, which is difficult to obtain from direct observation data. Therefore, taking account of the correlation of the time transition of each frequency spectrum of the EEG, the latent structure, which explains that correlation, is analyzed by using the FA. We have attempted to extract the characteristics data of the EEG by using the FA. Finally, the Neural Networks (NN) is used for the EEG analysis. The NN is possible that the non-line relations, which more than one factor was intricately intertwined, are expressed. The NN is used for estimating extracted the characteristics data of the EEG. In other words, taking account of the EEG interface, we propose the EEG analysis method, which is using the GA and the FA and the NN. In order to show the effectiveness of the proposed method, classifying the EEG pattern does computer simulations. The EEG pattern is 4 conditions, which are listening to Rock music, Schmaltzy Japanese ballad music, Healing music, and classical music.

2

Measurement of the EEG

In this paper, as for a goal for the final of our research, the EEG control system by any music is constructed. This system uses human’s physiologic and mental effect, which a music stimulus gives to human. There is causal relationship between the EEG and the music, for instance, α waves appear by listening to Classical music, and β waves appear by listening to Rock music. This system adjusts the outputs of the music automatically by that causal relationship. Moreover, the EEG is controlled by listening to that music, which is adjusted the output of, in this system. In the constructing this system, the EEG analysis and the music analysis are absolutely imperative. The most important part of this system is that the EEG analysis. So, the EEG analysis is done in this paper.

A Feature Extraction of the EEG

2.1

611

The EEG Patterns

In this paper, as basic research of constructing the EEG control system, some genre of music is classified by the EEG. The EEG patterns are 4 conditions, which are listening to Rock music, Schmaltzy Japanese ballad music, Healing music, and Classical music. We used the following questionnaire to decide the EEG pattern because classifying the genre of music becomes easier. In addition, the questionnaire is done 20 people, who are boys and girls of twentysomething. 2.2

Measurement Conditions of the EEG

The subjects of this paper were 5 people who are 4 boys (The average age: 22.3 years old) and 1 girl (age: 23 years old). The electroencephalograph used simple electroencephalograph. The simple electroencephalograph can be measured under the practical environment. The simple electroencephalograph is made by Brain Function Research & Development Center in Japan. Measurement part place is electrode arrangement FP1 (the left frontal lobe) in international 10/20. As for the data, FFT is being done, and frequency analysis is carried out to 24Hz at intervals of 1Hz by attached analytic software. As a measuring condition, the EEG measuring is carried out in the laboratory with some noise. Then subjects wear a sensor band and headphone. Measuring time is for four minutes in each condition. In addition, the data of using frequency components are from 4Hz to 22Hz. Fig.1 shows measured points of EEG.

3

The Procedure of the Proposed Method

In this paper, we propose the EEG analysis method, which is using the GA and the FA and the NN. The GA is used for specifying the personal characteristics frequency components. The FA is used for extracting the characteristics data of the EEG. The NN is used for estimating extracted the characteristics data of the EEG. In addition, Fig.1 show the flowchart of the EEG analysis method. The EEG analysis method is as follows: First of all, the individual, which has a random chromosome, is formed as an initial group. The value of an individual chromosome is “0” or “1”. An individual chromosome copes with each frequency components (4-22Hz). When an individual chromosome is “1”, the time series data of allocated frequency component is used for the FA. When an individual chromosome is “0”, the time series data of allocated frequency component is not used for the FA. In the second, the characteristics data of the EEG is extracted by using the FA. The FA is one of the statistical methods. Then the information that each variate (frequency components) with the correlation has is summarized in small number of latent factor. The FA can be denoising, dimensional compression, latent structure analysis that structure lurks behind each variate with the correlation. In this paper, the model of the FA is cross-factor model because we assume that common factor, which responds to particular stimulus, is identified when extracted common factor is noncorrelated [9],[10]. Moreover, taking

612

Shin-ichi Ito et al. Initialization Factor analysis Fitness Fitness < 0.99 Iteration < 50

End

Elite preservation Crossover Mutation Multiplication

Fig. 1. Flowchart account of measurement conditions of the EEG, the first common factor shows “the EEG changing by the music”, we think. The common factor is extracted by using principal factor analysis method. In this paper, the characteristics data of the EEG is the data of first factor loading. Third, the characteristics data of the EEG is estimated by the NN. The NN, which is 3-layer class pattern, is used for learning the characteristics data of the EEG, and the EEG pattern classification. Then back-propagation (BP) method is used for the way of learning the characteristics data of the EEG, and leave-one-out cross-validation (LOOCV) method is used for test method. Finally, the fitness of each individual is calculated by using the result (recognition accuracy) of the NN. When established termination conditions are not satisfied, next generation individuals are generated by genetic manipulations. Genetic manipulations are selection, crossover, mutation, multiplication, and a local solutions, which are used for not selection a local solution. In addition, when termination conditions are satisfied, the personal characteristics frequency components are allocated “1” of an individual chromosome, which is the highest fitness value. 3.1

The Fitness Value

First of all, the characteristics data is learned by using the NN. Mean squared error is calculated after learning using the learning data. Second, the recognition

A Feature Extraction of the EEG

1

0 1 1 0 1 0

1

1 0 0 1 1 0

613

genetic manipulation point

conjunction

disjunction 1

0 1 1 1 1 0

1

0 1 0 0 1 0

1

1 0 1 1 1 0

1

1 0 0 0 1 0

Fig. 2. Genetic manipulation based on the logical operations accuracy of an individual is calculated by using the NN. Finally, the fitness accuracy is calculated. The fitness value is as follows: f itness = (Recog −

N umber of 1 s 2 ) − Err N umData ∗ Length

Err Mean squared error Recog The result of NN (the recognition accuracy) Length The length of gene NumData The number of the test data pattern Number of 1’s The numben of 1’s in an gene 3.2

Genetic Manipulations

Genetic manipulations are selection, crossover, mutation, multiplication, and the logical operations. Selection is the elite preservation strategy. Crossover is one-point crossover and using the logical operations, which are shown Fig.4. Multiplication is using the logical operations. The logical operations are as follows: First of all, the conjunction (Elite conjunction) of the individuals, which are selected elite, is calculated. Second, the conjunction (Not elite conjunction) of the individuals, which are not selected elite, is calculated. Finally, the gene (0 or 1) of Not elite conjunction is inverted. Then the conjunction and the disjunction of Elite conjunction and inverted Not elite conjunction are calculated.

4

Computer Simulations

In this simulation, the EEG pattern is classified. The EEG pattern is 4 conditions, which are listening to Rock music, Schmaltzy Japanese ballad music,

614

Shin-ichi Ito et al.

Table 1. The parameter of GA The number of generation 50 The number of individual 50 The length of chromosome 19

Table 2. The parameter of NN The maximum number of input layer unit 19 The number of layer unit 5 The number of output layer unit 4 The number of learning 10000 The step size 0.01

Healing music, and Classical music. In addition, the subjects of this paper were 5 people who are 4 boys (The average age: 22.3 years old) and 1 girl (age: 23 years old). Furthermore, the parameter of the GA is shown in the Table 1. Then the parameter of the NN is shown in the Table 2. In addition, the step size and the number of hidden layer in parameter of the NN are being found experientially. Compared with using the frequency components (4-22Hz) in this paper. The recognition accuracy, in the case of using the personal characteristic frequency components, and in the case of using the frequency components (4-22Hz) is shown Fig.3. In addition, the selected the personal frequency components are shown in the Table 3. The result in the case of using the frequency components (4-22Hz) gave over 80 % accuracy. This result suggests that extracting the characteristics data of the EEG by using the FA is significant in the EEG analysis. Then the result in the case of using the personal frequency components gave over 95 % accuracy. This result suggests that specifying the personal frequency components is significant in the EEG analysis.

5

Conclusions

Taking account of using the EEG interface, the simple electroencephalograph of the band type is used for measuring the EEG. We propose the EEG analysis

Table 3. Selected frequency components by the GA subjact1 subject2 subject3 subject4 subject5

The extracted frequency components 6,8,10,12,19-21Hz 4,6-8,10,17,18Hz 6,8,10-12,22Hz 6-8,12,17,18,20Hz 5,6,8,12-14,16,17,21Hz

A Feature Extraction of the EEG

615

Fig. 3. The recognition accuracy, in the case of using the personal characteristics frequency components, and in the case of using the frequency components (422Hz)

method, which is using the GA and the FA and the NN. The GA is used for specifying the personal characteristics frequency components. The FA is used for extracting the characteristics data of the EEG. The NN is used for estimating extracted the characteristics data of the EEG. In order to show the effectiveness of the proposed method, classifying the EEG pattern does computer simulations. The EEG pattern is 4 conditions, which are listening to Rock music, Schmaltzy Japanese ballad music, healing music, and classical music. The result, in the case of not using the personal characteristics frequency components, gave over 80 % accuracy. Then the result, in the case of using the personal characteristics frequency components, gave over 95 % accuracy. The result of our experiment shows the effectiveness of the proposed method. In this paper, in such a way that classifying the EEG pattern is easy, we used the following questionnaire to decide the EEG pattern. Future studies will focus on boosting the number of classifying the EEG pattern, which is listening to the genre of music. Moreover, we will analyze the music.

References [1] O. Fukuda. T. Tsuji. and M. Kaneko. Pattern Classification of a Time-Series EEG Signal Using a Neural Network¨ uCIEICE (D-II)¨ uCvol.J80-D-II¨ uCNo.7¨ uCpp.18961903¨ uC(1997) 609 [2] H. Tanaka. and H. Ide. Intention Transmitting by the Single-Trial MRCP Analysis¨ uCT.IEE Japan¨ uCvol.122-C¨ uCNo.5¨ uC(2002) [3] S. Yamada. Improvement and Evolution of an EEG Keyboard Input Speed¨ uCIEICE (A)¨ uCvol.J79-A¨ uCNo.2¨ uCpp.329-336¨ uC(1996) [4] T. Shimada. T. Shina. and Y. Saito. Auto-Detection of Characteristics of Sleep EEG Intergrating Multi Channel Information by Neural Networks and Fuzzy Rule¨ uCIEICE (D-II)¨ uCvol.J81-D-II¨ uCNo.7¨ uCpp.1689-1698¨ uC(1998) 609, 610 [5] S. Tasaki. T. Igasaki. N. Murayama. and H. Koga. Relationship between Biological Signals and Subjective Estimation While Humans Listen to Sounds¨ uCT.IEE Japan u ¨Cvol.122-C¨ uCNo.9¨ uCpp.1632-1638¨ uC(2002) 610

616

Shin-ichi Ito et al.

[6] S. Ito. Y. Mitsukura. M. Fukumi. and N. Akamatsu. Neuro Rainfall Forecast with Data Mining by Real-Coded Genetical Preprocessing¨ uCT.IEE Japan, Vol.123-C, No.4, pp.817–822, (2003) 610 [7] M. Fukumi. and S. Omatsu. Designing an Architecture of a Neural Network for Coin Recognition by a Genetic Algorithm¨ uCT.IEE Japan¨ uCvol.113D¨ uCNo.12¨ uCpp.1403-1409¨ uC(1993) [8] M. Fukumi. and S. Omatsu. Designing a neural network by a genetic algorithm with partial fitness¨ uCProc.of Int¨ uCConf¨ uCNeural Netwirks¨ uCpp.18341838¨ uC(1995) 610 [9] L. R. Tuckey. and R. C. MacCallum. Exploratory Factor Analysis, (1997) 611 [10] A. L. Comrey. and H. B. Lee. A First Course in Factor Analysis 611

A Neural Network Approach to Color Image Classification Masayuki Shinmoto1, Yasue Mitsukura2, Minoru Fukumi2 and Norio Akamatsu2 1 Graduate

School of Engineering University of Tokushima 2-1 Minami-josanjima-cho, Tokushima, 770-8506, Japan [email protected] 2 Faculty of Engineering, The University of Tokushima 2-1 Minami-josanjima-cho, Tokushima, 770-8506, Japan {mitsu,fukumi,akamatsu}@is.tokushima-u.ac.jp

This paper presents a method for image classification by neural networks which uses characteristic data extracted from images. In order to extract characteristic data, image pixels are divided by a clustering method on YCrCb 3-dimensionl-color space and processed by labeling to select domains. The information extracted from the domains is characteristic data (color information, position information and area information) of the image. Another characteristic data, which is extracted by Wavelet transform, is added to the feature and a comparative experiment is conducted. Finally the validity of this technique is verified by means of computer simulations.

1

Introduction

In recent years, the Internet spreads rapidly and is widely used by various people. Much information from different sources has been accumulated over the Internet. In the viewpoint of these facts, there is an enormous quantity of images on the World Wide Web. However, those images are preserved in various Web sites, and there exist many kinds of image types. Therefore, it is difficult and consumes time to search an image, which people requires [1]. A search system for an image file is then needed. An image search system has been already used in practical use. Google (http://www.google.co.jp/) offers us an image search system. But, this system searches Web page and filename that include a target word, and retrieve image files that are found from those as a result. Search results depend on information of text key words. Such a method cannot find target images if a filename or a caption don't include key words. In general, an image has plural information. When we see a certain picture, some people feel that this includes a mountain, others feel that this includes the sky. Since a filename and a caption are named by someone, they imply his or her subjectivity. Therefore a text based image search technique doesn't work effectively in many cases. Image classification into some categories is important as preprocessV. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 617-622, 2003.  Springer-Verlag Berlin Heidelberg 2003

618

Masayuki Shinmoto et al.

ing for keywords extraction. The keywords can be easily extracted from a group of similar images. In this study, images are therefore divided into categories for keyword extraction. In this study, characteristic data is extracted from an image directly, and classification of rough categories using neural networks carried out by using it. Images utilized in this paper are scenery images (Fig.1), person images (Fig.2), buildings images (Fig. 3), illustrations images (Fig.4), and flower images (Fig.5).

2

Characteristic Data

2.1

YCrCb Transforms

At first, a target expressed in RGB color system is transformed into YCrCb color system using YCrCb transform. Y represents intensity. Cr and Cb represent color difference. Fig.6 are results obtained by the YCrCb transform in which the sample image in Fig.1 is uccccccsed.

Fig. 1. A sample of scenery images

Fig. 2. A sample of person images

Fig. 3. A sample of building images

Fig. 4. A sample of illustration images

Fig. 5. A sample of flower images

A Neural Network Approach to Color Image Classification

Parameter Y

Parameter Cr

619

Parameter Cb

Fig. 6. Result of YCrCb transform

Fig. 7. A result obtained using the K-means method

2.2

K-means Method

To quantize color information, pixels distributed on the YCrCb 3-dimensional space are classified into some clusters. This paper uses the K-means method for clustering [3][4][5]. Fig.7 is a result obtained by the K-means method for the image in Fig.1. 2.3

Labeling

The image quantized is processed by labeling transaction to divide pixels into some domains with the same value. In this paper, domains obtained as a result of labeling are 3 domains selected in order of large area (Fig.8) and then characteristic data are extracted. The number of characteristic data used in experiments are fifteen, average values of R, G, B, Y, Cr and Cb in a domain, standard deviation of R, G, B, Y, Cr and Cb in a domain, center of gravity (x and y coordinates), area of domain.

The largest area

The second largest area Fig. 8. A result of Labeling

The third largest area

620

Masayuki Shinmoto et al.

Fig. 9. Results obtained by Wavelet transform

2.4

Fig. 10. Distribution of frequency

Wavelet Transforms

The image is expressed by Wavelet transform to detect edges (Fig.9). In signal processing, Wavelet transform is equivalent to use low-pass filter and High-pass filter. The image is decomposed by low frequency and high frequencies (Fig.10). The low frequency (LL) means rough image. The high frequencies (LH, HL, HH) mean edges. Characteristic data of edges extracted from high frequencies, which correspond selected domains, is average of frequencies and standard deviation of frequencies.

3

Experiment

3.1

Experimental Conditions

The classification experiments are performed using characteristic data. A neural network as shown in Fig.11 is used in the experiments as mentioned above. The learning algorithm is the back propagation method [7]. Characteristic data are fed to input units. The number of characteristic data extracted from one domain is seventeen, each average of RGB and YCrCb, each standard deviation of RGB and YCrCb, x and y coordinates, area of domain, average of frequencies and standard deviation of frequencies. In this paper, five experiments are carried out. Each experimental condition has different categories as follows. 1. 2. 3. 4. 5.

Scenery, Person, Building, Illustration, Flower Scenery, Building, Illustration, Flower Illustration, Others (Scenery, Building, Flower) Scenery, Building, Flower (Scenery and Flower), Building

The number of output units is related for the number of experimental condition's categories.

A Neural Network Approach to Color Image Classification

621

Fig. 11. A neural network used in computer simulations

3.2

Experimental Result

Table.1 shows result of experiment 1. Total accuracy is more than 70%. In particular, illustration image's accuracy indicates a high value. This reason is that illustration images used in this study don't have various colors. Therefore color information's standard deviation is effective in this case. However, Person's image and Flower's image accuracy is lower value which is less than 60%. However, person image can be classified by skin color detection. Therefore, we evaluate the case four categories in experiment 2 (Table.2). In Table.2, Each category's accuracy and total accuracy are improved. However, Flower image's accuracy is lower than others, yet. This reason is that some flower images have similar characteristic data extracted from scenery images. To classify images into two categories, experiment 3 is carried out (Table.3). Others image's accuracy is nearly 100%. Illustration image, which is wrong in experiment 1 to 3, have similarity. Therefore, this misclassification depends on image kind, which differs completely from other illustration images. And also, Table.3 shows illustration image can be classified. Accurately Tables 4 and 5 show results of experiments, which don't used Illustration images. Accuracy is improved in Tables. Therefore, this approach is effective in classification. Table 1. Result of experiment 1

Table 2. Result of experiment 2

622

Masayuki Shinmoto et al. Table 3. Result of experiment 3

Table 4. Result of experiment 4

Table 5. Result of experiment 5

Scenery and Flower Building Total

4

91.45% 84.20% 89.03%

Conclusion

In this paper, characteristic data is extracted from images and is an input to neural networks in order that images are classified. In classification experiments, eighty percent or more accuracy is achieved. In the future, the number of images used for a classification is increased, and change of classification rate is examined. In addition, techniques that extract characteristic data from small domains is studied. This research was partially supported by the Tele communications Advancement Foundation and Grant-in-Aid for scientific Research© (No.1368044) in MEXT.

References [1] [2] [3] [4] [5] [6] [7]

James Z. Wang, Integrated Region-Based Image Retrieval, 2001, Kluwer Academic Publishers Mikio Takagi, Picture Analysis Handbook, 1991, University of Tokyo Press, in Japanese Sadaaki Miyamoto, A Guide to Cluster Analysis, 1999, Morikita Publishing, in Japanese Takeshi Agui and Tomoharu Nagao, Processing and Recognition of a Picture, 1992, Syokodo, in Japanese Hisoshi Ozaki and Keiji Taniguchi, Image Processing, 1988, Kyoritsu Publishing Company, in Japanese Hideyuki Tanaka and Japan Industry Engineering Center, A Guide to Computer Image Processing, 1985, Soken Publishing Company, in Japanese J. Dayhoff, Neural Network Architecture, 1990, Van Nostrand Reinhold

Recognition of EMG Signal Patterns by Neural Networks Yuji Matsumura1, Yasue Mitsukura2, Minoru Fukumi2 Norio Akamatsu2, and Fumiaki Takeda3 1

Graduate School of Engineering University of Tokushima Tokushima, 770-8506, Japan [email protected] 2 Faculty of Engineering University of Tokushima Tokushima, 770-8506, Japan {mitsu,fukumi,akamatsu}@is.tokushima-u.ac.jp 3 Graduate School of Engineering Kochi University of Technology Kochi, 782-8502, Japan [email protected]

Abstract. This paper tries to recognize EMG signals by using neural networks. The electrodes under the dry state are attached to wrists and then EMG is measured. These EMG signals are classified into seven categories, such as neutral, up and down, right and left, wrist to inside, wrist to outside by using a neural network. The NN learns FFT spectra to classify them. Moreover, we structuralized NN for improvement of the network. It is shown that our approach is effective to classify the EMG signals by means of computer simulations.

1

Introduction

Conventionally, there have been some pointing devices like a mouse, a data glove, a data suit, a motion capture and so on, which can measure data such as position, direction, and power. However these devices are not comfortable and not easy to use because of their form and weight. For example, the mouse is too large, and such machines and tools are too heavy to carry. Recently, information terminals such as a mobile phone, have been widely used. According to this, industrial standard of radio communication such as Bluetooth has been established. As a result, it would be possible to combine and to perform various interface. For example, a portable telephone is used on manners mode, a music player volume is easily regulated and so on, because of this standard. Now a days, the device which have various operation of portable machines and tools and can control networks (call “total operation device” for short) has not been provided yet. Therefore, V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 623-630, 2003.  Springer-Verlag Berlin Heidelberg 2003

624

Yuji Matsumura et al.

we investigate ElectroMyoGram (EMG) [1] which is a signal generated from a living body according to voluntary exercise of subjects. Moreover, the wristwatch type is preferable in the viewpoint of operationality. Therefore, for construction of recognition system of EMG patterns, we analyze the values of EMG from the wrist and perform recognition experiments using the Fast Fourier Transform (FFT) [2] and a neural network (NN) [3].

2

EMG Recognition System

A flowchart of Construction of a recognition system of EMG patterns which we propose in this paper is shown in Fig. 1. The proposed system is composed of an input, a signal processing and learning-evaluation parts. First, time series data for EMG is measured by electrodes in the input part. Second, this data is amplified and A/D transform is performed in the signal processing part. Next, this amplified data is converted to Fourier power spectra. Finally, we evaluate various data in the learning-evaluation part. Fig. 2 shows the outside feature of the proposed system. 2.1

Input Part

There are an insertion-type electrode and a surface-type electrode to measure EMG. The insertion-type electrode can measure EMG from a specific muscle because of sticking muscle directly. However, it is not preferable for the subject and in danger of sick contagion. As compared with this, the surface-type electrode is preferable for the subject and not in danger of sick contagion and pain. On the one hand, it is difficult to divide action style and motor control because EMG is measured as a signal which is a mixed plural potential of muscle fiber. In this paper, we use the surface-type electrode as an input device because we try to recognize a lot of behavior from the various muscles. There are two types of surface electrodes. One is a wet type and the other is a dry type. From the viewpoint of easiness of operation, we adopt the dry type one. Fig. 3 illustrates the device for measuring EMG of the wrist. We measure 7 patterns of the wrist behavior using this device. The wrist behavior is neutral, up, down, bend to right, bend to left, twist of inside, twist of outside. Also, Fig. 4 shows the patterns of the wrist movement.

Fig. 1. A flowchart of the proposed system

Fig. 2. The outside of the proposed system

Recognition of EMG Signal Patterns by Neural Networks

625

Fig. 3. The device of measuring

(a)Neutral

(b) Up

(e) Bend to left

(c) Down

(f) Twist to inside

(d) Bend to right

(g) Twist to outside

Fig. 4. The patterns of the wrist

2.2

Signal Processing Part

We measure EMG with the surface-type electrode. It has been reported that the main frequency of EMG is generated between a few Hz and 2 kHz [4],[5]. We extract frequency components between 70 Hz and 2 kHz to avoid the influence of the noise by the electromagnetic wave which is the commercial frequency of 60 Hz. As a result, the amplifier reduces the influence of the electromagnetic wave noise. The specification of the amplifier is shown in Table 1. Table 1. The specification of the amplifier

Low Cut-off Frequency High Cut-off Frequency Gain Output Voltage Input Voltage

70Hz (6th filter) 2kHz (6th filter) 2000 times (65dB) ±5V ±2.5mV

626

Yuji Matsumura et al.

Fig. 5. A method to yield FFT spectra

2.3

Data Transform Part

The EMG output program yields 20,480 time series data every channel for 2,048 milliseconds. The values obtained by performing 1,024-points FFT from 10,000th data point every channel and by doing FFT moving data window every 512 data point are used for learning data. Fig. 5 shows a schematic to yield of learning data. 2.4

Learning-Evaluation Part

NN is trained by using the FFT data and is evaluated in learning-evaluation part. In this paper, three methods are used in learning and evaluating. The methods used are the Back-Propagation (BP) network, the Learning Vector Quantization (LVQ) network and the BP&LVQ network. (1) The BP network The BP network is a three-layered feed-forward network that is composed of an input layer, a middle layer, and an output layer. Since the error between an output and a teacher signal are updated in order of an output layer, the middle layer, and the input layer in each weight w , it is also called the Back Propagation method. (2) The LVQ network The LVQ network is a method that changes Self-Organizing Map (SOM) by Kohonen [6] to learning with teacher data. Vector Quantization performs data compression approximating representative vectors. Here, the LVQ algorithm is summarized as follows: STEP1: Random values are set as initial weight values. STEP2: An input vector { X = ( x1 , x 2 ,L , x n )} is set in the input layer. STEP3: The distance between the weight of j-th neuron and the input vector is calculated as follows:

dj =

N

∑ (x i =1

− w ji )

2

i

(1)

Recognition of EMG Signal Patterns by Neural Networks

627

STEP4: The weight vector w ji that has the smallest d j is selected, and is renewed according to the following equation.

w ji (k ) + α (x i − w ji )  : The result of recognitio n is true w ji (k + 1) =  w ji (k ) − α (x i − w ji ) : The result of recognitio n is false  where

α

(2)

denotes a learning late and k is a time index.

Using this operation, Bayes discrimination border that is theoretically the most suitable in pattern classification is formed. Therefore, this method is useful for pattern recognition problem in many cases [7]. The LVQ network basically has three methods called LVQ 1, LVQ 2 and LVQ3. In this paper, the method using the LVQ 1 and 3 is utilized. (3) The BP&LVQ network There are the problems that BP has been applied to many pattern recognition but it requires a long time for learning and LVQ doesn't require a long time for learning but it doesn't have been applied to many pattern recognition. Therefore, we suggest BP&LVQ that mixed with good point of their networks for structuring of network. BP&LVQ's algorithm is shown below. Also, Fig. 6 shows construction of the BP&LVQ network. STEP 1: Neutral and outside are regarded as the same pattern because of resemblance. STEP 2: 6 patterns learning (up, down, left, right, inside, neutral & outside) is performed using BP. STEP 3: Concurrently, 2 patterns learning (neutral, outside) is performed using LVQ.

Fig. 6. BP&LVQ network

628

Yuji Matsumura et al.

STEP 4: If both of the learning is got through, 6 patterns recognition (major division) is performed by BP. STEP 5: Finally, using data that ignited neutral & outside and that appertained neutral & outside in BP, 2 patterns recognition (minor division) is performed by LVQ.

3

Recognition Experiments

In this paper, we performed experiments with three subjects (A:male; at age 30, B:male; at age 22, C:male; at age 21). They keep holding their right arm and grasp their hand a little. In this posture, behavior of the wrist is neutral, up, down, bend to right, bend to left twist to inside, twist to outside with bending their arm by 90 degrees. While the subjects keep this posture, we measure EMG. The 7 patterns of arm behavior are recognized by NN. We regard the 7 patterns as 1 set. The learning data is 10 sets out of 30 sets and the evaluation data is the rest 20 sets. We use the frequency under 1kHz for the input units of NN because we analyzed the frequency components by the BPWF network in advance [8]. 3.1

Experimental Conditions

Table 2 shows experimental conditions of the BP network, Table 3 shows experimental conditions of the LVQ network and Table 4 shows experimental conditions of the BP&LVQ. In Tables, CL means Competition Layer, IL means Input Layer, ML means Middle Layer, OL means Output Layer. Table 2. Experimental conditions of BP

No. of units on IL No. of units on ML No. of units on OL No. of Learning Data No. of Evaluation Data Maximum Learning No.

404 20 7 70 140 20,000

Table 3. Experimental conditions of LVQ

No. of units on IL No. of units on CL Category of Class No. of Learning Data No. of Evaluation Data Learning No. of the LVQ1 Learning No. of the LVQ3

404 49 (7×7) 7 70 140 1,000 1,000

Recognition of EMG Signal Patterns by Neural Networks

629

Table 4. Experimental conditions of BP&LVQ

BP Part (Major Classification) No. of units on IL 404 No. of units on ML 20 No. of units on OL 6 No. of Learning Data 70 No.of Evaluation Data 140 Maximum Learning No. 20,000 LVQ Part (Minor Classification) No. of units on CL 4 (2×2) Category of Class 2 No. of Learning Data 20 No. of Evaluation Data Variable Learning No. of the LVQ1 1,000 Learning No. of the LVQ 3 1,000

3.2

Experimental Results

Table 5 shows the experimental results. According to Table 5, we can obtain the rate of accurate recognition with BP&LVQ higher than BP and LVQ, which combined both the methods in, subject A and C. As a result, we consider that structuring of a network will become the one method of improving the recognition accuracy. But, between BP&LVQ's result and BP's result are almost the same in subject B. As this reason, I thought that it became fuzzy data because the noise may have been mixed in a part of learning and evaluation data of the subject B. Table 5. Experimental results Recognition accuracy (%)

Subject A BP LVQ Up Right Down Left Neutral Outside Inside Total

77.7 81.7 84.0 94.3 75.0 73.0 74.7 80.1

90.0 75.0 76.7 81.7 96.7 78.3 28.3 74.7

BP& LVQ 90.0 83.3 85.7 97.3 95.0 70.3 60.0 83.1

Subject B BP LVQ 67.3 82.7 97.0 85.3 82.3 73.0 50.3 76.9

65.0 61.7 83.3 98.3 83.3 70.0 30.0 70.2

BP& LVQ 79.7 86.0 95.3 76.3 63.7 78.7 68.0 78.2

Subject C BP LVQ 56.7 56.7 87.0 86.0 63.3 74.7 85.0 72.9

93.3 56.7 55.0 85.0 68.3 41.7 26.7 61.0

BP& LVQ 74.7 62.3 88.7 81.0 96.7 77.0 84.0 80.6

630

Yuji Matsumura et al.

4

Conclusions

In this paper, we have examined EMG signals that are signal in a living body for recognition of wrist behavior. Moreover, we have demonstrated recognition systems by NN. According to experimental results, we have found that the EMG recognition system by using the BP&LVQ networks that combined good point of both the BP network and the LVQ network are best in recognition accuracy. We want to go forward our study to adopt Independent Component Analysis (ICA) and fuzzy network for removing the noise, and to increase the subject for construction of robust system in the next step. This research was partially supported by Tateishi Foundation and Grant-in-Aid for Scientific Research (B) (No. 15300073) in MEXT.

References [1] [2] [3] [4] [5] [6] [7] [8]

K. Hashimoto, T. Endo: “How to Inspect to Vital Function”, Japan publishing service, (1983), in Japanese J.W Cooley, J. W. Turkey: “An Algorithm for Machine Calculation of Complex Fourier Series”, 19, pp. 297-301, (1965) J. Hearts, A. Krogh, Richard. G. Palmer: “Introduction to The Theory of Neural Computation”, Addison-Wesley Publishing Company, The Advanced Book Program, 350 Bridge Park0way, Redwood City, (1991) H.G Vaughan, Jr., L. D. Costa, L. Gilden and H. Schimmel : “Identification of Sensory And Motor Components of Cerebral Activity in Simple Reaction Time Tasks”, Proc. 73rd, Conf. Amer. Psychol. Ass., 1, pp. 179-180, (1965) C. J. De Luca: “Physiology and Mathematics of Myoelectric signals”, IEEE Trans, Biomed. Eng., 26, pp. 313-325, (1979) T. Kohonen: “Self-Organization and Associative Memory”, Springer-Verlag Berlin Heidelberg New York Tokyo, (1984) T. Kohonen: “Self-Organizing Maps”, Springer-Verlag Berlin Heidelberg New York Tokyo, (1995) Y. Matsumura, M. Fukumi and N. Akamatsu:“Recognition of EMG Signal Patterns by Neural Network”, ICONIP'02Workshop Proceeding, (2002)

True Smile Recognition Using Neural Networks and Simple PCA Miyoko Nakano1 , Yasue Mitsukura2 , Minoru Fukumi2 , Norio Akamatsu2 , and Fumiko Yasukata1 1

Faculty of Nursing Fukuoka Prefectural University 4395,Ida,Tagawa, Fukuoka 825-8585 Japan {mnakano, yasukata}@fukuoka-pu.ac.jp 2 Department of Information Science and Intelligent University of Tokushima 2-1,Minami-Jyosanjima,Tokushima 770-8506 Japan {mitsu,fukumi,akamatsu}@is.tokushima-u.ac.jp

Abstract. Recently, an eigenface method by using the principal component analysis (PCA) is popular in a filed of facial expressions recognition. In this study, in order to achieve high-speed PCA, the simple principal component analysis (SPCA) is applied to compress the dimensionality of portions that constitute a face. By using Neural Networks (NN), the difference in value of cos θ between true and false (plastic) smiles is clarified and the true smile is discriminated. Finally, in order to show the effectiveness of the proposed face classification method for true or false smile, computer simulations are done with real images.

1

Introduction

In previous studies, many researchers have proposed several methods for recognition of facial expressions[1]-[4]. Most of these, however, have been classified facial expressions (e.g. anger, sadness, happiness and so on). The classification using the expression made intentionally. In particular, this study focuses on a true smiling face considering application to ’man-machine interface‘ systems. An eigenface method is popular in this research field [1],[3],[4]. In PCA, it is not easy to compute eigenvectors with a large matrix when considering the cost of calculation to apply it to time-varying processing. Therefore, SPCA method has been proposed in order to make the PCA calculation faster[5]. Neural networks (NN) are an advanced parallel system, which are excellent especially in problems related to pattern recognition[6]. In this study, therefore, NN is used to classify a true and false smile. Because, it is used in order to avoid the difficulty of threshold value determination in many dimensions. Firstly, dimensionality compression using SPCA of the three portions of the right eye, the left eye, and mouth and nose that constitutes a face is carried out. Secondly, a value of cos θ is calculated using the eigenvector and a gray scale image vector of each image pattern. By using NN, cos θ between true and false V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 631–637, 2003. c Springer-Verlag Berlin Heidelberg 2003


632

Miyoko Nakano et al.

smiles is verified and true smile is discriminated. Finally, simulation results show that this approach is effective for true smile discrimination.

2

Simple Principal Component Analysis (SPCA)

In this section, the SPCA algorithm [5] is applied to a set of vectors V ={v1 , v2 , · · · , vm }. At first, in order to make the center of gravity of this set of vector into the origin, the average vector vˆ is subtracted from all vectors. The new vector set X = {x1 , x2 , · · · , xm } is used as input vectors, which are computed as follows: m

xi = vi − vˆ;

( vˆ =

1
vj ) . m j=1

(1)

Then the column vector an is defined as connection weights between the inputs xj and output yn . The n-th weight, an is used to approximate the n-th eigenvector, αn . The output function has a value given by: T

yn = an xj .

(2)

This function which yields a positive value if an input vector xm exists in the an side and a negative value if it exists in the same half-plane as the opposite side as shown in Fig. 1. Next, a threshold function equation (3) is introduced into the output function of the equation (2).  xj if yn ≥ 0 Φ(yn , xj ) = (3) 0 otherwise In the threshold function, there are four functions as the above. This study introduces the equation (3) as a threshold function. Using the above-mentioned function, repetitive calculation of arbitrary vector an suitably given as an initial value is carried out. Therefore the vector an can approach in the same direction as αn .

Fig. 1. The simple PCA algorithm

True Smile Recognition Using Neural Networks and Simple PCA

633

Initially a0 can be set as an arbitrary vector. The sum of Φ(yn , xj ) over all samples xj , is calculated where yn is calculated with respect to the old estimated vector ak . This sum is then normalized to give the new estimate (equation (4))  j Φ1 (yn , xj ) k+1  ; yn = (akn )T xj , an =  (4)   j Φ1 (yn , xj ) where k denotes the number of times of a repetition. The value of yn is determined by akn before repeating. This calculation can modify an to the center direction of distribution. Hence, akn is converged in the same direction as the n-th eigenvector αn by these repeating (equation 4). The n-th principle component is calculable with the above operation. Next, in order to find the next principle component, the effect of the n-th principle component is removed from the dataset to find many principal component. Based on the orthogonalization of Schmidt, that is performed by subtracting the component which is parallel to the principal component from the vector xi , leaving only an orthogonal component. T

xi = xi − (an xi )an

(5)

Therefore the principle components can be obtained in order with the large rate of contribution by repeating the same operation. The SPCA algorithm is geometrically illustrated in Fig. 1.

3

Image Data

Photography conditions are set as fixed; that means the same lighting condition, subjects, and camera position. As shown in Fig.2, at first, the image data as a total of four expressions, which are composed of natural smile (smile), false (plastic) smile (false), neutral expressionless (neutral) and other expression (comparison), are taken from 16 subjects. The 4th expression was chosen, in order to take into consideration the influence on the system by blink. That is the facial expression which closed the eyes and opened the mouth lightly. In Fig.2, a false smiling face means a forced smile. It is the image of smile we acquired when a subject does not want to laugh. Next, in order to discriminate other expression from smile and false, six expression (”anger“,”disgust“,”fear“,”happiness“,”sadness“,”surprise“) are taken from 19 subjects. Recognition of true smiling face is performed using these face images. It is said that much information exists in the territory of the eye, nose, and mouth though their rate to occupy in the face is very small. Hence, these areas are extracted from face images as data (Fig.3).

4

Computer Simulations

In order to show the effectiveness of the proposed method, it was applied to true smile classification using real images. When a few data is used for computer

634

Miyoko Nakano et al.

Fig. 2. The samples of data

simulations, in order to test at sufficient accuracy, the leave-one-out method of cross validation is often performed. For example, subjects are N persons. The value of cos θ in (N − 1) subject’s image is used as training data. The remaining subject’s images are used for test data Test data was changed one by one and N simulations were performed.

Fig. 3. The procedure of extracting each portion

True Smile Recognition Using Neural Networks and Simple PCA

635

Table 1. The classification result obtained by cross validation facial expressions average (%)

4.1

true smile 87.5

false smile 93.8 90.6

Calculation of the Value of cos θ

The value of cos θ is calculated using an eigenvector and a gray scale image vector of each picture pattern. The angle is made by two vectors. It is calculated by the following equation: cos θ =

(an , xj ) , an  · xj 

(6)

where an represents the eigenvector and xj means a gray scale images vector of each picture pattern. The feature cos θ is better in recognition accuracy than the inner product of them. 4.2

Structure of NN

A NN model used for the classification in this paper is a three-layered network. BP method is adopted for learning. All eigenvectors used to yield inputs has the sum of contribution that is 80.0% or more. The values of cos θ are fed into the input units. Hence, the number of units in the input layer is 26 (In each eyes image, the 1st-7th eigenvectors are used. And in mouths and noses image, the 1st-12th eigenvectors are used.). The hidden layer has ten units and the output layer has one unit. Therefore, 1, 0 means teachers signals for a true smile and 0, 1 means a false smile. 4.3

True Smile Recognition Using SPCA

The result is shown in Table 1. The rate of an average accurate classification is 90.6%, which shows the classification of smiles can be done by using NN. In the details, the average rate of classification of ”true“ is 87.5%, and the average rate of classification of ”false“ is 93.8%. 4.4

Consideration of the ” Blink “

On the other hand, when considering application to video processing, it is important to discriminate the true smiling face and another facial expression, which is for instance a blink closed the eyes and opened the mouth lightly. The smiling faces shown in Fig. 2 resemble the blink. Therefore, the next simulations are performed to consider the influence by blink. The value of cos θ in 18 subject’s images (smile and comparison) are used as training data. The remaining

636

Miyoko Nakano et al.

Table 2. The classification result in consideration of the blink facial expressions average (%)

true smile 89.5

comparison 97.4 92.1

subject’s images are used for test data. Test data was changed one by one and 18 simulations were performed. This result is shown in Table 2. The rate of an average classification is 92.1%, which shows the classification of smiles can be done by using NN. In the details, the average rate of accurate classification of ”true“is 89.5%, and the average rate of accurate classification of ”comparison“ is 97.4%. From these results, it is rare to recognize expression which closes both eyes and opened a mouth lightly to be a true smile. Moreover, a false smile is similar to a true smile than a ”blink“. Therefore, it can decide that the truth classification of a smiling face is difficult. 4.5

Classification with the Various Expressions

A human being has the large variety of expression. In this experiment, furthermore, a classification with other expressions and a natural smile is performed. NN is used for the classification. When 1, 0 is yielded in the output layer, NN recognize it is smile. When 0, 1 is yielded , it shows false, and if 0, 0 is yielded, it shows other expressions. This simulation is done in the same way as 4.3 by using the data of 19 subjects. The expression to treat as other expressions is 7 expression of”anger“, ”disgust“,”fear“,”happiness“,”sadness“,”surprise“, ”neutral expression“. A classified result is shown in Table 3. An average classified rate was 87.1%. It was difficult to classify ”disgust“, ”fear“ and so on, with among other expressions.

5

Conclusion

An accurate recognition method of true smile is proposed in this study, and its validity is verified by computer simulations. In this method, SPCA which is effective in information compression is used for dimensionality compression of the portion that constitutes a face. The eigenvectors, which are yielded by SPCA, are used as a feature vector. Then, NN is used in order to avoid the difficulty of the threshold value determination in many dimensions. By using NN, the value

Table 3. The classification result of smile facial expressions average(%)

true smile 94.7

false smile 86.8 87.1

others 80.4

True Smile Recognition Using Neural Networks and Simple PCA

637

of cos θ between true and false smile is clarified and used to recognize true smile by using it as input data of NN. From the results, the rate of a classification using NN was 90.0% as the whole. An experimental result in consideration of the blink was 92.1%. Moreover, it was 87.1% when other expressions were taken into consideration. From these results, we can conclude that the classification accuracy of proposed method is better than conventional methods.

References [1] S.Akamastu: ”Computer Recognition of Human Face“, IEICE Trans. on, Vol.DII-J80, No.8, pp.2031-2046 (1997) in Japanese 631 [2] H.Ohta, H.Saji, and H.Nakatani: ”Recognition of Facial Expressions Using Muscle-Based Feature Models“, IEICE Trans. on, Vol.D-II-J82, No.7, pp.11291139 (1999) in Japanese [3] O.Delloye, M.Kaneko, andH.Harashima: ”Operation of the Facial Feature Using Face Space“, IEICE Trans. on, Vol.A-J80, No.8, pp.1332-1336 (1997) in Japanese 631 [4] K.Matuno, C.Lee, and S.Tsuji: ”Recognition of Facial Expressions Using Potential Net and KL Expansion“: IEICE Trans. on, Vol.D-II-J77, No.8, pp.1591-1600 (1994) in Japanese 631 [5] M.Partridge and R.Calvo: ”Fast Dimensionality Reduction and Simple PCA“, IDA, Vol.2 (3), pp.292-298 (1997) 631, 632 [6] S. Y.Kung: Digital Neural Networks, PTR Prentice-Hall (1993)

Estimation of Sea Ice Movement Using Locally Parallel Matching Model Kong Xiangwei, Chen Feishi, Liu Zhiyuan, and Zong Shaoxiang Dept. of Electronic Engineering, Dalian Univ. of Technology Dalian 116024, China [email protected] [email protected] [email protected]

Abstract. In order to estimate the velocity of sea ice movement, according to the characters of sea ice image, an effect method has been adopted: Proper candidate points are found by using interest operators, a locally parallel model for matching is constructed to analyze the disparity of sequential images, after a certain post processing, the estimation of the velocity could be got by using a orientation choosing and maximum probability decision method. The algorithm has shown good performance in practice for estimating velocity of sea ice in Bohai.

1

Introduction

The sea ice is very harmful to the marine transport and production. So the study of the prediction of sea ice and the corresponding measure is very important. Sea ice inspection is an important part of sea ice management. There are many methods to inspect sea ice, one of which is using icebreaker. The artificial way of inspecting sea ice movement in ice breaker which is widely used at present is not accurate enough. And it can't work continually for a long time. In this paper we adopt a series of methods to estimate the velocity in real time by processing the digital video signal of sea ice movement, which is got by a camera bound on the ice breaker. To estimate the speed of sea ice from the digital video signal, we present a method that can use a sequence of video to find the most similar area of them, and then gain the velocity from the difference of area's position in the two frames. So, our research focus on finding a matching method which performances in real-time, stably and suitably for sea ice's images. Usually image matching includes template matching, object matching and dynamic pattern matching etc. Because the object matching has the advantages of low computation, high efficiency and precision, the interesting object are used widely for matching. In this paper, a candidate point matching method is adopted. Candidate point matching must solve two problems: One problem is how to select the candidate points? Not all the points can be used as candidate points, such as that V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 638-645, 2003.  Springer-Verlag Berlin Heidelberg 2003

Estimation of Sea Ice Movement Using Locally Parallel Matching Model

639

brightness in some region may not be obvious, or some points are visible in current frame, but invisible in the next frame. Therefore, we must take those points which can be distinguished from their neighborhoods. Interest operator [2] is effective in choosing candidate points. In section 2, we will describe the method in detail. The other problem is how to confirm which matching is correct. Many researches have adopted correlation matching or minimum error square matching as the similar measurement. Ordinarily, in the current frame, the region which has obvious features is chosen as template. In the next frame a full search has been taken to find the most similar one according to the above measurement. However, this method has two disadvantages: (1) Sometimes, proper matching may not be the maximum similarity one as a result of noise and error. (2) A full research requires heavy computation. One of the improvement methods is to increase the sensitivity of similarity measurement. Levine, O'Handley, Yagi[3] adopt an adaptive correlation window. The size of the window depends on the variance of the region surrounding the point. Mori, Kidode, Asada[4] use a Gaussian-weighted correlation window to minimize the errors due to distortion of the extremities. In limiting search area, several strategies have been proposed. Nevatia[5] uses a series of progressive views to constrain disparity to small values. Another method is to use a coarse search to approximately locate the matching points, then do a fine search to locate it more accurately. However, all the above methods assume the camera is fixed. If the camera moves, the movement orientation and the shape of the characteristic window will change relatively. So, additional measures must be taken to eliminate the effect, which need great computation. In this article, a locally parallel and wholly serial model contained of a relaxation labeling algorithm[1] is adopted to compute the disparity between images and get the velocity of the sea ice movement. It doesn't need the precise information of the camera. The change of observed region and candidate points moving orientation caused by camera rotation will not have effect on the result of the algorithm. In section 3 we will describe this model particularly. After getting the matching couple points in continual frames, we present an orientation choosing and maximum probability method to estimate the velocity of the sea ice movement in section 4. In section 5, we will give the experiment results and conclusion.

2

Choosing Candidate Points

For inspection sea ice movement, we bound a camera on a ice breaker to get the sequence video. Sea ice video has follow characters: (1) The intensity in views is smooth; (2) The grey level of these image change slightly. In order to achieve a good matching result we try to select the points which are the centers of highly variable areas and furthermore the variance should be high in all four directions. The goal can be achieved by an interest operator [2].

640

Kong Xiangwei et al.

First, choose a certain region (here the window size is 5×5) and compute the difference value of all pixels along four directions according to equation(1).

(1)

Then we can get the summation of squares of the differences along four directions by equation (2). Sx =

∑∇

f (i, j )

∑∇

f (i, j )

x ( i , j )∈SelectedArea

Sy =

y ( i , j )∈SelectedArea

S 45 =

∑∇

45 ( i , j )∈SelectedArea

S x135 =

∑∇

f (i, j )

135 ( i , j )∈SelectedArea

(2)

f (i, j )

Among the four above direction measures, choose the minimum one as the initial value of interest operator of the center point.

I 0 (i , j ) = min{ S x , S y , S 45 , S 135 }

(3)

After computing the initial values of all pixels in one frame, we can get the final value of interest operator following the rule below: 1

I o (i , j )> I 0 ( m,n )

0

Other

I n (i. j ) = {

( m , n )∈N 8 ( i , j ), m ≠ i , n ≠ j

(4)

Choosing Rule: Only the point (i, j) whose final value I n (i . j ) equals one can be used as a candidate point for matching. In fact the pixels of sea ice image always changes slightly in gray. So the point which has a local maximum, however, small initial value may be used as candidate point. It will make an error final result. To avoid this situation we set a threshold based on such points to ensure the initial value to be a big value. In our estimating the velocity of sea ice, the threshold is selected by computing the maximum initial value of a sea ice background image. Fig. 1(a) and (b) are two adjacent images. Fig.1(c) and (d) are the final results in which the candidate points are displayed

Estimation of Sea Ice Movement Using Locally Parallel Matching Model

(a) Image 1

(b) Image 2

(c) Candidate points for image1

(d) Candidate points for image2

641

Fig. 1. Candidate points for two adjacent images

3

A Locally Parallel Model for Matching Candidate Points

After finding the two sets of candidate points, we would like to construct a set of matches. Because some points may be occluded, shadowed, or not visible in the next image for some reason, an initial set of possible matches are constructed by pairing each candidate point from the current image with each candidate from the next image. We regard the set of matches as a collection of “nodes” { ai } . Associated with each node ai is coordinate (x i , y i ) in image 1 and a set of labels L i which represents possible disparity that may be assigned to the point. Each label in L i is either a disparity vector (l x , l y ) ( l x ∈ [ − r , r ], l y ∈ [ − r , r ] , r is the maximum detectable disparity) or a distinguished label l * denoting “undefined disparity”. Each possible is assigned an initial likelihood using (5), (6), (7) and (8). wi (l ) =

1 l ≠ l* 1 + c * s i (l )

(5)

642

Kong Xiangwei et al.

p i0 ( l * ) = 1 − max* ( w i ( l ))

(6)

p i0 ( l ) = p i ( l | i ) * (1 − p i0 ( l * )), l ≠ l *

(7)

l≠l

pi (l | i) =

wi (l ) ∑ wi (l ' )

(8)

'

l ≠l *

si (l ) is the sum of the squares of the differences between a small window from image 1 centered on (x i , y i ) and a window from image 2 centered of ( xi + l x , yi + l y ) . c is a constant. After constructing the initial set of possible matches, we iteratively refine these likelihood using (9), (10), (11).

qik (l ) =

^ k +l

pi

^ k +l

pi

∑[ ∑p

j∃ai l '∃ nearai ||l −l ' ||≤Θ j ≠i

k j

(l ' )], l ≠ l *

(l ) = pik (l ) * ( A + B * qik (l )), l ≠ l *

(9)

(10)

(l * ) = pik (l * ) ^ k +l

p

k +l i

(l ) =

pi

(l )

^ k +l

∑ pi

(l ' )

(11)

l 'inLi

A and B are constants. Labels are considered consistent if they satisfy the following relation: || l − l ' ||≤ Θ (12) A node a j may be considered near ai if max(| xi − x j |, | yi − y j |) ≤ R

(13)

the probability of the label l * is affected only by the normalization (11). If ^ k +l

∑ pi l ≠l *

^ k

(l ) < ∑ pi (l ) l ≠l *

(14)

Estimation of Sea Ice Movement Using Locally Parallel Matching Model

643

the probability that ai has label l * increases. Otherwise ^ k +1

∑p l ≠l *

i

^ k

(l ) ≥ ∑ pi (l )

(15)

l ≠l *

it decrease or remain the same. Such iteration procedure is repeated for a certain times. Matching Rule: If Pi (l ) ≥ T , the pair points are considered to be matched. Pi (l ) < T , the pair points will be cancelled.

T is the threshold for estimating the likelihood. Some nodes may remain ambiguous, with several potential matches retaining nonzero probabilities.

4

Estimation of the Velocity of the Sea Ice Movement

Because the gray of sea ice images always change slightly, sometimes there would be an invalid match. So we present a orientation choose and probability decision method to estimate the velocity of sea ice like that: After matching each neighbor images, we can get a set of matching pairs: Ai : {( x1i , y1i ), ( x2i , y 2i )}

(16)

Ai is the i th matching pair of the two neighbor image; ( x1i , y1i ) is the coordinates of the i th matching point of the first image. ( x2i , y 2i ) : the coordinates of the i th

matching point of the next image. Compute the velocity vector of each matching pair:

v Vi = [( x2i − x1i ), ( y2i − y1i )T

(17)

Fig. 2. Matching results

r According to the angle of Vi , each matching pair can be allocated to one of the eight orientation subsets(each orientation subset represents one orientation):

644

Kong Xiangwei et al.

j = 1,2L8 , Am ⊂ A

Subset ( j ) = { Am }

(18)

Here, Subset ( j ) is one of the eight orientation subsets. The eight orientations include: up, down, left, right, left-up, left-down, right-up, right-down. The probability that the velocity orientation of the sea ice belongs to Subset ( j ) is defined as: P( j ) =

Number ( j ) 8

∑ Number ( j )

(19)

k =1

Here Number( j ) is the number of the elements in Subset ( j ) . Find the maximum probability,

Pmax = max P( j ) j

j = 1,2L8

(20)

Assume that the maximum probability corresponds to Subset (n) , all the elements in Subset (n) will be used to estimate the velocity of the sea ice. Then, the final result of r the estimating velocity is V : r Vm ∑ r A ⊂ Subset ( n ) (21) V= m Number ( j ) Am is all the matching pairs belonging to the Subset (n) . r Vm is the velocity vector of the matching pair corresponding to Am .

5

Results and Conclusion

According to the character of the sea ice images, we adopt a locally parallel matching model to estimate the velocity of the sea ice. First, use an interest operator to choose the candidate points. Then, a locally parallel model is adopted to match the neighbor images of sea ice. Finally, we present a method to select the maximum probability subset for estimating the velocity. An important property of this matching algorithm is that it works for any mode of disparity and does not require precise information about camera orientation, position and distortion. And the method of selecting maximum probability subset for estimating the velocity can eliminate the case of some invalid matching. By bounding a camera on the ice breaker and using a general image collection board, we have applied the above system to estimate the velocity of the sea ice in Bohai of north China. In practice, the parameters in the above expressions are selected as below: In section3, The window size for matching is 5×5. Constant c in the

Estimation of Sea Ice Movement Using Locally Parallel Matching Model

645

equation (5) equals 0.001. The constants A and B in the equation (10) are selected as 0.6 and 3. In equation (12), (13), Θ = 1 , R = 15 . The iteration procedure is repeated for eight times. In the matching rules, the threshold T=0.5. These methods have been proved to have nice effects and stabilization in practice of estimating velocity of sea ice. The precise level of estimating velocity of the sea ice can meet the demand of application.

References [1] [2] [3] [4] [5] [6] [7]

S. T. Barnard, W. B. Thompson, “Disparity analysis of images”, IEEE Transactions on PAMI, Vol. PAMI-2, pp.333-340, July 1980 H. P. Moravec, “Towards automatic visual obstacle avoidance,” in Proc. 5th Int. Joint Conf. Artificial Intell., Cambridge, MA, Aug. 1977, p. 584 M. D. Levine, D. A. O' Handley, and G. M. Yagi, “Computer determination of depth maps,” Comput. Graphics Image Processing, Vol. 2, pp. 131-150, 1973 K. Mori, M. Kidode, and H. Asada, “An iterative prediction and correction method for automatic stereo-comparison,” Comput. Graphics Image Processing, Vol. 2, pp.393-401, 1973 R. Nevatia, “Depth measurement by motion stereo,” Comput. Graphics Image Processing, Vol.5, pp.203-214, 1976 N. Evans, “Glacier surface motion computation from digital image sequences”, To appear in IEEE Transaction on Geoscience and Remote Sensing, 2000 Zhang Yujin, Image engineering: Image understanding and computer vision, Tsinghua press, 2000

Evaluation of a Combined Wavelet and a Combined Principal Component Analysis Classification System for BCG Diagnostic Problem Xinsheng Yu1, Dejun Gong2, Siren Li2, and Yongping Xu2 Marine Geology College, Ocean University of China Qingdao 266003, P. R. China 2 Institute of Oceanology, Chinese Acadamy of Sciences Qingdao 266071, P. R. China 1

Abstract. Heart disease is one of the main factor causing death in the developed countries. Over several decades, variety of electronic and computer technology have been developed to assist clinical practices for cardiac performance monitoring and heart disease diagnosis. Among these methods, Ballistocardiography (BCG) has an interesting feature that no electrodes are needed to be attached to the body during the measurement. Thus, it is provides a potential application to asses the patients heart condition in the home. In this paper, a comparison is made for two neural networks based BCG signal classification models. One system uses a principal component analysis (PCA) method, and the other a discrete wavelet transform, to reduce the input dimensionality. It is indicated that the combined wavelet transform and neural network has a more reliable performance than the combined PCA and neural network system. Moreover, the wavelet transform requires no prior knowledge of the statistical distribution of data samples and the computation complexity and training time are reduced.

1

Introduction

Traditionally, the physicians in hospitals need to interpret characteristics of the measured records and to calculate relevant parameters to determine whether or not the heart shows signs of cardiac disease. Recently, the advances in computer and electronic technology have provided a basis for automatic cardiac performance monitoring and heart disease diagnosis to assist clinical practice by saving diagnostic time. These technologies along with artificial intelligence research have also established entirely new applications in detecting the risk of heart disease [4]. However, the demands of practical health care require that these technologies are not limited to hospital environments but are able to detect the vital signs of heart disease during daily life under unrestricted conditions. For example, it would be helpful for both doctors and patients, if the preliminary heart condition could be monitored regularly at the home before making the decision whether or not it is necessary to visit V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 646-652, 2003.  Springer-Verlag Berlin Heidelberg 2003

Evaluation of a Combined Wavelet

647

hospital, for further multiple assessment and treatment. Thus, the time and transport expenses could be saved. These requirements lead an increasing innovation in portable systems in reduction in size, weight and power consumption to an extent that the instruments are available in routine life monitoring and diagnosis at home. BCG provides a potential application to monitor heart condition in the home because no electrodes are required to be attached to the body during the measurements. In the past several years, the BCG has been used in a variety of clinical studies such as prognosis, monitoring, screening, physical conditioning, stress tests, evaluation of therapy, cardiovascular surgery and clinical conditions [1]-[4]. It would therefore be of value if the BCG could be used as a non-invasive alternative for the prediction of subjects at risk of heart attack that could help clinicians to establish an early treatment. Although studies have been made in the last few decades on the subject of improving BCG measurement technology, there has tended to be relatively little work done to attempt to improve the BCG process and analysis methods by using modern signal processing technology There is little work on developing computer assisted analysis systems that are capable of operating in real time, especially those incorporating artificial intelligence methods for BCG pattern classification. Moreover, no reports have been made in the literature to implement decision making algorithms in a portable system for real world application [5]. In this paper, two BCG classification models are evaluated. One is a combined PCA and a three layer neural network system, and the other is to use wavelet transform to decompose the original signal into a series of views for neural network classifier. The performance and computation efficiency of the two models are compared. It illustrated that the combined wavelet transform with backpropagation neural network has a more reliable performance than the combined PCA and neural network system.

2

BCG Signal Feature Selection

A classification of the whole BCG waveform can be computationally intensive by the neural network classifier. The high dimension number inputs forces us to use a network with a higher number of free parameters which means a large training samples is required to estimate these parameters. The time involved in the network training is considerable. Moreover, the presence of irrelevant feature combinations can actually obscure the impact of the more discriminating inputs. A small input means a smaller neural network structure is required and thus could save in time and system implementation for real world applications. 2.1

BCG Signal and Preprocessing

The BCG data set used in this study was provided by the medical Informatics Unit of the Cambridge University. The signal was sampled at 100 samples/second from the seat of the chair in a clinical trial. During the BCG recording, the reference ECG signals were recorded simultaneously from the arms of the chair at the same sampling rate. The R-wave of the ECG signal is used to identify each BCG cycle. The BCG

648

Xinsheng Yu et al.

signal are filtered and then averaged to reduce the background noise. Each averaged BCG signal was then normalized into a standard 80 points. Altogether, there are 58 normal subjects, 275 mild hypertension subjects and 6 subjects who died suddenly due to myocardial infarction within 24 months of the BCG recording. Figure 1 shows the standard length BCG signal of normal and hypertension subjects.

1

11

21

31

41

51

61

71

1

11

21

31

(A)

41

51

61

71

(B)

Fig. 1. BCG of normal (A) and hypertension (B)

2.2

Using PCA for Feature Extraction

Principal components are usually derived by numerically computing the eigenvector of the variance matrix. Those eigenvectors with largest eigenvalue (variance) are used as features onto which the data are projected [7],[8]. This method may be considered optimal, in the sense that mean square linear reconstruction error is minimised. For a given data set X = ( X1 , X 2 ,... X N ) with a zero mean, a simple approach to PCA is to compute the simple variance matrix

C=

1 N X i X iT ∑ N i =1

(1)

The next step is to use any of the existing standard eigenvector analysing algorithms to find the m largest eigenvalues and their corresponding eigenvectors of the data covariance matrix:

Cφ = λφ

(2)

where λ is the eigenvalue, and φ is its corresponding eigenvector. The m principal components of n dimensional data are the m orthogonal directions in n spaces which capture the greatest variation in the data. These are given by the eigenvectors φ . In this study, it is found that 10 principal components are have the ability to maintain the required information for the neural classifier 2.3

BCG Signal Analysis Using Discrete Wavelet Transform

In the BCG signal classification, most of the information needed for classification is in the BCG shape and related parameters, such as the wave amplitudes of H, I, and J,

Evaluation of a Combined Wavelet

649

slopes of base line to H, H-J, and I-J, and times of H-J, I-J as well as H-I and H-J in percent of heart period [6]. These kinds of features are often dependent on the degree of resolution. At a coarse resolution, information about the large features of the complex waves is represented; while at a fine resolution, information about more localised features of individual waves is formed. Upon discarding selected detail, one can obtain the required information for the classifier without much signal distortion. It is true that not every resolution is equally important to provide the classification information in the classification phase. Therefore, it is possible to choose a proper resolution and discard the selected component without much signal distortion. Using the wavelet transform, one can project the raw BCG data into a small dimension space and decompose the original signal into a series of sub-signals at different scales. The coarse components which still have the significant shape information can be used to present to the neural network classifier. One of the most useful features of wavelets is that one can choose the defining coefficients for a given wavelet system to be adapted for a given problem. In Mallat's original paper [9], he developed a specific family of wavelets that is good for representing polynomial behaviour. As there is no prior information to suggest that one wavelet function would be better suited to BCG analysis than others, the filter coefficients derived by Mallat [9] are adopted for this study. In this study, the signal reconstruction procedure is not required. Thus, only low pass filter processing is used to approximate the original BCG signal. The fast discrete wavelet transform algorithm is implemented in C code for BCG signal decomposition. As the original data frame has 80 sample points, at each successive resolution level, the pyramidal algorithm compresses the data by a factor of 2. At the first process level, the N dimension signal f ( x ) is taken as C0 and is decomposed into two bands: coarse component C j ,n and detail component D j ,n which both have N/2 samples. At the next level, only the low-pass output signal is further split into two bands. Signals at various resolutions give information about different features. In time scale, they are 40, 20, and 10 samples respectively. According to our experiment results, the level of 20 coarse components have relatively good classification performance.

3

BCG Classification

The neural network model used in this study is a variation of the backpropagation neural network. with a single hidden layer. The network parameters are empirically defined. The whole BCG data set ispatitioned randomly into two data set which shows in Table 1. One data set is used for the neural classifier training and another data set is used for test. We then swap the training data set and testing data set to repeat the training and testing procedure again. The same partitioning of the data set was used in all runs of this investigation and the generalization capability of the neural classifier was evaluated using the same strategy.

650

Xinsheng Yu et al. Table 1. Training and Testing Data Sets

Subjects Normal (N) Hypertension (H) Risk of Heart Attack (RHA)

4

Results

4.1

Recognition Performance

Training Data Set 30 111 3

Testing Data Set 28 164 3

After training the neural network, the weights were fixed and the network was evaluated. The results of using data set 1 as training data set for combined PCA and neural classifier is shown in Table 2. Table 3 illustrates the classification results of using data set 2 as training data set. The results for the combined wavelet transform and neural network classification system are shown in Table 4 and Table 5 respectively. Table 2. Classification Results Using 10 Principal Components

Class N H RHA Overall Performance

Training Data N H 30 0 0 111 0 1

RHA 0 0 2

99.31%

Testing Data N H 22 6 3 160 1 2

RHA 0 1 0

93.33%

Table 3. Classification Results of Swapped Data Sets Using 10 Principal Components

Class N H RHA Overall Performance

Training Data N H 28 0 1 163 0 1

RHA 0 0 2

98.97%

Testing Data N H 25 5 2 105 0 2

RHA 0 4 1

90.97%

Table 4. Classification Results Using Combined Wavelet Transform and Neural Classifier

Class N H RHA Overall Performance

Training Data N H 28 0 0 111 0 1 99.3%

RHA 0 0 2

Testing Data N H 25 2 2 159 0 1 95.38%

RHA 1 3 2

Evaluation of a Combined Wavelet

651

Table 5. Classification Results of Using Swapped Data Sets

Class N H RHA Overall Performance 4.2

Training Data N H 28 0 0 164 0 1 99.49%

RHA 0 0 2

Testing Data N H 27 3 0 107 0 1

RHA 0 4 2

94.44%

Computational Complexity

For real time applications, the reduction of computation complexity is of greatest concern for the engineering design. Table 6 shows a comparison of the forward pass computation complexity of the two combined classification systems. It shows that the computation operation required for the combined wavelet transformation is similar to that of the combined PCA network with 10 outputs. However, the memory storage required for the combined wavelet transform and neural classifier system is much smaller. Moreover, unlike the PCA method, no training procedure is required to generate the features for classifier inputs. This provides a great advantage for real time implementation. The comparison results of performance and computation efficiency suggest that the combined wavelet transformation and neural network system is suitable for real time implementation.

5

Conclusion

A study was made to determine the usefulness of a combination of different types of features for BCG classification. The simulation results using the partition training scheme show that the wavelet transform with neural network system has better performance than a combined PCA and neural networks approach. The simulation results also suggested that the improvement in system performance is constrained by the limited number of data samples. The results indicate that the number of data samples and balance of the class patterns in the samples have a significant influence on the generalization capability of the combined PCA and neural network system. Hoffbeck and Landgrebe [11] have suggested that the statistical method needs a large training data set to provide reliable statistical estimation, for example, to accurately estimate a sample covariance matrix. Although constructing a compact supported mother wavelet involves complex mathematics, the calculation of the wavelet transform is relatively simple. The wavelet transform can be implemented as finite impulse response (FIR) filters [10]. It has been demonstrated that the wavelet transform has a compact computational complexity. From the classification performance and real time implementation point of view, the multiresolution wavelet is promising for an on-line dimensionality reduction in BCG analysis in term of performance, storage and operations per decision.

652

Xinsheng Yu et al. Table 6. Comparison of the Computation Complexity for BCG Classification

Feature Classifier Total

Combined PCA Method Multiplication Addition Storage 800 808 808 115 115 115 915 923 923

Combined Wavelet Method Multiplication Addition Storage 720 660 12 99 99 99 819 759 111

References [1]

I. Starr and F.C. Wood, “Twenty years studies with ballistocardiograph, the relation between the amplitude of the first record of ‘health' adults and eventual mortability and morbidity from the heart disease,” Circulation, Vol. 23, pp. 714-732, 1961 [2] 2 C.E. Kiessling, “Preliminary appraisal of the prognostic value of Ballistocardiography,” Bibl. Cardiol., Vol. 26, pp. 292-295, 1970 [3] 3 T.N. Lynn and S. Wolf, “The prognostic significance of the ballistocardiogram in ischemic heart disease,” AM. Heart, J., Vol. 88, pp. 277280, 1974 [4] 4 R.A. Marineli, D.G. Penney, W.A. Marineli, and F. A. Baciewicz, “Rotary mation in the heart and blood vessels: A review,” Journal of Applied Cardiology, Vol. 6, pp. 421-431, 1991 [5] X. Yu, “Ballistocardiogram classifier prototyping with CPLDs,” Electronic Engineering, Vol. 68, No. 834, pp. 39-40, 1996 [6] W.K. Harrison, S.A. Talbot and Baltimore, “Discrimination of the Quantitative Unltralow-frequency Ballistocardiogram in Coronary Heart Disease,” American Heart Journal, Vol. 74, pp. 80-87, 1967 [7] I.T. Jolliffe, “Principal Component Analysis,” Springer-Verlog Press, 1986 [8] E. Oja, H. Ogawa, and J. Wangviwattana, “Principal component analysis by homogeneous neural networks, Part I & Part II: The Weighted Subspace Criterion,” IEICE Trans. INF. & Syst., Vol. E75-D, No. 3, pp. 366-381, 1992 [9] S. G. Mallat, “A theory for multiresolution signal decomposition: The wavelet representation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 11, No. 7, pp. 674-693, 1989 [10] O. Rioul and M. Vetterli, “Wavelet and signal processing”, IEEE Transaction on Signal Processing Magazine, Vol. 8, pp. 14-38, 1991 [11] J.P. Hoffbeck and D.A. Landgrebe, “Covariance matrix estimation and classification with limited training data,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 18, No. 7, pp. 763-767, 1996

Learning from the Expert: Improving Boundary Definitions in Biomedical Imagery Stewart Crawford-Hines Colorado State University Fort Collins, Colorado USA [email protected]

Abstract. Defining the boundaries of regions of interest in biomedical imagery has remained a difficult real-world problem in image processing. Experience with fully automated techniques has shown that it is usually quicker to manually delineate a boundary rather than correct the errors of the automation. Semi-automated, user-guided techniques such as Intelligent Scissors and Active Contour Models have proven more promising, since an expert guides the process. This paper will report and compare some recent results of another user-guided system, the Expert's Tracing Assistant, a system which learns a boundary definition from an expert, and then assists in the boundary tracing task. The learned boundary definition better reproduces expert behavior, since it does not rely on the a priori edge-definition assumptions of the other models.

1

Background

The system discussed in this paper provides a computer-aided assist to human experts in boundary tracing tasks, through the application of machine learning techniques to the definition of structural image boundaries. Large imagery sets will usually have a repetition and redundancy on which machine learning techniques can capitalize. A small subset of the imagery can be processed by a human expert, and this base can then be used by a system to learn the expert's behavior. The system can then semiautomate the task with this knowledge. The biomedical domain is a rich source of large, repetitive image sets. For example, in a computed tomographic (CT) scan, cross-sectional images are generated in parallel planes typically separated by millimeters. At a 2mm separation between image planes, approximately 75 images would be generated in imaging the complete brain. Images such as this, generated along parallel planes, are called sectional imagery. Such sectional imagery abounds in medical practice: X-ray, MRI, PET, confocal imagery, electron microscopy, ultrasound, and cryosection (freezing and slicing) technologies all produce series of parallel-plane 2D images. Generating a three dimensional polygonal model of a structure from sectional imagery requires bounding the structure across the whole image set. Currently, the V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 653-659, 2003.  Springer-Verlag Berlin Heidelberg 2003

654

Stewart Crawford-Hines

reference standard for high-quality outlining tasks is an expert's delineation of the region. The state-of-the-practice is that this is done manually, which is a repetitive, tedious, and error-prone process for a human. There has been much research directed toward the automatic edge detection and segmentation of images, from which the extraction of a boundary outline could then proceed. Systems based on this work have run into two significant problems: (1) the cost of user adjustments when the system runs into troublesome areas often exceeds the cost of manually tracing the structure from the start, and (2) the a priori assumptions implicit in these methods impact the ability of the system to match the expert's performance on boundary definition tasks where an expert's judgement is called into play. The system discussed herein, the Expert's Tracing Assistant (ETA), provides viable assistance for such tracing tasks which has proven beneficial in several regards:

• • •

the specialist's time can be significantly reduced; errors brought on by the tedium of tracing similar boundaries over scores of similar images can be reduced; and the automated tracing is not subject to human variability and is thus reproducible and more consistent across images.

The thrust of this work is to learn the boundary definitions of structures in imagery, as defined by an expert, and to then assist the expert when they need to define such boundaries in large image sets. The basic methodology of this Expert's Tracing Assistant is: 1) 2) 3) 4)

in one image, an expert traces a boundary for the region of interest; that trace is used in the supervised training of a neural network; the trained network replicates the expert's trace on similar images; the expert overrides the learned component when it goes astray.

The details of the neural network architecture and training regimen for this system have been previously documented by Crawford-Hines & Anderson [1]. This paper compares these learned boundaries to the semi-automated boundary definition methods of Intelligent Scissors by Barrett & Mortensen [2] , and Active Contour Models begun by Kass, Witkin, & Terzopoulos [3].

2

Methods for Boundary Tracing

To understand the relative merits of learning boundary contours, the Expert's Tracing Assistant (ETA) was studied in comparison to other user-guided methods representing the current best state-of-the-practice for boundary delineation. The techniques of Active Contour Models (ACM) and Intelligent Scissors (IS) were chosen for comparison to ETA because of they have been brought into practice, they have been studied and refined in the literature, and they represent benchmarks against which other novel methods are being compared. The ground truth for the comparison of these methods is an expert's manual tracing of a structure's boundary in an image.

Learning from the Expert: Improving Boundary Definitions in Biomedical Imagery

655

The structures chosen for comparison were taken from the imagery of the Visible Human dataset, from the National Library of Medicine [4], which is also becoming a benchmark set upon which many image processing and visualization methods are being exercised. Several structures were selected as representative cross-sections. For each, the IS, ACM, and ETA methods were used to define the structure's boundary and an expert manually delineated the boundary in two independent trials. Figure 1 shows three structures to be compared. The leg bone and skin are shown clearly, without much confusion. The leg muscle is fairly typical, surrounded by highly contrasting fatty tissue. However sometimes only a thin channel of tissue separates one muscle from the next. IS, also known as Live-Wire, is a user-guided method. With an initial mouse click, the user places a starting point on a boundary of interest; the system then follows the edges in an image to define a path from that initial control point to the cursor's current screen location. As the cursor moves, this path is updated in real time and appears to be a wire snapping around on the edges in an image, hence the terminology "live wire" for this tool. ACM, another user-guided methodology, uses an energy minimizing spline, that is initialized close to a structure of interest and then settles into an energy minima over multiple iterations. The energy function is defined so these minima correspond to boundaries of interest in the imagery. Since the initial contour is closed, the final result will always be a closed, continuous curve.

Fig. 1. A transverse image of the leg, highlighting the femur (bone), the biceps femoris (muscle), and the skin

656

Stewart Crawford-Hines

3

Comparing Boundaries

For a ground truth in this comparison, an expert was asked to manually trace the structures. The expert traced the structures twice, generating two independent contours for each structure. This permits a basic measure of the variation within the expert's manual tracings to be quantified. It might be argued that this ground truth is not really a truth, it is only one user's subjective judgement of a structural boundary. But the expert user brings outside knowledge to bear on the problem and is dealing with more than simple pixel values when delineating a boundary. And for a system to be useful and acceptable as an assistant to an expert, it should replicate what the expert is attempting to do, rather than do what is dictated by some set of a priori assumptions over which the expert has no input or control. The boundary definitions are to be quantitatively compared to each other and to the ground truth of the expert. The boundaries produced by each of these methods are basically sets of ordered points which can be connected by lines or curves or splines to create a visually continuous bound around a region. To compare two boundaries, A and B, we first connect the points of B in a piecewise linear curve, and measure the minimum distance from each point in A to the curve of B. We then flip the process around, and measure from each point in B to the piecewise linear curve of A. The collection of these measures is called a difference set. Figure 2 illustrates several visualizations of this difference set. The first graph, in the upper half of the figure, plots the distance between the curves (on the vertical axis) as a function of position on the curve (on the horizontal axis). In this example, there are perhaps three places where the curves are significantly more than one pixel apart from each other, shown in the plot by the six excursions of the graph over the distance of 1.0 (remembering the plot measures A to B and B to A, thus typically excursions show up twice). If the goal is to have the curves within one pixel of each other, this indicates that there are three places where operator intervention would be required to adjust the curves so as to meet that objective. The lower left of Figure 2 is a simple histogram of the distance set, with the number of occurrences on the vertical axis and distance bins on the horizontal. The lower right is an empirical cumulative distribution function (CDF) over the distance set. The vertical axis measures the fraction of occurrences that are within a tolerance specified on the horizontal axis. The CDF allows quantification of the inter-curve distances by selecting a tolerance such as one pixel and stating, "The curves are within one pixel of each other 86% of the time" or by selecting a percentile such as 90% and stating, "The curves are within 1.1 pixels of each other 90% of the time".

Learning from the Expert: Improving Boundary Definitions in Biomedical Imagery

657

Fig. 2. Three visualizations of the distance set

4

Key Results

The three structures of Figure 1 typify the range of results found so far across many images. The expert manually outlined each structure on two independent trials. The expert's first boundary is used as the Ground Truth (GT), while the second manually traced boundary (M2T) is used to provide a measure of intra-expert variation, i.e., the inherent variation a user shows in tracing a boundary at different times. Given there exists variation within what an expert might manually trace, a good boundary delineation method needn't exactly match any specific expert's boundary definition, but it should be within the range of that expert's variance. Looking at the left-hand side of Figure 3, this is exactly what is observed. The black curve illustrates the CDF of M2T compared to the Ground Truth. Note the three methods are roughly comparable, all close to the bound of the black CDF. The righthand side of the figure shows the performance for the muscle. The performance is consistently worse overall. Figure 4 shows a detail of the five boundaries superimposed on the original image; the expert's traces are in black, the semiautomated traces in white. In the lower-left of the muscle, however, there is no consistency of definition - even the expert made different traces at different times. All did equally poorly. Figure 5 shows the results for the leg skin. Here the performance difference is dramatic between ETA (far left) and the IS and ACM methods (to the right). Figure6 illustrates what is happening in this situation. The expert judgement of the skin boundary places the boundary slightly inside what a more classically-defined boundary would be; note that both IS and ACM are agreeing on where the boundary lies, and apriori this appears to be a sensible boundary to draw. In this case, however, the body was encased in a gel before freezing, and the expert is accounting for both gel effects and the image pre-processing in locating the actual skin boundary. The expert is consistent in this judgement, and the ETA system has learned this behavior and replicated it.

658

Stewart Crawford-Hines

Fig. 3. CDF s of M2T (black) and IS, ACM, and ETA (grey) for the bone (left) and the muscle (right) from Figure 1

Fig. 4. Detail of the five boundaries

Fig. 5. Results for the leg's skin: CDFs for, from left to right: ETA, M2T, ACM, and IS

Learning from the Expert: Improving Boundary Definitions in Biomedical Imagery

659

Fig. 6. In this detail of the skin, the expert (in black) has traced consistently to the inside of what would be clasically considered the image's edge; ETA (white) follows the expert's lead, while IS and ACM follow more traditional edge definitions.

The range of results in Figures 3 and 5 typify what has been seen so far: the learned boundary was either consistent with the classically defined IS and ACM methods, or it did better when expert judgement was called into play.

References [1] [2] [3] [4]

S. Crawford-Hines & C. Anderson, "Learning Expert Delineations in Biomedical Image Segmentation", ANNIE 2000 - Artificial Neural Networks In Engineering, November 2000. E.N. Mortensen & W.A. Barrett, "Interactive Segmentation with Intelligent Scissors", Graphical Models and Image Processing, v.60, 1998, pp.349-384. M. Kass, A. Witkin, D. Terzopoulos, "Snakes: Active Contour Models", First International Conference on Computer Vision, 1987, pp.259-268. National Library of Medicine, as of Jan 2001: http://www.nlm.nih.gov/research/visible

Low Complexity Functions for Stationary Independent Component Mixtures K. Chinnasarn1
, C. Lursinsap1 , and V. Palade2 1

AVIC Center, Department of Mathematics, Faculty of Science Chulalongkorn University, Bangkok, 10330, Thailand [email protected] [email protected] 2 Oxford University Computing Laboratory Parks Road, Oxford, OX1 3QD, UK [email protected]

Abstract. Obtaining a low complexity activation function and an online sub-block learning for non-gaussian mixtures are presented in this paper. The paper deals with independent component analysis with mutual information as a cost function. First, we propose a low complexity activation function for non-gaussian mixtures, and then an online sub-block learning for stationary mixture is introduced. The size of the sub-blocks is larger than the maximal frequency Fmax of the principal component of the original signals. Experimental results proved that the proposed activation function and the online sub-block learning method are more efficient in terms of computational complexity as well as in terms of learning ability. Keywords: Blind signal separation, independent component analysis, mutual information, unsupervised neural networks

1

Introduction

In ”independent component mixture” or ”non-gaussian mixture” problems, the source signals are mutually independent and no information about the mixture environment is available. The recovering system receives unknown mixed signals from the receivers, such as microphones or sensors. In order to recover the source signals, some unsupervised intelligent learning systems are needed. We used, in our approach, a combined learning system, including unsupervised neural networks, principal component analysis, and independent component analysis using mutual information with natural gradient. The learning rule is based on a Mutual Information (MI) using the Kullback-Leibler distance measure:
KLp(y)||p(˜y) = 

p(y) log

p(y) dy p(˜ y)

(1)

This work is fully supported by a scholarship from the Ministry of University Affairs of Thailand.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 660–669, 2003. c Springer-Verlag Berlin Heidelberg 2003 

Low Complexity Functions for Stationary Independent Component Mixtures

661

where p(y) is a true probability density function of y, and p(˜ y) is an estimated parameter of the probability density function of y. For low computational time, we propose some approximation functions of 2nd order for non-gaussian mixtures. In our previous paper [3], we presented only the batch mode learning, which needs a computational complexity of at least O(K 3 ). In this paper, we propose an online sub-block learning method which requires a complexity of at most O(K.k 2 ), where k < K. The paper is organized as follows: Section 2 describes some background of an ICA problem; Section 3 proposes some low complexity approximation of the activation functions; Experimental design and results are presented in section 4 and 5, respectively; Section 6 shows an analytical considerations on complexity; some conclusions are presented in section 7.

2

Independent Component Analysis Problem

An Independent Component Analysis (ICA) problem is an adequate approach because, usually, the sources si are statistically independent from each others. A very well-known practical example of an ICA application is the cocktail party problem. This problem assumes there are some people talking simultaneously in the room, which is provided with some microphones for receiving what they are talking about. Herein, we assume that there are n people, and m microphones as illustrated in figure 1 (for n = m = 3). Each microphone M icj gives you a recorded time signal, denoted as xj (t), where 1 ≤ j ≤ m and t is an index of time. Each of these recorded signals is a linear combination of the original signals si , (1 ≤ i ≤ n) using the mixing matrix A, as given below: xj (t) =

n 

aji si (t)

(2)

i=1

where aji , 1 ≤ j, i ≤ n are the weighted sum parameters, that depend on the distance in between the microphones and the speakers [7]. If the sources si are near to the receivers M icj , the elements of the mixing matrix A are similar to a diagonal matrix, a special case of ICA problem. In case of mixture in diag environment, it is said that there is no mixing occurrance between the sources si , because an original signal is only scaled and/or permuted by the diag mixing matrix. Commonly, the elements of mixing matrix A and the probability density function of si (t) are unknown in advance. The only basic assumption of the cocktail party problem is that all of the sources si (t) are independent and identically distributed (iid). The basic background of ICA were presented in [1] [4] [7]. The objective of an ICA problem is to recover the source signals ˜ s = y = Wx from an observed signal x = As, where each component of the recovered signals yi (t) is iid. The equation for transforming the mixed signals is the following: ˜ s = y = Wx = WAs = A−1 As = Is = s

(3)

662

K. Chinnasarn et al.

A

W x1

y1

s

2

x

y2

s3

x

s1

2

y3

3

Fig. 1. The cocktail party problem with 3 speakers and 3 receivers

Equation (3) shows that the full rank de-mixing matrix W is needed for recovering the mixed signal xi . There are many successful algorithms for finding the de-mixing matrix W, such as minimization of mutual information [1], infomax and maximum likelihood estimation [2], projection persuit [7] etc. Herein, we prefer the mutual information with natural gradient (MING algorithms), which was proposed by Amari et al. [1] in 1996. Lee et al. [9] and Chinnasarn and Lursinsap in [3] added a momentum term β∆Wt , for speeding up the convergence. The learning equation is described below: Wt+1 = Wt + η[I − φ(y)yT ]Wt + β∆Wt

(4)

where η is the learning rate, β is the momentum rate, I is an identity matrix, t is i) the time index, φi (y) = ∂p(y ∂yi is the nonlinear activation function which depends on the probability density function of the source signals si , and y = Wx. Let’s consider that the activation functions for de-mixing of the sub-gaussian and the super-gaussian distributions are φ(y) = y3 and φ(y) = tanh(αy), respectively. In this paper, we are looking for some activation functions of lower complexity than the functions given above.

3

Low Complexity Activation Functions

In the ICA problem, at most one gaussian channel is allowed because the transformation of two gaussianities are also gaussian in another variable [7]. The nongaussian signals can be classified into the super-gaussian and the sub-gaussian distributions. A super-gaussian signal has a sharp peak and a large tail probability density function (pdf). On the other hand, sub-gaussian signals have a flat pdf. As we described in the previous section, the nonlinear activation function φi (y) in equation (4) is determined by the sources’ distribution. In this paper, we used the Kurtosis [4, 5] for selecting an appropriate activation function.

Kurtosis(s) =

E[s4 ] −3 (E[s2 ])2

(5)

where Kurtosis(s) values are negative, zero, and positive for the sub-gaussianity, the gaussianity, and the super-gaussianity, respectively.

Low Complexity Functions for Stationary Independent Component Mixtures

3.1

663

Approximation for Super-Gaussianity

In [8], Kwan presented the KTLF (Kwan Tanh-Like activation Function), which is a 2nd order function. This function is an approximation of tanh(2y) function. He divided the approximation curve into 3 regions, which are the upper bound y(t) ≥ L, the nonlinear logistic tanh-like area −L < y(t) < L, and the lower bound y(t) ≤ −L. All regions are described below:  1,  y

(y ≥ L) y (γ − θ ), (0 ≤ y < L) L L φ(y) = y y (γ + θ ), (−L < y < 0)  L L  −1, (y ≤ −L)

(6)

The shape of KTLF curve is controlled by γ = 2/L and θ = 1/L2. The approximation function given in equation (6) corresponds to the tanh(2y) function. Consequently, the term α2 is needed for controlling y, and we also suggest L = 1. Then the modified equation can be rewritten as follow:  1,  

(y ≥ 1) ´ (2 − y ´ ), (0 ≤ y < 1) y φ(y) = ´ (2 + y ´ ), (−1 < y < 0)  y −1, (y ≤ −1)

(7)

´ = αy where α is an upper-peak of the derivative of the activation function and y 2 . Figure 2 shows tanh(αy), its approximation (the dash line) and their derivatives. From the figure we can conclude that the fraction α2 is fitted for all tanh(αy).

Logistic tanh(2y) and tanh−like(2y) activation function

Derivative of Logistic tanh(2y) and tanh−like(2y) activation function 2 φ(y)= tanh(2y) φ(y)= tanh−like(2y)

1

φ(y)= tanh(2y) φ(y)= tanh−like(2y)

1.8

0.8 1.6 0.6 1.4 0.4 1.2

0.2 0

1

−0.2

0.8

−0.4 0.6 −0.6 0.4 −0.8 0.2

−1

−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

0 −2.5

−2

(a)

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

(b)

Fig. 2. (a) An activation function φ(y) = tanh(2y), its approximation from equation (7) and (b) their derivatives

3.2

Approximation for Sub-gaussianity

In this subsection, we propose a 2nd order approximation of φ(y) = y11 and φ(y) = y3 , presented by Amari et al. [1] and Cichocki et al. [4], respectively.

664

K. Chinnasarn et al. Sub−gaussian activation functions

Derivative of sub−gaussian activation functions

2.5

6

φ(y)=Amari Fn. φ(y)=y3 2 φ(y)=+/−y

2

φ(y)=Amari Fn. φ(y)=y3 2 φ(y)=+/−y

4

1.5 2 1 0 0.5 −2 0 −4 −0.5 −6

−1

−1.5

−8

−2

−2.5 −2

−10

−1.5

−1

−0.5

0

0.5

1

1.5

2

−2.5

−2

−1.5

(a)

−1

−0.5

0

0.5

1

1.5

2

2.5

(b)

Fig. 3. (a) Graphical representation of 11th , 3rd , and 2nd order activation function and (b) their derivatives

Given the graphical representation of the sub-gaussian activation functions illustrated in figure 3, it can be seen that the sub-gaussian activation functions can be separated into 2 regions: the positive and the negative regions. For demixing the sub-gaussian distribution, we proposed the bisection paraboloid function given in equation (8) which is a good approximation for the previously reported functions in the literature.  φ(y) =

4

+y2 , (y ≥ 0) −y2 , (y < 0)

(8)

Experimental Design

For the de-mixing efficiency, the signal separating procedure is divided into 2 subprocedures which are the preprocessing and the blind source separating procedure. 4.1

Preprocessing

For speeding up the procedure of signal transforming, some preprocessing steps are required. We used 2 steps of preprocessing as described in [7], which are centering and prewhitening. The first step is the centering. For an ICA learning procedure, an input signal si , and also xi , must be zero mean E[si ] = 0. This is an important assumption because the zero mean signal causes the covariance E[(si − mi )(sj − mj )T ] equal to the correlation si sTj , where sTj is the transpose matrix of sj , and mi is a channel average or a statistical mean. Consequently we can calculate an eigenvalue and an eigenvector from both their covariance and correlation. If the input signal is nonzero mean, it must be subtracted by some arbitary constant, such as its mean.

Low Complexity Functions for Stationary Independent Component Mixtures

665

The second step is prewhitening. This step will decorrelate the existing correlation among the observed channels xi . Sometimes, the convergent speed will be increased by the prewhitening step. The Principal Component Analysis (PCA) is a practical and a well-known algorithm in the multivariate statistical data analysis. It has many useful properties, including data dimensionality reduction, principal and minor feature selection and data classification, etc. In this paper, without loss of generality, we assume that all sources are principal components. Then the PCA is designed as a decorrelated filter. The PCA procedure is the ˜ = VT x where VT is the transpose matrix of an eigenvector of projection of x covariance of the observed signals E[xxT ]. After the projection of x, it becomes ˜ T ] = I, where I is an identity matrix. the covariance E[˜ xx 4.2

Blind Source Separating Processing

In this subsection, we describe an online sub-block ICA learning algorithm. We used an unsupervised multi-layer feed forward neural network for de-mixing the non-gaussian channels. Our network learning method is a combination of the online and the batch learning techniques. Unknown signals xi (k0 : k0 + k) are fed into the input layer, where k0 is the start index of the sub-block, k ≥ Fmax is the length of the sub-block, and Fmax is the value of the maximal frequency in time domain of the principal source si . Output signals yi (k0 : k0 +k) are produced by yi (k0 : k0 + k) = Wxi (k0 : k0 + k), where W is called the de-mixing matrix. If the output channels yi (k0 : k0 + k) depend on each other, the natural gradient descent in equation (4) still updates the de-mixing matrix W. Then repeat to produce the output signals yi (k0 : k0 + k) until they become independent. The increase of the convergence speed of the online sub-block method is proved by the following theorem. Theorem 1. ICA online sub-block learning is converging faster than batch learning. Proof Considering K is the total time index of the signal and k is the time index for each sub-block, where k < K. The learning equation (4) can be rewritten as follow: Wt+1 = Wt + η[I − φ(y)yT ]Wt + β∆Wt

(9)

The computational complexity of equation (9) depends on the correlation φ(y)yT , where yT is a transpose matrix of y. For the batch learning with time index K, the complexity of (9) is of O(K 3 ) On the other hand, for the online sub-block learning, we have K k sub-blocks. 3 2 The computational complexity of equation (9) is of K O(k ) = O(K.k ). It is k 2 3 proved that O(K.k ) < O(K ), where k < K. Hence, the ICA online sub-block learning method is faster than the batch learning method.

666

5

K. Chinnasarn et al.

Experimental Results

Some simulations have been made on both super-gaussian and sub-gaussian signals, which contained 191,258 data points for each channel. The super-guassian data sets were downloaded from http://sound.media.mit.edu/ica-bench/, and consist of 3 sound sources. For sub-gaussianity, we simulated our algorithm using the following three-synthesized signals: 1. s1 (t) = 0.1 sin(400t) cos(30t) 2. s2 (t) = 0.01sign[sin(500t + 9 cos(40t))] 3. s3 (t) = uniform noise in range [-0.05,0.05]

A mixing matrix A is randomly generated. An initial de-mixing matrix W is set to the transpose of an eigenvector of the covariance of an observed signal x. As presented in [3], the convergent criterion should be set so that the distance of Kullback-Leibler divergence is less than 0.00001 (∆KL ≤ 0.00001). We have run 10 simulations using a variable learning rate with initial value 0.05 and a momentum rate value of 0.01η. Anyway, both the learning rate value and momentum rate value can be arbitrary set over range (0..1]. At each learning iteration, the η ). Some of the experimental results learning was decreased by 1.005 (η = 1.005 were presented in our previous paper [3]. For improving the learning performance, we determined the relationship between the online sub-blocks. The final de-mixing matrix W of sub-block j is set to the initial de-mixing matrix of j + 1 sub-block, where 1 ≤ j ≤ T b, T b is the total number of blocks K k , k ≥ Fmax = 20, 000 Hz, and K is the total time index. The weight inheritance will maintain the output channel of unknown mixture environments. Figure 4(a) and 4(b) display the original sources and recovered sources of the sub-gaussian and the super-gaussian distributions, respectively. As an algorithmic performance measurement, we used the performance correlation index, derived from the performance index proposed by Amari et al. in [1]. In practice, the performance index can be replaced with the performance correlation index.

E=

N  N  ( i=1 j=1

N  N  |cij | |cij | − 1) + − 1) ( maxk |cik | maxk |ckj | j=1 i=1

(10)

The matrix C = φ(y)yT is close to the identity matrix when the signal yi and yj are mutually uncorrelated or linearly independent. The results are averaged over the 10 simulations. Figure 5 illustrates the performance correlation index on a logarithmic scale, during the learning process, using our proposed activation functions in section 3, and the typical activation functions for the non-gaussian mixtures from section 2. Figure 5(a) corresponds to the mixture of super-gaussian signals. The curve of tanh(.) was matched with its approximation function, and it can be seen that they converge over the saturated region with the same speed. Figure 5(b) corresponds to the mixture of sub-gaussian distribution. The proposed activation function φ(y) = ±y2 converges faster than

Low Complexity Functions for Stationary Independent Component Mixtures Input:S

X : Block No. 8 0.1

After PCA

667

Output:Y

0.05

0.05

0.1 0.05

0.05 0

0

0

−0.05

0

−0.05

−0.1 8317

−0.1 8317

8367

0.1

0.05

−0.05

−0.05 8317

8367

8367

8317

0.05

8367

0.05

0.05 0

0

0

0

−0.05 −0.05 8317

−0.1 8317

8367

0.05

8367

8317

8367

0.1

0

0

0

−0.05

−0.1

−0.1

8367

8367

0.05

0.1

8317

−0.05

−0.05 8317

8317

8367

0

−0.05

8317

8367

8317

8367

(a) Original Signals:

s

Observed Signals:

x

Recovered Signals:

6 5

2 0

0

−10

−2 −5

0

y1

s1

y

10

4

−4 5

10

15

5

10

4

15

5

10

4

x 10

15 4

x 10

x 10

5

4

5 0

0

y2

s2

2

−2

−5

−4

−5 5

10

15

5

10

4

15

5

10

4

x 10

15 4

x 10

x 10

6

5

5

4 2 0

y3

s3

0

0

0

−2 −5

−5

−4 5

10

15

5

10

4

15

5 4

x 10

x 10

10

15 4

x 10

(b)

Fig. 4. The mixing and de-mixing of (a) sub-gaussianity and (b) supergaussianity φ(y) = y3 in the beginning, and slows down when the outputs yi are close to saturated region. The derivative of the function φ(y) = ±y2 is slower than the derivative of the function φ(y) = y3 when y ≥ ±0.667, see the slope of each function in figure 3.

6

Analytical Considerations on Complexity

For the mixture of the super-gaussian signals, the unknown source signals can be recovered by tanh(αy) and its approximation as given in equation (8). Considering the same input vector, both activation functions produce a similar output vector, because the curve of the approximation was matched to the curve of tanh(αy), as illustrated in figure 2. Hence, they required the same number of epochs for recovering the source signals, as shown in figure 5(a). But, an approximation function requires lower computational micro-operations per instruction than tanh(αy), and is more suitable for hardware implementation. Regarding the recovery of the mixture of more sub-gaussian signals, the curve of φ(y) = ±y2 did not exactly match either y3 or y11 , but they produced the same results with different convergent speed as shown in figure 5(b). The lower activation function needs smaller memory representation during the running process. And, also, the φ(y) = ±y2 requires only XOR and complementary micro-operations per instruction.

668

K. Chinnasarn et al. Performance correlation index for de−mixing of Super−gaussianity

2

Performance correlation index for de−mixing of Sub−gaussianity

10

φ(y)= mKTLF(2y) φ(y)= tanh(2y)

2

φ(y)=+/−y φ(y)=y3

4

10 1

log−scale of performance correlation index

log−scale of performance correlation index

10

0

10

−1

10

−2

10

−3

10

2

10

1

10

0

10

−1

10

−2

−4

10

3

10

10 0

500

1000

1500 2000 number of iterations (epochs)

(a)

2500

3000

3500

400

800 1200 number of iterations (epochs)

1600

2000

(b)

Fig. 5. Performance correlation index for the separation of (a) the supergaussian mixtures and (b) the sub-gaussian mixtures (averaged on 10 simulations)

7

Conclusions

In this paper, we presented a low complexity framework for an independent component analysis problem. We proposed a 2nd order approximation used for the demixing of the super-gaussian and sub-gaussian signals. Moreover, the number of the multiplication operations required by the separating equation was reduced by using an online sub-block learning method. The proposed activation functions and the online sub-block learning algorithm are efficient methods for demixing the non-gaussian mixtures, with respect to the convergence speed and learning abilities.

References [1] S.-I.Amari, A.Cichocki, and H. H.Yang. A New Learning Algorithm for Blind Signal Separation, MIT Press, pp.757-763, 1996. 661, 662, 663, 666 [2] J.-F.Cardoso. Infomax and Maximum Likelihood for Blind Source Separation, IEEE Signal Processing Letters, Vol. 4, No. 4, pp. 112-114, 1997. 662 [3] K.Chinnasarn and C.Lursinsap. Effects of Learning Parameters on Independent Component Analysis Learning Procedure, Proceedings of the 2nd International Conference on Intelligent Technologies, Bangkok/Thailand, pp. 312-316, 2001. 661, 662, 666 [4] A.Cichocki and S.-I.Amari. Adaptive Blind Signal and Image Processing: Learning Algorithms and Applications, John Wiley & Sons, Ltd., 2002. 661, 662, 663 [5] M.Girolami. An Alternative Perspective on Adaptive Indepndent Component Analysis Algorithms, Neural Computation, Vol. 10, No. 8, pp. 201-215, 1998. 662 [6] Haykin.S. Neural Network a Comprehensive foundation. 2nd, Prentice Hall,1999. [7] A.Hyvarinen and E.Oja. Independent Component analysis: algorithms and applications, Neural Networks. vol. 13 pp. 411-430, 2000. 661, 662, 664 [8] H. K.Kwan. Simple Sigmoid-like activation function suitable for digital hardware implementation, Electronics Letter Vol.28, no.15, pp.1379-1380, 1992. 663

Low Complexity Functions for Stationary Independent Component Mixtures

669

[9] T.-W.Lee, M.Girolami and T. J.Sejnowski. Independent Component Analysis Using an Extended Informax Algorithm for Mixed Sub-Gaussian and SuperGaussian Sources Neural Computation, Vol. 11, No. 2, pp. 409-433, 1999. 662

Knowledge-Based Automatic Components Placement for Single-Layer PCB Layout Tristan Pannérec Laboratoire d'Informatique de Paris VI Case 169, 4 place Jussieu, 75005 Paris, France [email protected]

Abstract. Printed Circuit Board layout involves the skillful placement of the required components in order to facilitate the routing process and minimize or satisfy size constraints. In this paper, we present the application to this problem of a general system, which is able to combine knowledge-based and search-based solving and which has a meta-level layer to control it's reasoning. Results have shown that, with the given knowledge, the system can produce good solutions in a short time.

1

Introduction

The design of a PCB (Printed Circuit Board) for a product goes through three main stages. The first one is the designing of the logical scheme, which defines the components used (part list) and their interconnections (net list). The second one is the layout of the PCB with the definition of positions for all components and tracks, which link the components together. Generally, these two steps (components placement and routing) are carried out sequentially. The last stage is the industrial production, where the PCB are obtained by chemical processes and components are soldered. In this paper, we are interested in the components placement stage. The problem is to find a position on the board and a direction for each component in such a way that the components do not overlap and that the placement allows a good routing result. A good routing result means that all wires are taken on and that tracks are shortest. Depending on the nets, the length of the tracks can be of varying importance. For example, unprotected alimentation wires or wires between quartz and microcontrollers must absolutely be short for physical reasons (the quality of the product will depend on these lengths). A secondary objective during the component placement step can be to minimize the surface of the PCB if no predefined form is given. While the routing step is automatically done for a long time in most CAD tools, fully automated components placement is a more recent feature. Most applications thus allow “interactive” placement, where the user can instantly see the consequence of placing a component in a given position, but few can achieve good automatic placement, especially for single-layer PCB, where the problem is much harder. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 669-675, 2003.  Springer-Verlag Berlin Heidelberg 2003

670

Tristan Pannérec

In this paper we report on the use of our MARECHAL framework for solving this complex problem of automatic component placement for a single-layer PCB (the method works of course for multi-layer PCB as well). The rest of this paper is organized as follows: the second section presents some related work. Then, the MARECHAL framework is described briefly in section 3. Section 4 is dedicated to the knowledge we have given to this framework and section 5 presents the results we have achieved. Some final conclusions are drawn in section 6.

2

Related Work

The placement of components is a well-known problem, which has been extensively studied. Research has mainly focused on the VLSI design. The methods used for VLSI layout can be divided into two major classes [1]: constructive methods and iterative improvement methods. The first group (also referred to as global or relative placement) builds a global placement derived from the net-cell connections. In this class, we can find the force-directed (analytical) methods [3] and the min-cut (partitioning-based) methods [2]. The iterative methods start with an initial random placement and try to improve it by applying local changes, like reflecting a cell or swapping two cells. Deterministic methods or randomized methods (either with Simulated Annealing [9] or Genetic Algorithm [11]) belong to this class. Hybrid methods have also been investigated [4] but are limited to the use of a constructive method to produce the initial placement for an iterative method. In our approach, the two processes are more integrated. Little academic work addresses specifically the PCB framework, although the problem is broadly different from the VLSI one. There are not hundreds of thousands of cells, which have all the same shape and have to be organized in predefined rows. Instead, we have from ten to a few hundred components, with very variable sizes and numbers of pins (from 2 to 30 or more). Components cannot be modeled as points as in the common VLSI methods and the objective function has to be more precise. In the VLSI layout, objective functions use the semi-perimeter method [10] to estimate wiring length and density estimation to avoid crossing between wires. In a single layer PCB, these approximations are not precise enough and more complex models have to be considered, as we will see in the next section. But these more complex models prevent the use of classical constructive methods and lead to more CPUconsuming objective functions, which also prevent the use of classical iterative approaches (we have for example tested a genetic algorithm with our objective function but, even after 2 hours of computation, the function was not correctly optimized and the solution was of no interest). We use an objective function based on the minimum spanning tree model (MST) for each net (cf. Fig. 1 and Fig. 4). On the basis of these trees, the evaluation function is the sum of two terms. The first term is the weighted sum of all segment lengths (the weights depend on the wire type). The second term represents the crossing value: when two segments cross, a value is added to the evaluation function depending on the a priori possibilities to resolve the crossing and proportionally to the avoiding path if it can be easily computed (cf. Fig. 2). The resulting objective function takes a long time to compute but is more precise.

Knowledge-Based Automatic Components Placement for Single-Layer PCB Layout

671

Fig. 1. Example of a MST for a 5-pins net

Fig. 2. Example of importance of crossing

3

The Marechal System

The MARECHAL system is a general problem solver, which uses domain-dependent knowledge. For the moment, it is limited to optimisation problems where the goal is to coordinate numerous concurrent decisions to optimise a given criterion (e.g. the evaluation function defined in the previous section). Except to automatic component placement, it is also currently applied to a complex semi-concurrent game [7] and timetabling. The approach materialized in the MARECHAL framework is first of all based on three principles. The first one is the integration of a knowledge-based solving with a search-based solving [6]. The system uses domain-dependent knowledge to select the best a priori choices, but it can also question these choices to explore the search space. The second principle is to use abstract reasoning levels to limit the search to coherent solutions. The problem is thus decomposed into sub-problems (thanks to domainspecific knowledge) and for each decomposition level, the system can make abstract choices (choices about intentions, stage orders…). The third principle is to use autoobservation to control the solving process. By means of domain-dependent and domain-independent meta-level rules, the system can analyse it's own reasoning and determine how to improve the current solution [5]. Instead of randomly trying changes to the current solution, it can thus limit the search to interesting improvement possibilities. These three important features of our approach allow dealing with complex problems where pure search methods cannot be applied (because of huge

672

Tristan Pannérec

search space and irregular time-consuming objective functions) and where good specific heuristics cannot be found for all possible instances of the problem. The MARECHAL system materializes this approach, by providing a kind of an interpreter to a specific language used to define the domain-specific knowledge. It is build on a meta-level architecture: the first part is the basic system, which is able to construct solutions, and the second part is the meta level, which is responsible of managing and monitoring the solving process by observing the first part [8]. To achieve that, a bi-directional communication mechanism is used: on the one hand, the basic system sends notifications and traces about what it is doing and what has happened (a sub-problem has been successfully solved…) and on the other hand, the supervision layer sends orders or recommendations, with eventually new constraints or objectives to precisely drive the resolving process.

4

Knowledge

In this section, we describe in more detail the application of the MARECHAL system to component placement, with a particular emphasis on the knowledge given to solve this problem. The knowledge allows decomposing the problem (decomposition expertise), to construct a solution for each sub-problem (construction expertise) and to intelligently explore the solution space by improving the solution already constructed (improvement expertise). 4.1

Decomposition and Construction Knowledge

As regards the other applications, the decomposition knowledge for the PCB layout problem is simple. The main problem is decomposed into two stages. First, the system tries to generate macro-components by looking for repeated patterns in the circuit. Components belonging to a macro-component are then placed locally and replaced by a single new component. This stage is especially useful for large circuits. The second stage is the main placement stage. If we have n components, n calls to the “Single Component Placement” (SCP) sub-problem are made, according to the best a priori order. Heuristics used to define this order are based on the number of connections and the size of the components (big components with many connections are set up first). The SCP sub-problem first chooses an already placed "brother" component and tries to place the new component close to this brother by scanning the local area (the last point is the most complex one, because it poses difficult perception problems). To find a ‘brother” component, the heuristics emphasis on the components which share many connections. Heuristics used to choose the concrete positions are based on simulation of the local “rat nest” that will be computed for the component to be placed. 4.2

Improvement Knowledge

With the above knowledge, the system is thus able to construct a solution but, as the choices are done a priori (next component to consider, position of the components…),

Knowledge-Based Automatic Components Placement for Single-Layer PCB Layout

673

they can be proved wrong after some attempts, when a future component cannot be correctly placed. That is why a priori choices have to be questioned by means of improvement tests, which will allow a partial exploring of the solution space. Two types of improvement are tested by the system for the automatic component placing application. The first type are called “immediate tests” because they are executed as soon as the possibilities are discovered, without waiting for a full solution. For example, if a component Cx cannot be placed next to Cy (but where it would be well placed) because of the presence of Cz, the position of Cz is immediately questioned before placing any other component: the system tries to place Cz in another place that will allow to place Cx next to cy. For example in Fig. 3, RL3 is placed first, but then U6 cannot be placed close to pre-placed component B4, although a short connection is required between U6 and B4. The position of RL3 is thus put into question to obtain the layout on the right, which is better. Then, the placement process go on with a new component. A total of seven rules produce improvement possibilities of this type.

Fig. 3. Example of immediate improvement

The second type of improvement acts after having constructed a full solution. When all components have been placed, the system tries to modify the worst placed component or to minimize the size of the circuit by generating new form constraints. The process stops after a fixed amount of time and the best-constructed solution is sent to the CAD program.

5

Results

Our system has been tested on several circuits involving from 20 to 50 components and with one to three minutes to produce each solution. Two examples of a solution produced by the system for simple circuits are given in Fig. 4. The figure displays the associated MST: the bold lines represent connections that have to be especially minimized. In all cases, the solutions produced by the systems had a better value than those produced manually by a domain expert, in terms of the evaluation function presented in section 3 (cf. Fig. 5). Of course, this function is not perfect, as only a full routing process can determine if the placement is really good.

674

Tristan Pannérec

Fig. 4. Examples of solution produced by the system

Objective function

MARECHAL

human expert

160000 140000 120000 100000 80000 60000 40000 20000 0 #1

#2

#3

#4

#5

#6

Circuits

Fig. 5. Comparison of objective function values obtained by our system and expert humans (the value have to be minimized)

That's why the solutions have also been evaluated by a domain expert, who was asked to build a final solution (with the tracks) from the solution produced by the system. It appeared that only minor changes were necessary to obtain a valid solution. In addition, these manual routing processes did not use more "straps" (bridges to allow crossings) than manual routing operated on the human placements.

6

Conclusion

In this paper, we have presented the application of the MARECHAL system to automatic component placement for single-layer PCB layout. Thanks to the meta-

Knowledge-Based Automatic Components Placement for Single-Layer PCB Layout

675

level solving control and the combination of knowledge-based and search-based approaches, the system is able to produce good solutions in a short time, compared to human solutions. As the single-layer problem is more difficult than the multi-layer problem, the knowledge could be easily adapted to deal with multi-layer PCB design. More constraints, such as component connection order, could also easily be added. Future work will first consists in improving the current knowledge to further minimize the necessary human intervention. A more important direction is to implement an automatic router with the MARECHAL framework and to connect the two modules. Thus, the router will serve as an evaluation function for the placement module, which will be able to modify component positions according to the difficulties encountered in the routing process. The system will then be able to produce a full routed solution from scratch and without any human intervention.

References [1]

H.A.Y. Etawil: Convex optimization and utility theory: new trends in VLSI circuit layout. PhD. Thesis of the University of Waterloo, Canada, (1999). [2] D.J.H. Huang & A.B. Kahng: Partitioning-based standart-cell global placement with an exact objective: Inter. Symp. On Physical Design (ISPD) (1997), 18-25. [3] F. Johanes, J.M. Kleinhaus, G. Sigl & K. Antereich: Gordian: VLSI placement by quadratic programming and slicing optimization, IEEE. Trans. On CAD, 10(3) (1991), 356-365. [4] A. Kennings: Cell placement using constructive and iterative methods (PhD. Thesis of the University of Waterloo, Canada (1997). [5] T. Pannérec: Using Meta-level Knowledge to Improve Solutions in Coordination Problems, Proc. 21st SGES Conf. on Knowledge Based Systems and Applied Artificial Intelligence, Springer, Cambridge (2001) 215-228. [6] T. Pannérec: An Example of Integrating Knowledge-based and Search-based Approaches to Solve Optimisation Problems. Proc. of the 1st European STarting A.I. Researchers Symp. (STAIRS 02), p. 21-22, Lyon, France (2002). [7] T. Pannérec: Coordinating Agent Movements in a Semi-Concurrent TurnBased Game of Strategy. Proceedings of the 3rd Int. Conf. on Intelligent Games and Simulation (GameOn 02), p. 139-143, Harrow, England (2002). [8] J. Pitrat: An intelligent system must and can observe his own behavior. Cognitiva 90, Elsevier Science Publishers (1991), 119-128. [9] C. Sechen: VLSI placement and global routing using simulated annealing. Kluwer Academic Publishers (1988). [10] 10.C. Sechen & A. Sangiovanni: The TimberWolf 3.2: A new standard cell placement and global routing package: Proc. 23rd DAC, IEEE/ACM, Las Vegas (1986), 408-416. [11] 11.K. Shahookar & P. Mazumder: A genetic approach to standard cell placement using meta-genetic parameter optimization, IEEE. Trans. on Computers, 9(5) (1990), 500-511.

Knowledge-Based Hydraulic Model Calibration Jean-Philippe Vidal1 , Sabine Moisan2 , and Jean-Baptiste Faure1 1

Cemagref, Hydrology-Hydraulics Research Unit, Lyon, France {vidal,faure}@lyon.cemagref.fr 2 INRIA Sophia-Antipolis, Orion Project, France [email protected]

Abstract. Model calibration is an essential step in physical process modelling. This paper describes an approach for model calibration support that combines heuristics and optimisation methods. In our approach, knowledge-based techniques have been used to complement standard numerical modelling ones in order to help end-users of simulation codes. We have both identified the knowledge involved in the calibration task and developed a prototype for calibration support dedicated to river hydraulics. We intend to rely on a generic platform to implement artificial intelligence tools dedicated to this task.

1

Introduction

Simulation codes are scientific and technical programs that build numerical models of physical systems, and especially environmental ones. We are interested in river hydraulics where simulation codes are based on the discretisation of the simplified fluid mechanics equations (de Saint-Venant equations) that model streamflows. Theses codes have been evolving during the last forty years from basic numerical solvers to efficient and user-friendly hydroinformatics tools [1]. When a numerical model is built up for a river reach and its corresponding hydraulic phenomena (e.g., flood propagation), the model must be as representative as possible of physical reality. To this end, some numerical and empirical parameters must be adjusted to make numerical results match observed data. This activity – called model calibration – can be considered as a task in the artificial intelligence sense. This task has a predominant role in good modelling practice in hydraulics [2] and in water-related domains [3]. Users of simulation codes currently carry out model calibration either by relying on their modelling experience or by resorting to an optimisation code. The most widely used method is trial-and-error, which consists in running the code, analysing its outcomes, modifying parameters and restarting the whole process until a satisfactory match between field observations and model results has been reached. Few indications to carry out this highly subjective task in an efficient way are given in reference books [4] and experienced modelers follow heuristic rules to modify parameters. This task thus requires not only a high degree of expertise to analyse model results but also fast computers to perform numerous time-consuming simulation runs. V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 676–683, 2003. c Springer-Verlag Berlin Heidelberg 2003


Knowledge-Based Hydraulic Model Calibration

677

In order to overcome these difficulties, many automatic calibration methods have been developed over the last thirty years [5]. These methods rely on three main elements: an objective function that measures the discrepancy between observations and numerical results, an optimisation algorithm that adjusts parameters to reduce the value of the function, and a convergence criterion that tests its current value. The major drawback of this kind of calibration stands in the equifinality problem – as defined by Beven [6] – which predicts that the same result might be achieved by different parameter sets. Consequently, local minima of the objective function might not be identified by the algorithm and thus lead to unrealistic models. Artificial intelligence techniques have recently been used to improve these automatic methods, with the use of genetic algorithms, alone [7] or combined with case-based reasoning systems [8]. Our objective is to propose a more symbolic approach for hydraulic model calibration support. The aim is to make hydroinformatics expert knowledge available for end-users of simulation codes; a knowledge-based system encapsulating the expertise of developers and experienced engineers can guide the calibration task. Contrary to Picard [9], we focused more on the operational use of simulation codes and not on their internal contents. The paper first presents the cognitive modelling analysis of calibration and hydroinformatics domain, then outlines the techniques used for the development of a knowledge-based system dedicated to model calibration.

2

Descriptive Knowledge Modelling

From a cognitive modelling perspective, we have first identified and organised the knowledge in computational hydraulics that may influence the model calibration task. Our approach was to divide this knowledge into generic knowledge, corresponding to concepts associated with numerical simulation of physical processes in general, and domain knowledge, corresponding to concepts specific to open-channel flow simulation. Concerning generic computational knowledge, we defined a model as a composition of data, parameters, and a simulation code (see Fig. 1). Following Amdisen [10], we distinguished static data from dynamic data. Static data are components of the model; they depend only on the studied site (e.g., in our application domain, topography and hydraulic structures). Dynamic data represent inputs and outputs of the simulation code; they include observed data (e.g., water levels measured during a flood) and corresponding computed data. Concerning domain knowledge, we extracted from river hydraulics concepts a hierarchy of bidimensional graphical objects which instantiate the previous meta-classes of data. Figure 2 shows a partial view of this hierarchy of classes and displays objects commonly used in 1-D unsteady flow modelling. For instance, CrossSection in Fig. 2 derives from StaticData in Fig. 1 and DischargeHydrograph may derive from any subclass of DynamicData.

678

Jean-Philippe Vidal et al.

Data

PhysicalEvent

1..*

1..*

StaticData

DynamicData

1 InputData

CalibrationInputData *

OutputData

PredictionInputData

CalibrationOutputData

*

uses for calibration

SimulationOutputData

* are compared to for calibration 1..*

1..*

uses for prediction SimulationCode

1

1..*

*

1..*

Model Parameters

1

1

produces *

1..*

Fig. 1. UML class diagram of generic computational knowledge

GraphicalObject

Curve

Point

DynamicCurve

StaticCurve

GivenReachCurve

StaticPoint

GivenSectionCurve

DynamicPoint

TimeLimitedPoint

{incomplete}

TimeExtendedPoint

{incomplete} {incomplete}

GivenTimeCurve

{incomplete} {incomplete}

WaterSurfaceProfile 0..1

RatingCurve

StageHydrograph 0..1

0..1 2..*

2..* WaterLevel

DischargeHydrograph

0..1

GaugingPoint

0..1

0..1

2..*

3..*

{incomplete}

1..* 1

CrossSection

0..1

1

Discharge

GroundPoint

Fig. 2. Simplified UML class diagram of graphical objects

Floodmark

Knowledge-Based Hydraulic Model Calibration

:Model [uncalibrated]

:DynamicData

solve inverse problem

679

:PredictionInputData

solve direct problem

preprocess

simulate

determine :SimulationOutputData

dispatch

define [else] initialise

[parameter definition = ok]

simulate

compare :Model [calibrated]

evaluate [model calibration = unsatisfactory] [else]

Fig. 3. UML activity diagram of problem solving in modelling

3

Operative Knowledge Modelling

The second category of knowledge concerns the problem solving strategy of the calibration task. Modellers of physical processes may face two situations [11]: solving a direct problem, which means computing the output of a model, given an input, or solving an inverse problem, which means finding out the model which transforms a given input into a given output. The first situation occurs during productive use of simulation codes. In hydraulics, the second situation amounts to determine the model parameters – since hydraulic modelling follows deterministic laws – and corresponds in fact to a calibration task. Figure 3 shows the relation between these two situations – calibration must of course be performed before using simulation codes for prediction – and emphasises the first-level decomposition of both modelling problems. We determined the problem solving strategy of this task essentially from interviews of experts, since literature on calibration task is very scarce. We identified six main steps, listed in Fig. 3. We detail them below for a typical case of unsteady flow calibration of a river reach model: 1. dispatch: Available dynamic data are dispatched among calibration input and output data. Boundary conditions compose the core of calibration input data

680

2. 3. 4. 5.

6.

4

Jean-Philippe Vidal et al.

(e.g., a flood discharge hydrograph determined at the upstream cross-section and the corresponding stage hydrograph at the downstream cross-section). Field observations like floodmarks (maximum flood water levels) measured along the reach constitute calibration output data. define: Generally, only one model parameter is defined at first: here, a single riverbed roughness coefficient for the whole reach; initialise: Parameter default value(s) is(are) set up, often thanks to literature tables; simulate: The code is run to produce simulation output data, usually in the form of water-surface profiles for each time step; compare: Simulation output data are compared to calibration output data, e.g., by putting maximum envelops of water-surface profiles and floodmarks side by side; evaluate: Model error is judged; an unsatisfactory calibration leads to adjust coefficient value (back to point 3) or to give another definition of model parameters, e.g., to take into account a spatially distributed roughness coefficient (back to point 2).

Prototype for Calibration Support

We intend to rely on a platform for knowledge-based system design that has been developed at Inria and applied to program supervision [12]. Program supervision is a task that emulates expert decisions in the skilled use of programs, such as their choice and organisation, their parameter tuning, or the evaluation of their results. Like calibration, this task involves activities such as initialise parameters, run codes, evaluate results, etc. Both tasks also use very similar concepts, such as the common notion of parameters. We plan to take advantage of these similarities to reuse – with slight modifications – both an existing engine and the attached knowledge description language, named Yakl. The previous sections provide us with a conceptual view of the calibration task, which serves as a basis to determine the required modifications. From this conceptual view it is possible to derive modifications in both the language to describe the task concepts and the inference engine to manipulate them. For instance, while initialise and evaluate are existing reasoning steps in the program supervision task, dispatch is a new one that should be added for the calibration task. In parallel to this conceptual view, we also conducted an experiment to implement a calibration knowledge-based system, by using the program supervision language and engine. This experiment allowed us to refine the specifications for new tools dedicated to calibration task. The platform will support the implementation of these tools by extension of program supervision ones. Two concepts provided by the program supervision task have proven useful for our purpose: operators and criteria. Operators correspond to actions, either simple ones (e.g., run a code) or composite ones, which are combinations of operators (sequence, parallel or alternative decompositions) that solve abstract processing steps. Operators have arguments corresponding to inputs/outputs

Knowledge-Based Hydraulic Model Calibration

Initialisation criteria Rule { name determine base value for gravel comment ”Determination of base value for gravel roughness coefficient” Let obs an observation If obs.bed material == gravel Then nb.lower := 0.028, nb.upper := 0.035 }

681

Choice criteria Rule { name select flood dynamics operator comment ”Selection of operator for flood dynamics modelling” Let s a study If s.phenomena == flood dynamics Then use operator of characteristic unsteady flow }

Fig. 4. Examples of criteria and attached rules in Yakl syntax of actions. Both kinds of operators also include various criteria composed of production rules, capturing the know-how of experts on the way to perform actions. For instance, in program supervision, criteria express how to initialise input argument of programs (initialisation criteria), to evaluate the quality of their results (assessment criteria), to modify parameter values of the current operator (adjustment criteria), or to transmit failure information in case of poor results (repair criteria). Additional criteria are related to composite operators (for choices among sub-operators or optional applications of sub-operators in a sequence). This range of criteria mostly suits our needs. Figure 4 shows examples of rules that belong to our prototype knowledgebased calibration system. The first rule is connected to the initialise step and expresses a conversion from an observation of bed material nature to an appropriate range of numerical values for roughness coefficient. The second rule is related to the simulate step. It helps to select an appropriate sub-operator for flood dynamics modelling, in the line of current research on model selection [13].

5

Conclusion and Perspectives

We have currently achieved an important phase which was the specification of artificial intelligence tools dedicated to the calibration task, with a focus on its application to hydraulics. Our approach is based on a conceptual view of the task and on a prototype knowledge-based system. A generic platform will support the implementation of these tools by extending program supervision ones. The specifications of the new calibration engine, following the problem solving method in Fig. 3, has been completed and the implementation is under way. We reuse the notions of operators and rules from program supervision for the simulate, initialise and compare subtasks of calibration. Checking the resulting system with real data will allow us not only to test calibration results but also to improve the encapsulated knowledge which should mimic an experienced modeller reasoning in all situations. We intend to use existing river models (e.g., on Ard`eche river) that have been built up with different simulation codes (Mage, Hec-Ras). We will compare the problem solving processes with and without support: since simulation code use is difficult to grasp, we expect that a knowledge layer should reduce the time spent by end-users without impairing the quality of the results.

682

Jean-Philippe Vidal et al.

The presented approach allows experts to share and to transmit their knowledge and favours an optimal use of simulation codes. The complementarity between numerical and symbolic components makes the resulting system highly flexible and adaptable. Data flow processing is thus automated while taking into account the specificities of each case study. Our approach appears to be a relevant alternative to current automatic calibration methods, since it includes knowledge regarding both physical phenomena and numerical settings, and may thus avoid equifinality pitfalls. As we have already mentioned, this approach, while focusing on hydraulic domain, is general and can be applied with minimal modifications to any physical process modelling that requires a calibration.

References [1] Faure, J. B., Farissier, P., Bonnet, G.: A Toolbox for Free-Surface Hydraulics. In: Proceedings of the fourth International Conference on Hydroinformatics, Cedar Rapids, Iowa, Iowa Institute of Hydraulic Research (2000) 676 [2] Abbott, M. B., Babovic, V. M., Cunge, J. A.: Towards the Hydraulics of the Hydroinformatics Era. Journal of Hydraulic Research 39 (2001) 339–349 676 [3] Scholten, H., van Waveren, R. H., Groot, S., van Geer, F. C., W¨ osten, J. H. M., Koeze, R. D., Noort, J. J.: Good Modelling Practice in Water Management. In: Proceedings of the fourth International Conference on Hydroinformatics, Cedar Rapids, Iowa, Iowa Institute of Hydraulic Research (2000) 676 [4] Cunge, J. A., Holly, F. M., Verwey, A.: Practical Aspects of Computational River Hydraulics. Volume 3 of Monographs and Surveys in Water Resources Engineering. Pitman, London, U. K. (1980) 676 [5] Khatibi, R. H., Williams, J. J. R., Wormleaton, P. R.: Identification Problem of Open-Channel Friction Parameters. Journal of Hydraulic Engineering 123 (1997) 1078–1088 677 [6] Beven, K. J.: Prophecy, Reality and Uncertainty in Distributed Hydrological Modeling. Advances in Water Resources 16 (1993) 41–51 677 [7] Chau, K. W.: Calibration of Flow and Water Quality Modeling Using Genetic Algorithm. In McKay, B., Slaney, J. K., eds.: Proceedings of the 15th Australian Joint Conference on Artificial Intelligence. Volume 2557 of Lecture Notes in Artificial Intelligence., Canberra, Australia, Springer (2002) 720 677 [8] Passone, S., Chung, P. W. H., Nassehi, V.: The Use of a Genetic Algorithm in the Calibration of Estuary Models. In van Harmelen, F., ed.: ECAI 2002 - Proceedings of the Fifteenth European Conference on Artificial Intelligence. Volume 77 of Frontiers in Artificial Intelligence and Applications., Lyon, France, IOS Press (2002) 183–187 677 [9] Picard, S., Ermine, J. L., Scheurer, B.: Knowledge Management for Large Scientific Software. In: Proceedings of the Second International Conference on the Practical Application of Knowledge Management PAKeM’99, London, UK, The Practical Application Company (1999) 93–114 677 [10] Amdisen, L. K.: An Architecture for Hydroinformatic Systems Based on Rational Reasoning. Journal of Hydraulic Research 32 (1994) 183–194 677 [11] Hornung, U.: Mathematical Aspects of Inverse Problems, Model Calibration, and Parameter Identification. The Science of the Total Environment 183 (1996) 17–23 679

Knowledge-Based Hydraulic Model Calibration

683

[12] Thonnat, M., Moisan, S., Crub´ezy, M.: Experience in Integrating Image Processing Programs. In Christensen, H. I., ed.: Proceedings of the International Conference on Computer Vision Systems, ICVS’99. Volume 1542 of Lecture Notes in Computer Science., Las Palmas, Spain, Springer (1999) 200–215 680 [13] Chau, K. W.: Manipulation of Numerical Coastal Flow and Water Quality Models. Environmental Modelling and Software 18 (2003) 99–108 681

Using Artificial Neural Networks for Combustion Interferometry Victor Abrukov, Vitaly Schetinin, and Pavel Deltsov Moskovsky prosp, 15, Chuvash State University 428015, Cheboksary, Russia [email protected]

Abstract. We describe an application of artificial neural networks (ANNs) for combustion interferometry. Using the ANN we calculate the integral and local characteristics of flame presented by using incomplete set of features that characterize interferometric images. Our method performs these calculations faster than the standard analytical approaches. In prospects, this method can be used in automated systems to control and diagnose combustion processes.

1

Introduction

Interferometry has wide possibilities in combustion research. It allows to determine simultaneously the local characteristics, such as temperature or density field in flame as well as the integral characteristics, such as mass of flame, Archimedean force acting upon flame, quantity of heat in flame, etc [1 - 3]. Among other integral characteristics, which can be extracted from data of interferometric images, are non-stationary mass burning rate and heat release power during ignition, heat release rate, the force of powder, changing of mechanical impulse of non-stationary gas flow, mechanical impulse of arise flow, profile of heat release rate in the stationary wave of burning, etc. [1, 3]. However, for the determination of the mentioned characteristics, first of all, it is necessary to measure a lot of values (big set discrete values) of the function of the phases difference distribution on the interferogram plane, S(x,y). The results often have a subjective nature and depend greatly on the experience of a user in the interferometric measurement. Besides this, the measurement of a lot of values of S(x,y) is labor-intensive. These circumstances confine the level of usage of interferometry in the quantitative analysis of combustion. We can mark here also that, from the point of view of interferometric requirements, the possibilities of determination of mentioned characteristics of flame exist only in the case of ideal conditions of interferometric experiment. There are a lot of real conditions of experiment when it is impossible to measure accurately the S(x,y) and to implement the wide possibilities of interferometry. In any cases, after the accurate measurement a lot of values of S(x,y), it is necessary to make the integration of S(x,y) in a plane of the interferogram (a direct task dealing with the determination of integral characteristics) or to make the differentiation of S(x,y) (an inV. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 684-690, 2003.  Springer-Verlag Berlin Heidelberg 2003

Using Artificial Neural Networks for Combustion Interferometry

685

verse task dealing with the determination of local characteristics). As one can see, the full realization of the all possibilities of interferometry demands the solving of some not simple problems the decision of which is not ever possible. In this paper we describe examples of methods of application of artificial neural networks (ANNs) for solving of problems of combustion interferometry in real condition of experiments. The main problem established in our work was how to determine the integral and local characteristics of flame without necessity of measurement of complete set of data about features of interferometric image, first of all, without necessity of measurement of a lot of values of S(x,y), e.g. by means of an incomplete set of values of S(x,y) (the case of determination of local characteristics of flame) or even without them (the case of determination of integral characteristics of flame). This problem is very important for solving of problems of combustion research in real condition of experiments as well as for creation of automated systems of diagnostics and control of combustion processes.

2

The Artificial Neural Network Method

Using ANNs we can solve problems, which have no algorithmic solutions. The ANNs are applicable when problems are multivariate and relationships between variables are unknown. The ANN methods can induce realistic models of complex systems from experimental data. In the special cases, the ANNs are able to induce relations between variables, which can be represented in a knowledge-based form [4]. 2.1

Integral Characteristics of Flame. Direct Task

The integral characteristics that we consider here are the mass of flame, m, the Archimedean lifting force acting on flame, Fa, and quantity of heat in flame, H. We use the following geometrical parameters of the interferometric images: the maximal height, h, and width of the image, w, its square, s, and perimeter, e. We consider these parameters as an incomplete set of features characterizing interferometric images, because, in usual case, we should measure the complete set of value of S(x,y). In our work, we used a feed-forward ANN Neural Networks Wizard 1.7 (NNW) developed by BaseGroupLabs [4]. For training the NNW, we used some the combinations of the above listed features and integral characteristics. The integral characteristics that we have to calculate are assigned as the target variables. Under these assumptions we aim to train the NNW on the input data to calculate the desirable integral characteristics of flames. The number of hidden neurons was 8 and the learning rate was 0.1. The learning stops when the classification error is less than 0.01. Table 1 presents six sets of the integral characteristics (m, Fa, H) determined by usual numerical method for six interferometric images of flame as well as six sets of corresponding geometrical parameters of the interferometric images (h, w, s, e).

686

Victor Abrukov et al.

Table 1. Sets of values of the features characterizing interferometric images (images geometrical parameters) and integral characteristics of flames

№ 1 2 3 4 5 6

w, cm, 10-2 65 88 108 130 142 151

h, cm, 10-2 65 85 147 222 274 341

S, cm2

e, cm

0,32 0,62 1,26 2,25 2,98 3,91

1,99 2,77 4,08 5,61 6,68 8,11

m, g, 10-3 0,08 0,22 0,52 1,25 1,90 2,67

Fa, dynes

H, J

0,08 0,22 0,56 1,06 1,42 1,90

0,024 0,070 0,175 0,324 0,430 0,575

The geometrical parameters were as input data and the integral characteristics (m, Fa, H) were as output data. Five sets were used for training the ANN, and set N 4 was used for testing the ANN. We used 12 various combinations of the geometrical parameters for each set of integral characteristics. The one of results of testing are shown on Fig.1. The horizontal line in Fig. 1 is the target value. The vertical columns correspond to the calculated outputs of the ANN. Each column corresponds to a unique set of geometrical parameters. For example, the term hse means that the height h, the square s as well as the perimeter e were used as the input data. This example shows that the ANN can calculate the integral characteristics of flame successfully. Analyzing this result, we can see that the accuracy of calculation is dependent on the combination of features. For example, if the ANN uses a combination of height and perimeter of image, then the error is minimal. And contrary, the error is large if it is combined width and height of images. More detailed analysis shows, that the combinations of features, which include width and square of images give a large error. On the other hand, less error is achieved if we use the combinations height and perimeter of the image. So we conclude that the values of height and perimeter of the image are more essential for calculation of integral characteristics of flame than width and square of images. Fig.2 depicts the significance of the geometrical parameters for calculation of integral characteristics. The first column corresponds to an average error of calculations for combinations, which include value of width of images. The second column shows an average error of calculation when we use the combinations that include value of height, etc.

Quantity of heat, J

0,4 0, 35 0,3 0, 25 0,2 0, 15 0,1 0, 05 0

e

h

he

w

wh

w he

hs

h se

s

se

w h s w h se

V a rio us co mbina tio ns of the e ntra nce da ta

Fig. 1. Results of calculation of heat quantity of heat for different combinations of features

687

1 0 ,0 % 8 ,3 % 8 ,0 %

6 ,8 %

6 ,0 % 4 ,4 %

4 ,3 %

%

Average error of the calculation,

Using Artificial Neural Networks for Combustion Interferometry

4 ,0 % 2 ,0 % 0 ,0 %

w

h

s

e

Fig. 2. Errors of calculation for geometrical parameters and integral characteristics of flames

Fig. 2 shows that using height and perimeter of images we can improve the accuracy by 1.5 than using square of images and by 2 than width. Our results show that in order to calculate the integral characteristics of flame we have to use the following combinations of geometrical parameters of images: e, h, he, whe, hs. In some cases we are interesting in an analytical representation of the relations induced from experimental data representing the inputs and outputs of real systems. Below we describe an analytical model of flame mass for ignition of powder by laser radiation, which was considered. m = 0.0534 + 1.5017x1 + 2.3405x2 - 2.5898x3 - 0.3031x4 ,

(1)

where x1, …, x4 are normalized width, height, square and perimeter of flame image, respectively, xi = zi/max(zi), and zi is current value, max(zi) is maximum rating of a variable. The ANN model has no hidden neurons. A standard least squared error method was used during creation of ANN model. Obviously that we can interpret the coefficients in Equation 1 as the contribution of variables x1, …, x4 to the value of m. This Equation is an example of the interferometric diagnostics model.

2.2

The Local Characteristics of Flame. Optical Inverse Problem

In the section we apply Neural Networks Wizard 1.7 (NNW) for solving an inverse problem of optics and demonstrate an example of calculation of a refractive index distribution in flame. We consider a determination of integrand meanings on basis of incomplete integral meanings and exam Abel integral equation for the case of cylindrical symmetry. The main problem is how to solve the inverse problem using an incomplete set of features presenting the phase difference distribution function in an interferogram plane. We can use only meaning of the integral, which allows us to calculate all the meanings of integrand, the refractive index distribution and temperature distribution in flame. To solve this problem we can write a dimensionless Abel's equation: 2 1− p

S( p) = 2

∫ (n 0

0

− n (r )) dz

688

Victor Abrukov et al.

where z2 + p2 = r2, z is a ray's path in the object, p is an aim distance (0 < p < 1), and r is a variable radius, see Fig.3. p 1 p

r 1

z

Fig. 3. Geometrical interpretation

Using this equation, we can then calculate the integrals S(p) from different integrands of form nо – n(r) = 1 + ar – br2, where a and b are the given constants. In total we used seven different integrands that reflect a real distribution of refractive index distribution in flames. The training data were as follows. For various r, values nо – n(r) were calculated. Then the integrals of each of these seven integrands were calculated. For example, for the function nо – n(r) = 1 + 4r – 5r2, we can write the following expression 1 + 1− p 2  8 10  S(p ) = 1 − p 2 ⋅  − p 2  + 2p 2 ⋅ ln 3 3  1− 1− p2 .

Here values of S(p) are calculated for different p. The input data for training the NNW were values of S(p), p and r. Values of integrand corresponding to each radius are the target values. In total, we used about 700 sets of these parameters. The number of hidden layers was 5, the number of neurons in them was 8, the learning rate was 0.1, the condition when the maximum errors during both learning and training is less than 0.01 were choused as the stopping conditions. The results of testing the NNW for an integrand nо – n (r) =1 + 0.5r – 1.5r2 are represented in Fig. 4.

Fig. 4. Relative errors of ANN integrand calculation

Relative errors of integrand calculated in the interval of p and r between 0.1 up to 0.9 were lower than 6%. We have executed also the complete LV1 (leave-one-out)

Using Artificial Neural Networks for Combustion Interferometry

689

procedure in the computational experiment. The average errors over all seven tests were in the narrow range from 5% to 6%. More fullness train-test experiment with using different initial weights of neural networks will be done in the nearest future. We also explored the robustness of our calculation to errors in the input data. The calculation error increases by 3 % per 2% of noise added to the input data. This means a good stability of our calculation. In total, the results show that the ANN can solve the inverse problem with an acceptable accuracy in a case of cylindrical symmetry. The definition of one integral value (and also it changes during time in case of non-stationary objects) can be automated. Therefore it is possible to use an ANN cheap (microprocessor) embedded into automated control systems. Further perspectives of the work are concerned with the implementation of neural networks for solving the inverse optical problems for other optical methods and also solving the practical problems of control of combustions processes.

3

Conclusions

The ANN applied for combustion interferometry performs well. Despite an incomplete set of features characterizing flame images, we can solve both the direct and inverse optics problems required for diagnostics of combustion processes. The main advantages of the ANN application are: 1.

2.

3. 4.

It will be possible to calculate the distribution of local thermodynamic characteristics, including density of separate components, by means of measuring in one point of plane of signal registration, for example in the case of laser-diode technique [5]. The ANN method does not require additional real experiments for solving of inverse problems. ANN model for solving of inverse problems can be obtained by means a knowledge base created with using a enough simple numerical calculation as well as there are possibilities of realizing of this approach for solving of the direct task (determination of integral characteristics of flame and other object including industry objects. The method can be extended for other optical methods, for which data are integrated on a line of registration. For diagnostics goals, the ANN can be applied to analysis optical images and signals of flows various types of engine, in particular in a very promising pulse detonation engine [5].

So, we conclude that the ANN method allows us to extend significantly the possibilities of optical diagnostics techniques and improve control systems of combustion processes as well as other industry processes.

690

Victor Abrukov et al.

References [1] [2] [3] [4] [5]

Abrukov, V.S., Ilyin, S.V., Maltsev, V.M., and Andreev I.V.: Proc. VSJ-SPIE Int. Conference, AB076 (1998). http://www.vsj.or.jp/vsjspie/ Abrukov, V.S., Andreev, I. V., Kocheev, I. G.: J. Chemical Physics 5 (2001) 611 (in Russian) Abrukov V.S., Andreev I.V., Deltsov P.V.: Optical Methods for Data Processing in Heat and Fluid Flow. Professional Engineering Publishing, London (2002) 247 The BaseGroup Laboratory http://www.basegroup.ru Sanders S.T., Jenkins T.P., Baldwin J.A. at al.: Control of Detonation processes. Moscow, Elex-KM Publisher (2000) 156-159

An Operating and Diagnostic Knowledge-Based System for Wire EDM Samy Ebeid1, Raouf Fahmy2, and Sameh Habib2 1

Faculty of Engineering, Ain Shams University, 11517 Cairo, Egypt [email protected] 2 Shoubra Faculty of Engineering Zagazig University, Egypt

Abstract. The selection of machining parameters and machine settings for wire electrical discharge machining (WEDM) depends mainly on both technologies and experience provided by machine tool manufacturers. The present work designs a knowledge-based system (KBS) to select the optimal process parameters settings and diagnose the machining conditions for WEDM. Moreover, the present results supply users of WEDM with beneficial data avoiding further experimentation. Various materials are tested and sample results are presented for improving the WEDM performance.

1

Introduction

Wire electrical discharge machining (WEDM) Fig.1 is a special form of electrical discharge machining wherein the electrode is a continuously moving conductive wire. The mechanism of material removal involves the complex erosion effect from electric sparks generated by a pulsating direct current power supply. With WEDM technology, complicated difficult-to-machine shapes can be cut. The high degree of accuracy obtainable and the fine surface finishes make WEDM valuable [1]. One of the serious problems of WEDM is wire breakage during the machining process. Wire rupture increases the machining time, decreases accuracy and reduces the quality of the machined surface [2,3,4]. WEDM is a thermal process in which the electrodes experience an intense local heat in the vicinity of the ionized channel. Whilst high erosion rate of the workpiece is a requirement, removal of the wire material leads to its rupture. Wire vibration is an important problem and leads to machining inaccuracy [5,6,7]. There is a great demand to improve machining accuracy in WEDM as it is applied to the manufacture of intricate components. The term knowledge engineering has been used quite recently and is now commonly used to describe the process of knowledge-based-systems [8,9]. The knowledge base is obviously the heart of any KBS. The process of collecting and structuring knowledge in a problem domain is called data acquisition. As the user consults the expert systems, answers and results build up in the working memory. In order to make use of the expertise, which is contained in the knowledge base, the V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 691-698, 2003.  Springer-Verlag Berlin Heidelberg 2003

692

Samy Ebeid et al.

system must posses an inference engine, which can scan facts and rules, providing answers to the queries given to it by the user. Shells are the most widely available tools for programming knowledge. Using a shell can speed up the development of a KBS. Snoeys et al [10] constructed a KBS for the WEDM process. Three main modules were constructed namely: work preparation, operator assistance and fault diagnosis, and process control. The software has been built in PROLOG language. They concluded that the knowledge based system for WEDM improved the performance of the process. Sadegh Amalink et al [11] introduced an intelligent knowledge-based system for evaluating wire-electro erosion dissolution machining in a concurrent engineering environment using the NEXPERT shell. The KBS was used to identify the conditions needed to machine a range of design features, including various hole shapes of required surface roughnesses.

Fig. 1. Principle of WEDM

2

Experimental Work

The present experiments have been performed on a high precision 5 axis CNC WED machine commercially known as Robofil 290, manufactured by Charmilles Technologies Corporation. The Robofil 290 allows the operator to choose input parameters according to the material, height of the workpiece and wire material from a manual supplied by the manufacturer. The machine uses wire diameters ranging from 0.1 to 0.3 mm. However, the wire used in the present tests is hard brass 0.25 mm diameter, under a wire traveling speed ranging between 8-12 m/min. To construct the knowledge based system, the experimental work has been planned to be done on four categories of materials. The first one is alloy steel 2417 (55 Ni Cr Mo V7), while the second category includes two types of cast iron. The third material category is the aluminum alloys (Al-Mg-Cu and Al-Si-Mg 6061). Conductive composites are the fourth materials category where seven types are selected (see Table I).

3

Knowledge-Based System Design

The proposed WEDM knowledge-based system is based on the production rule system of representing knowledge. The system has been built in “CLIPS” shell

An Operating and Diagnostic Knowledge-Based System for Wire EDM

693

software. CLIPS is a productive development and delivery expert system tool which provides a complete environment for the construction of rule and/or object based expert systems [12]. In the CLIPS program the method of representing knowledge is the (defrule) statement instead of (IF…THEN) statements. A number of such (defrule) rules in the forward chaining method has been built into the proposed KBS. The general layout of the architecture of the proposed system for WEDM process is depicted in Fig.2. Four main modules are designed namely: machining parameters, WEDM machine settings, problem diagnosis and machine technology database. Table 1. Types of composite materials

Al 1 2 R 3 R 4 R 5 R 6 R 7 5 Values are in %. 3.1

Cu Gr R 10 4 4 4 4 (R = remainder)

SiC 10 15 30 -

SiO2 10 -

Al2O3 24 -

Zn R

System Modules

3.1.1 Machining Parameters Module This module includes the required workpiece information such as the material to be machined, height and the desired final machining quality. The model is based upon the following: CS = Vf * H

(1)

MRR = (dw+2Sb) * Vf * H

(2)

2

where: CS = cutting speed, mm /min. dw = wire diameter, mm. Sb = spark gap, mm. Vf = machining feed rate, mm/min. H = workpiece height, mm. MRR = material removal rate, mm3/min. 3.1.2 WEDM Machine Settings Module The machine settings module includes the required parameters for operating the machine such as pulse frequency, average machining voltage, injection flushing flow rate, wire speed, wire tension and spark gap. The equations of the above two modules are presented in a general polynomial form given by: P = a H4 + b H3 + c H2 + d H + e

(3)

694

Samy Ebeid et al.

Where, P denotes any of the KBS outputs as cutting speed, MRR, .. etc. The constants “a” to “e” have been calculated according to the type of material, height of workpiece and roughness value. 3.1.3 Problem Diagnosis Module The problems and faults happening during the WEDM process are very critical when dealing with intricate and accurate workpieces. Consequently, faults and problems must be solved soon in order to minimize wire rupture and dead times. The program suggests the possible faults that cause these problems and recommend the operators with the possible actions that must be taken. The correction of these faults might be a simple repair or a change of some machine settings such as feed rate, wire tension, voltage, etc.. 3.1.4 Machine Technology Database Module This module contains a database for recording and organizing the heuristic knowledge of the WEDM process. The machining technology data has been stored in the form of facts. The reasoning mechanism of the proposed KBS proceeds in the same manner as that of the operator in order to determine the machine settings. The required information of workpiece material type, height and required surface roughness is acquired through keyboard input by the operator. On the basis of these data, the KBS then searches through the technology database and determines the recommended WED machine settings. The flow chart of the KBS program is shown in Fig.3. The knowledge acquisition in the present work has been gathered from: 1.

Published data of WEDM manuals concerning machining conditions for some specific materials as steel, copper, aluminum, tungsten carbide and fine graphite. 1. Experimental tests: The system has been built according to: Number of workpiece materials = 12 (refer to section of experimental work), number of surface roughness grades = 2 (ranging between 2.2 to 4 µm depending on each specific material) and number of workpiece heights for each material = 5 (namely: 10, 30, 50, 70 and 90 mm).

Fig. 2. Architecture of KBS.

An Operating and Diagnostic Knowledge-Based System for Wire EDM

695

Fig. 3. Flow chart of KBS

4

Results and Discussion

Figure 4 shows a sample result-taken from the above twelve materials-for the variation of cutting speed with workpiece height for two grades of surface roughness namely Ra=3.2 and Ra=2.2 µm for alloy steel 2417. The results show that the cutting speed increases until it reaches its maximum value at 50 mm thickness, then begins to decrease slightly again for the two grades of roughness. The results also indicate that the values of cutting speed for Ra=3.2 µm are higher than those of Ra=2.2 µm. Nevertheless, the work of Levy and Maggi [13] did not show any dependence of roughness on wire feed rate. Figs.5 and 6 show the variation of material removal rate with cutting speed for alloy steel 2417 and Al 6061 respectively for two grades of roughness. Both charts indicate a linear increase in MRR with cutting speed. Fig.5 shows that the most optimal zone for machining steel 2417 lies between nearly 40-70 mm2/min (Ra=3.2 µm) and between 20-40 mm2/min (Ra=2.2µm) for workpiece thickness ranging from 10-90 mm. Whereas, for Al 6061 this zone shows higher values between 120-180 mm2/min (Ra=4 µm) and between 50-90 mm2/min (Ra=2.2µm) for thickness values from 10-60 mm.

696

Samy Ebeid et al.

The variation between spark gap and cutting speed for both alloy steel 2417 and Al 6061 is shown in Fig.7 for Ra=3.2 and 4 µm respectively. The spark gap increases directly with the increase of cutting speed thus leading to a wider gap. This result is in accordance with the work of Luo [14] for cutting speeds up to 200 mm2/min. It was realized during the tests that the variation in the spark gap with cutting speed at Ra=2.2 µm for both steel 2417 and Al 6061 was nearly negligible.

Fig. 4. Variation of cutting speed with workpiece thickness

Fig. 5. Variation of MRR with cutting speed for steel 2417

An Operating and Diagnostic Knowledge-Based System for Wire EDM

697

Fig. 6. Variation of MRR with cutting speed for Al 6061

Fig. 7. Variation of spark gap with cutting speed

5

Conclusions

The present results of the KBS supply users of WEDM with useful data avoiding further experimentation. The designed system enables operation and diagnosis for the WEDM process. The system proves to be reliable and powerful as it allows a fast retrieval for information and ease of modification or of appending data. The sample results for alloy steel 2417 and Al 6061 of the various twelve tested materials are presented in the form of charts to aid WEDM users for improving the performance of the process.

698

Samy Ebeid et al.

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

Lindberg, R.A., “Processes and Materials of Manufacture”, Published by Allyn and Bacon, Boston, (1990), 796-802 Rajurkar, K.P., Wang, W.M. and McGeough, J.A., “WEDM identification and adaptive control for variable height components”, CIRP Ann. of Manuf. Tech., 43, 1, (1994), 199-202 Liao, Y.S., Chu, Y.Y. and Yan, M.T., “Study of wire breaking process and monitoring of WEDM”, Int. J. Mach. Tools and Manuf., 37, 4, (1997), 555-567 Yan, M.T. and Liao, Y.S., ”Monitoring and self-learning fuzzy control for wire rupture prevention in wire electrical discharge machining”, Int. J. of Mach. Tools and Manuf., 36, 3, (1996), 339-353 Dauw, D.F. and Beltrami, I., “High-precision wire-EDM by online wire positioning control” CIRP Ann. of Manuf. Tech., 43, 1, (1994), 193-196 Beltrami, I., Bertholds, A. and Dauw, D., “A simplified post process for wire cut EDM”, J. of Mat. Proc. Tech., 58, 4, (1996), 385-389 Mohri, N., Yamada, H., Furutani, K., Narikiyo, T. and Magara, T., “System identification of wire electrical discharge machining”, CIRP Ann. of Manuf. Tech., 47, 1, (1998), 173-176 Smith, P., “An Introduction to Knowledge Engineering”, Int. Thomson Computer Press, London, UK, (1996) Dym, C.L. and Levitt, R.E., “Knowledge-Based Systems in Engineering”, McGraw-Hill, Inc., New York, USA, (1991) Snoeys, R., Dekeyser, W. and Tricarico, C., “Knowledge based system for wire EDM”, CIRP Ann. of Manuf. Tech., 37, 1, (1988), 197-202 Sadegh Amalink, M., El-Hofy, H.A. and McGeough, J.A., “An intelligent knowledge-based system for wire-electro-erosion dissolution in a concurrent engineering environment”, J. of Mat. Proc. Tech., 79, (1998), 155-162 Clips user guide, www.ghgcorp.com/clips, (1999) Levy, G.N. and Maggi, F., “WED machinability comparison of different steel grades”, Ann. CIRP 39, 1, (1990), 183-185 Luo, Y.F., “An energy-distribution strategy in fast-cutting wire EDM”, J. of Mat. Proc. Tech., 55, 3-4, (1995), 380-390

The Application of Fuzzy Reasoning System in Monitoring EDM Xiao Zhiyun and Ling Shih-Fu Department of Manufacturing and Production Engineering Nanyang Technological University, Singapore 639798

Abstract. EDM (electrical discharge machining) is a very complicated and stochastic process. It is very difficult to monitor its working conditions effectively as lacking adequate knowledge on the discharge mechanism. This paper proposed a new method to monitor this process. In this method, electrical impedance between the electrode and the workpiece was taken as the monitoring signal. Through analyzing this signal and using a fuzzy reasoning system as classifier, sparks and arcs were differentiated effectively, which is difficult when using other conventional monitoring methods. The proposed method first partitions the collected voltage and current trains into separated pulses using Continuous Wavelet Transform. Then apply Hilbert Transform and calculate electrical impedance of each pulse. After that, extract features from this signal and form a feature vector. Finally a fuzzy logic reasoning system was developed to classify the pulses as sparks and arcs.

1

Introduction

Discharge pulses of EDM can commonly be classified into five kinds, which are open, spark, arc, short-circuit and off [1]. Open, short circuit and off stages can be distinguished easily for their distinctive characteristics, but it is very difficulty to distinguish spark and arc effectively because they are quite similar in many characters. Traditionally, the signals used to monitor EDM include the magnitude of voltage and current, the high frequency components of discharge, the delay time of discharge pulses and the radio frequency during discharge. From literature, though these signals had been widely used, the results are not good enough, especially in differentiating sparks and arcs [2-3]. Electrical impedance is an inherent characteristic of electrical system that will not change with the input signals and its change can reflect the changes of the electrical system. Though EDM is a complicated process, it can still be regarded as an electrical system approximately because the discharge is the result of the applied voltage. As the discharge is so complicated and we lack enough knowledge on the discharge mechanism, it is quite difficult to find an accurate mathematical model for sparks and arcs. Fuzzy reasoning system has advantages in processing fuzzy, uncertainty information and has a lot of successful application in engineering field. This paper proV. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 699-706, 2003.  Springer-Verlag Berlin Heidelberg 2003

700

Xiao Zhiyun and Ling Shih-Fu

posed a new monitoring method for EDM, through deriving a new monitoring signal and extracting features from it, a fuzzy reasoning system was used to classify the pulses into sparks and arcs correctly. In section 2, Wavelet transform and the partition of the discharge process were studied. In section 3, the definition and calculation of the electrical impedance signal was given. In section 4, the feature extraction from the impedance signal and voltage signal was introduced. In section 5, a Takagi-Sugeno fuzzy reasoning system was developed to differentiate sparks and arcs. In the last section, conclusions were remarked.

2

Signal Segmentation

The signal segmentation method we developed partitions the voltage and current of the whole discharge process into time slices representing separately discharge pulses. Theoretically, between every two pulses, there is an off stage that means no voltage and no current. During this stage, dielectric fluid was de-ionized and prepared for next discharge. We can regard this off stage as the edge of a discharge pulse, a kind of singularities, which commonly appears in both the one-dimensional signals and two-dimensional images. Until present, the Fourier transform was the main mathematical tool for analyzing singularities. The Fourier transform is global and provides a description of the overall regularity of signals but it is not well adapted for finding the location and the spatial distribution of singularities. The subject of Wavelet transform is a remarkable mathematical tool to analyze the singularities including the edges, by decomposing signals into elementary building blocks that are well localized both in space and frequency, the Wavelet transform can characterize the local regularity of signals, and further, to detect them effectively. A significant study related to this research topic had been done by Mallat, Hwang and Zhong [4][5]. The continuous Wavelet transform of a function is defined as W

f

(a, b) = a

−1 / 2

∫

f ( t )h (

t − a ) dt b

(1)

Here, f (t ) is the function to apply wavelet transform, a is dilation, b is translation and h ( t − a ) is wavelet basic function. b

Wavelet analysis presents a windowing technique with variable-sized regions in contrast to the other methods of signal analysis. Wavelet analysis employs longer time intervals where more precise low frequency information is required and shorter regions for high frequency information. This allows wavelet analysis to capture aspects like trends, breakdown points, and discontinuities in higher derivatives and selfsimilarity that are missed by the other signal analysis techniques. Singularities contain the high frequency information, using the coefficients of low scale of CWT, we can find the location of the singularities correctly. Calculation of the Wavelet transform is a key process in applications. The fast algorithm of computing W s f ( x ) for detecting edges and reconstructing the signal can

The Application of Fuzzy Reasoning System in Monitoring EDM

701

be found in [6] when ψ ( x ) is a dyadic wavelet defined in that article. However, in edge detection, the reconstruction of signals is not required. Therefore, the choice of the wavelet function will not be restricted in the conditions that were presented in [6]. Many wavelets other than dyadic ones can be utilized. In fact, almost all the general integral wavelets satisfy this particular application. Figure 1 is the voltage and current signal from the experiment. The experiment setting is: open voltage 80V, peak current 1.0A, on-time 2.4µs, off-time 0.8µs and sample frequency is 10M. Figure 2 is the segmentation result. The second part of this figure is the CWT coefficients of scale 5. It shows that we can extract pulse 1 to pulse 6 from the original signal accurately.

Fig. 1. Original voltage and current

Fig. 2. Segmentation result

3

Electrical Impedance of Discharge

Electrical impedance is a very important property of an electrical system. It is widely used to study stationary process and always appears as a frequency dependent variable. For an electrical system, impedance is always calculated by voltage over current in frequency domain. The ratio of the voltage and current amplitudes gives the absolute value of the impedance at the given frequency, and the phase shift between the current and the voltage gives the phase of it. The electrical impedance of an electrical process is defined as:

Z =V /I

(2)

702

Xiao Zhiyun and Ling Shih-Fu

The difficulty in calculating impedance in time domain is the sensed voltage and current are AC signals and they cannot be divided directly. In order to avoid this problem, we first convert the sensed voltage and current in real representations into their corresponding complex representations. The converted signals are called analytic signals. Then electrical impedance can be calculated by dividing the analytic voltage by the analytic current. The electrical impedance can express in the complex form, the real part and the imaginary part. The real part represents the resistance and the imaginary part represents the capacitance and inductance.

Z ( t ) = R ( t ) + jX (t )

(3)

It also can be expressed in magnitude and phase representation.

I (t ) = Z (t ) =

V ( t ) jφ ( t ) e Z (t )

(4)

R (t ) 2 + X (t ) 2

ϕ ( t ) = arctg

(5)

X (t ) R (t )

The commonly used transform is FFT, and the result is the electrical impedance at a given frequency. In order to get electrical impedance in time domain and study its varying trend with time, here, we use the Hilbert transform to transfer the measured voltage and current to their corresponding analytical signals and then calculate the electrical impedance. Hilbert transform of signal x(t ) are defined as: x (t ) = H

−1

[ x ( s )] =

π

x ( t ) = H − 1 [ x ( s )] =

π

1

1

∫

∞ x(s) −∞ t − s

ds

(6)

∫

∞ x(s) −∞ t − s

ds

(7)

This shows that the Hilbert transform are defined using the kernel

Φ (t , s ) = 1 /[π ( s − t )] and the conjugate kernel ψ (t , s ) = 1 /[π (t − s )] . That is, the kernels differ only by sign. Here the variable s is a time variable. As a result, the Hilbert transform of a function of time is another function of time of different shape. Accordingly, the corresponding analytical signal of v(t) and i(t) is respectively V ( t ) = v ( t ) + j vˆ ( t ) = a ( t ) exp( j ω t + φ 1 )

(8)

I ( t ) = i ( t ) + j iˆ ( t ) = b ( t ) exp( j ω t + φ 2 )

(9)

In the above two equations, v ( t ) and i (t ) is the real signal, vˆ ( t ) and

iˆ (t ) represent the Hilbert transform of v (t ) and i (t ) . After being transformed into complex form, the following calculation becomes possible for evaluating the electrical impedance.

The Application of Fuzzy Reasoning System in Monitoring EDM

Z =

V (t ) e I (t )

j (φ 1 − φ 2 )

703

(10)

Figure 5 is the calculated electrical impedance of each pulse using Hilbert transform. In the next step, we will extract some features from these signals, which will be used as the inputs of the fuzzy classifier.

Fig. 5. Magnitude of electrical impedance

4

Features Extraction

In this step, we will extract some features from the electrical impedance and voltage signals and determine the determinant features for classification. We define a feature to be any property of the impedance or voltage signal within a pulse that is useful for describing normal or abnormal behavior of this pulse. Based on the electrical discharge machining and signal processing knowledge, we have found the following features to be useful in determining a pulse is spark or arc: Voltage Average: As the breakdown voltage is lower and the delay time is shorter in arc than in spark. The voltage average of arc should be smaller than that of spark. Current Average: Because the dielectric was not fully de-ionized or the debris between the electrode and the workpiece was not flush away effectively. The average current of arc should be bigger than that of spark. Pulse Duration. As arcs are easy to start, the shorter the duration is, the more possible this pulse belongs to arc. Average Impedance. This is the most important feature in differentiating spark and arc because arc is believed to occur at the same spot of previous pulse, the dielectric between the two spot of electrode and workpiece was not fully de-ionized. Delay Time: Extracted from the voltage signal which can be obtained easily when partition the voltage signal using the coefficients of low scale. Commonly, spark is believed to have delay time before discharge starts while arc has no observable delay time.

704

Xiao Zhiyun and Ling Shih-Fu

Table 1 is the values of these features of pulse 1 to pulse 6. In the next section, we will study using a fuzzy reasoning system, taking all the features as the inputs, to differentiate spark and arc. Table 1. Features extracted from pulse 1 to pulse 6

Average V (V) Average I (A) Average Im (Ω) Duration (µs) Delay time (µs)

5

P1 61.448 0.181 409.06 11.8 7.2

P2 74.098 0.157 467.09 4.2 3.2

P3 24.733 0.350 61.31 3.4 0.6

P4 28.256 0.287 91.65 4.2 0.9

P5 22.027 0.290 58.92 4.2 0.8

P6 25.638 0.291 47.11 4.2 1.6

Fuzzy Reasoning System

Presently, Fuzzy system and neural network have been widely and successfully used in monitoring and controlling EDM [8-9]. The theory of fuzzy logic is aimed at the development of a set of concepts and techniques for dealing with sources of uncertainty, imprecision and incompleteness. The nature of fuzzy rules and the relationship between fuzzy sets of different shapes provides a powerful capability for incrementally modeling a system whose complexity makes traditional expert systems, mathematical models, and statistical approaches very difficult. The most challenging problem in differentiating sparks and arcs is that many characters of these two kinds pulses are similar and the knowledge about how to differentiate them is incomplete and vague due to the complexity of discharge phenomena. This uncertainty leads us to seek a solution using fuzzy logic reasoning method. Fuzzy reasoning is performed within the context of a fuzzy system model, which consist of control, solution variables, fuzzy sets, proposition statements, and the underlying control mechanisms that tie all these together into a cohesive reasoning environment. The fuzzy rules can be completely characterized by a set of control variables, X = { x1 , x 2 , L , x n } and solution variables, y1 , y 2 , L y k . In our application, we have five control variables corresponding to the extracted features and one solution variable representing the discharge state, i.e. a value of 1 indicate the normal discharge while a value of 0 indicate a abnormal discharge. Each control variable xi is associated with a set of fuzzy terms ∑ = {α 1 , α 2 , L a p i } , and the solution i

i

i

i

variable has its own fuzzy terms. Each fuzzy variable is associated with a set of fuzzy membership functions corresponding to the fuzzy terms of the variable. A fuzzy membership function of a control variable can be interpreted as a control surface that responds to a set of expected data points. The fuzzy membership functions associated with a fuzzy variable can be collectively defined by a set of critical parameters that uniquely describe the characteristics of the membership function, and the characteristic of an inference engine is largely affected by these critical parameters.

The Application of Fuzzy Reasoning System in Monitoring EDM

705

Takagi-Sugeno fuzzy reasoning system was first introduced in 1985 [10]. It is similar to the Mamdani method in many respects. In fact the first two parts of the fuzzy inference process, fuzzifying the inputs and applying the fuzzy operator, are exactly the same. The main difference between Mamdani-type of fuzzy inference and Sugeno-type is that the output membership functions are only linear or constant for Sugeno-type fuzzy inference. A typical fuzzy rule in a zero-order Sugeno fuzzy model has the form If x is A and y is B then z = k Where A and B are fuzzy sets in the antecedent, while k is a crisply defined constant in the consequent. When the output of each rule is a constant like this, the similarity with Mamdani's method is striking. The only distinctions are the fact that all output membership functions are singleton spikes, and the implication and aggregation methods are fixed and cannot be edited. The implication method is simply multiplication, and the aggregation operator just includes all of the singletons. In our application, as the fuzzy reasoning system has only one output, that is a pulse belongs to spark or arc, it is more convenient to use Sugeno-type reasoning system than conventional used Mamdani system. The fuzzy reasoning system we construct shown in figure 6. It has five inputs that corresponding to the five features we obtained, every input has three fuzzy terms: low, medium and high. There is only one output, the discharge state, which has two values, zero corresponding to arc and one corresponding to spark. Table 2 is the fuzzy reasoning results. We can see, pulse 1 and pulse 2 were classified as sparks, and pulse 3 to pulse 6 were classified as arcs. Average voltage Average current Average impedance Delay time

Fuzzy reasoning system

Discharge state (0: Arc, 1: Spark)

Pulse duration

Fig. 6. Fuzzy reasoning system Table 2. Fuzzy reasoning results for pulse 1 to pulse 6

Pulses 1 Reasoning result 1(spark)

6

2 1(spark)

3 0(arc)

4 0(arc)

5 0(arc)

6 0(arc)

Conclusions

This paper proposed a new monitoring method for EDM process. Through analyzing the electrical impedance signal and extracting features from this signal and using fuzzy reasoning system as the classifier, this method can differentiate sparks and arcs effectively. The proposed method has the following advantages: (1) this method can detect off stage easily when segmenting the voltage and current signal. (2) this

706

Xiao Zhiyun and Ling Shih-Fu

method can process pulse individually, so the monitoring result will be precise and can be quantitated, not just the varying trend as taking frequency as monitoring signal. (3) The proposed method can effectively differentiate sparks and arcs. (4) The monitoring signal is an inherent character of the EDM system, so the monitoring result is more credible. (5) This method system is easier to be implemented. The cost of the monitoring system will be lower because we just need to collect the voltage and current between the discharge gap.

References [1]

“Summary specifications of pulse analyzers for spark-erosion machines”, CIRP scientific technical committee E, 1979. [2] M. Weck, J. M. Dehmer. Analysis and adaptive control of EDM sinking process using the ignition delay time and fall time as parameter. Annals of the CIRP Vol. 41/1/1992 [3] M. S. Ahmed. Radio frequency based adaptive control for electrical discharge texturing. EDM digest, Sept./Oct. 1987, pp. 8-10 [4] Mallat, S. and Hwang, W. L. (1992) Singularity detection and processing with wavelet. IEEE Trans. On information theory, 38: 617~643 [5] Mallat, S. and Zhong, S. (1992) Characterization of signals from multiscale edges. IEEE Trans. On pattern analysis and machine intelligence, 14(7): 710 ~ 732 [6] Mallat, S (1989) Multiresolution approximations and wavelet orthonormal base of L2(R). Trans. Amer Math. Soc, 315: 69 ~ 87 [7] Stefan L. Hahn, Hilbert Transform in signal processing, Artech House, 1996 [8] Y. S. Tarng, C. M. Tseng and L. K. Chung. A fuzzy pulse discriminating system for electrical discharge machining. International Journal of Tools and Manufacturing. Vol. 37, No. 4, pp. 511-522, 1997 [9] J. Y. Kao, Y. S. Tarng. A neural network approach for the on-line monitoring of electrical discharge machining process. Journal of Materials Processing Technology 69 (1997) 112-119 [10] Sugeno, M., Industrial applications of fuzzy control, Elsevier Science Pub. Co., 1985.

Knowledge Representation for Structure and Function of Electronic Circuits Takushi Tanaka Department of Computer Science and Engineering, Fukuoka Institute of Technology 3-30-1 Wajiro-Higashi Higashi-ku, Fukuoka 811-0295, Japan [email protected]

Abstract. Electronic circuits are designed as a hierarchical structure of functional blocks. Each functional block is decomposed into sub-functional blocks until the individual parts are reached. In order to formalize knowledge of these circuit structures and functions, we have developed a type of logic grammar called Extended-DCSG. The logic grammar not only defines the syntactic structures of circuits but also defines relationships between structures and their functions. The logic grammar, when converted into a logic program, parses circuits and derives their functions.

1

Introduction

Electronic circuits are designed as a hierarchical structure of functional blocks. Each functional block consists of sub-functional blocks. Each sub-functional block is also decomposed into sub-sub-functional blocks until the individual parts are reached. In other words, each functional block, even a resistor, has a special goal for its containing functional block. Each circuit as a final product itself can also be viewed as a functional block designed for a special goal for users. As these hierarchical structures were analogous to syntactic structures of language, we developed a grammatical method for knowledge representation of electronic circuits[1]. In the study, each circuit was viewed as a sentence and its elements as words. Structures of functional blocks were defined by logic grammar called DCSG (Definite Clause Set Grammar)[2]. Here, the DCSG is a DCG[3]like logic grammar developed for analyzing word-order free languages. A set of the grammar rules, when converted into Prolog clauses, forms a logic program which executes top-down parsing. Thus knowledge of circuit structures were successfully represented by DCSG, but knowledge of circuit functions could not be represented. In this study, we extend DCSG by introducing semantic terms into grammar rules. The semantic terms define relationships between syntactic structures and their meanings. By assuming functions as meaning of structures, we can represent relationships between functions and structures of electronic circuits.

V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 707–714, 2003. c Springer-Verlag Berlin Heidelberg 2003


708

2 2.1

Takushi Tanaka

DCSG Word-Order Free Language

A word-order free language L(G’) is defined by modifying the definition of a formal grammar[2]. We define a context-free word-order free grammar G’ to be a quadruple < VN , VT , P, S > where: VN is a finite set of non-terminal symbols, VT is a finite set of terminal symbols, P is a finite set of grammar rules of the form: A −→ B1 , B2 , ..., Bn . (n ≥ 1) A ∈ VN , Bi ∈ VN ∪ VT (i = 1, ..., n) and S(∈ VN ) is the starting symbol. The above grammar rule means rewriting a symbol A not with the string of symbols “B1 , B2 , ..., Bn ”, but with the set of symbols {B1 , B2 , ..., Bn }. A sentence in the language L(G’) is a set of terminal symbols which is derived from S by successive application of grammar rules. Here the sentence is a multi-set which admits multiple occurrences of elements taken from VT . Each non-terminal symbol used to derive a sentence can be viewed as a name given to a subset of the multi-set. 2.2

DCSG Conversion

The general form of the conversion procedure from a grammar rule A −→ B1 , B2 , ..., Bn .

(1)

to a Prolog clause is: subset(A, S0 , Sn ) : − subset(B1 , S0 , S1 ), subset(B2 , S1 , S2 ), ... subset(Bn , Sn−1 , Sn ).

(1)’

Here, all symbols in the grammar rule are assumed to be a non-terminal symbol. If “[Bi ]”(1 ≤ i ≤ n) is found in the right hand side of grammar rules, where “Bi ” is assumed to be a terminal symbol, then “member(Bi , Si−1 , Si )” is used instead of “subset(Bi , Si−1 , Si )” in the conversion. The arguments S0 , S1 , ..., Sn in (1)’ are multi-sets of VT , represented as lists of elements. The predicate “subset” is used to refer to a subset of an object set which is given as the second argument, while the first argument is a name of its subset. The third argument is a complementary set which is the remainder of the second argument less the first; e.g. “subset(A, S0, Sn )” states that “A” is a subset of S0 and that Sn is the remainder. The predicate “member” is defined by the following Prolog clauses: member(M, [M |X], X). member(M, [A|X], [A|Y ]) : − member(M, X, Y ).

(2)

The predicate “member” has three arguments. The first is an element of a set. The second is the whole set. The third is the complementary set of the first.

Knowledge Representation for Structure and Function of Electronic Circuits

3

709

Extended-DCSG

In order to define relationships between syntactic structures and their meaning, we introduce semantic terms into grammar rules. The semantic terms can be placed both sides of grammar rules. Different from DCSG, we do not distinguish terminal symbols from non-terminal symbols in grammar rules so that we can treat meaning of words same as meaning of structures. 3.1

Semantic Term in Left-Hand Side

The semantic terms are placed in curly brackets as: A, {F1 , F2 , ..., Fm } −→ B1 , B2 , ..., Bn .

(3)

This grammar rule can be read that the symbol A with meaning {F1 , F2 , ..., Fm } consists of the syntactic structure B1 , B2 , ..., Bn . This rule is converted into a Prolog clause in Extended-DCSG as: ss(A, S0 , Sn , E0 , [F1 , F2 , ..., Fm |En ]) : − ss(B1 , S0 , S1 , E0 , E1 ), ss(B2 , S1 , S2 , E1 , E2 ), ... , ss(Bn , Sn−1 , Sn , En−1 , En ). (3)’ As the conversion is different from DCSG, we use predicate “ss” instead of “subset”. When the rule is used in parsing, the goal ss(A, S0 , Sn , E0 , E) is executed, where the variable S0 is substituted by an object set and the variable E0 is substituted by an empty set. The subsets “B1 , B2 , ..., Bn ” are successively identified in the object set S0 . After all of these subsets are identified, the remainder of these subsets (complementary set) is substituted into Sn . While, semantic information of B1 is added with E0 and substituted into E1 , semantic information of B2 is added with E1 and substituted into E2 ,..., and semantic information of Bn is added with En−1 and substituted into En . Finally, semantic information {F1 , F2 , ..., Fm } which is the meaning associated with symbol A is added, and whole semantic informations are substituted into E. 3.2

Semantic Term in Right-Hand Side

Semantic terms in right-hand side define semantic conditions for the grammar rule. For example, the following rule (4) is converted into the Prolog clause (4)’. A −→ B1 , {C1 , C2 }, B2 .

(4)

ss(A, S0 , S2 , E0 , E2 ) : − ss(B1 , S0 , S1 , E0 , E1 ), member(C1 , E1 , ), member(C2 , E1 , ), ss(B2 , S1 , S2 , E1 , E2 ).

(4)’

When the clause (4)’ is used in parsing, conditions C1 , C2 are tested whether the semantic information E1 fills these conditions after identifying the symbol B1 . If it succeeds, the parsing process goes on to identify the symbol B2 .

710

3.3

Takushi Tanaka

Terminal Symbol

Both terminal and non-terminal symbols are converted with the predicate “ss”. Only difference is that terminal symbols are defined by rules which do not have right-hand side to rewrite. The terminal symbol A with meaning {F1 , F2 , ..., Fm } is written as (5). A, {F1 , F2 , ..., Fm }.

(5)

This rule is converted into the following clause (5)’ in Extended DCSG. ss(A, S0 , S1 , E0 , [F1 , F2 , ..., Fm |E0 ]) : −member(A, S0 , S1 ).

(5)’

That is, when the rule (5) is used in parsing, the terminal symbol A is searched in the object set S0 . If it is found, the complementary set is returned from S1 , and the semantic term {F1 , F2 , ..., Fm } associated with A is added with current semantic information E0 to make the fifth argument of “ss”. Thus, we can treat meaning of wards with the same manner of syntactic structures.

4 4.1

Knowledge Representation Representation of Circuits

In the previous study[1], the circuit in Figure 1 is represented as the free wordorder sentence (6). The compound term “resistor(r1, 2, 10)” is a word of the sentence. It denotes a resistor named r1 connecting node 2 and node 10. The word “npnT r(q1, 3, 5, 6)” denotes an NPN-transistor named q1 with the base connected to node 3, the emitter to node 5, and the collector to node 6 respectively. [ resistor(r1, 2, 10), resistor(r2, 9, 1), npnT r(q1, 3, 5, 6), npnT r(q2, 4, 5, 7), npnT r(q3, 10, 1, 5), npnT r(q4, 10, 1, 10), npnT r(q5, 10, 1, 8), npnT r(q6, 8, 9, 2), ... , terminal(t4, 9), terminal(t5, 1) ] 4.2

(6)

Grammar Rules

The rule (7) defines an NPN-transistor in active state as a terminal symbol. Its semantic term defines relationships of voltages and currents in the state. Each compound term such as “gt(voltage(C, E), vst)” is a logical sentence for circuit functions. Here, “gt(voltage(C, E), vst)” states “voltage between C and D is greater than collector saturation voltage”. Similar rules are defined for other states of transistor. npnT R(Q, B, E, C), { state(Q, active), gt(voltage(C, E), vst), equ(voltage(B, E), vbe), gt(current(B, Q), 0), gt(current(C, Q), 0), cause(voltage(B, E), current(B, Q)), cause(current(B, Q), current(C, Q)) }.

(7)

Knowledge Representation for Structure and Function of Electronic Circuits

711

t3 2

q7

q8

6

7

7

t1 q1 3

6 5

7

2

q2

5

q9

2

6

8

2

q6

2

10

8

9 t4

r1

4

t2

9

4 5

q3

1

10

q4

10 q5

8

1

1

r2 1 t5 1

Fig. 1. Operational Amplifier cd42 The rule (8) which was originally introduced to refer a resistor as non-polar element[1] defines causalities of voltage and current on the resistor. res(R, A, B), { cause(voltage(A, B), current(A, R)), cause(current(A, R), voltage(A, B)) } −→ ( resistor(R, A, B); resistor(R, B, A) ).

(8)

The rule (9) defines a simple voltage regulator in Figure 2. The semantic term of left-hand side is the function of this structure that the circuit named vreg(D, R) controls the voltage between Out and Com. The right-hand side consists of the syntactic structures of diode D and resistor R, and the semantic term which specifies an electrical condition of diode D. vbeReg(vreg(D, R), V p, Com, Out), { control(vreg(D, R), voltage(Out, Com)) } −→ dtr(D, Out, Com), { state(D, conductive) }, res(R, V p, Out). vreg(D,R) R Vp

Out D

Com

Fig. 2. Voltage regulator

(9)

712

Takushi Tanaka sink(VR,Q) VR

In

Q B

Com

Fig. 3. Current Sink

The rule (10) is defined for the current sink in Figure 3. The semantic term in left-hand side is the function of this structure. The right-hand side is the condition of this circuit. The disjunction “;” specifies either a structural condition or a semantic condition. It becomes useful to analyze context dependent circuits[1]. The next line specifies the connections of transistor Q, and requires the transistor Q must be in active state. cSink(sink(V R, Q), In, Com), { control(sink(V R, Q), current(In, Q)) } −→ ( vbeReg(V R,, Com, B); { control(V R, voltage(B, Com))} ), npnT r(Q, B, Com, In), { state(Q, active) }. (10) The rule (11) defines the active load (current mirror) in Figure 4. Functional blocks such as common-emitter, emitter-follower, and differential amplifier are also defined as grammar rules. activeLoad(al(D, Q), Ref, V p, Ld), { control(al(D, Q), current(Q, Ld)), cause(current(al(D, Q), Ref ), current(Q, Ld)), equ(current(Q, Ld), current(al(D, Q)Ref )) } −→ dtr(D, V p, Ref ), { state(D, conductive) }, pnpT r(Q, Ref, V p, Ld), { state(Q, active) }.

al(D,Q)

Vp

Vp D Q

Ref

Ld

Fig. 4. Active Load

(11)

Knowledge Representation for Structure and Function of Electronic Circuits

5

713

Deriving Circuit Functions

The grammar rules in the previous section are converted into Prolog clauses. Using these clauses, the goal (12) parses the circuit in Figure 1. The identified circuit is substituted into the variable X. The first argument of the composite term is a name given to the circuit, and represents the syntactic structure as shown in Figure 5. The circuit functions which consist of more than 100 logical sentences are substituted into the variable Y as meanings of the circuit structure. ? − cd42(Circuit), ss(X, Circuit, [ ], [ ], Y ).

(12)

X = opAmp(opAmp(sdAmp(ecup(q1, q2), al(pdtr(q8), q7), sink(vreg(dtr(q4), r1), q3)), pnpCE(q9, sink(vreg(dtr(q4), r1), q5)), npnEF (q6, r2)), 3,4,9,2,1)

(13)

Y = [ input(opAmp(...), voltage(3, 4)), output(opAmp(...), voltage(9, 1)), cause(voltage(3, 4), voltage(9, 1)), enable(sdAmp(...), amplif y(opAmp(...), dif f erential inputs)), enable(pnpCE(...), high(voltage gain(opAmp(...)))), enable(npnEF (...), low(output impedance(opAmp(...)))), input(npnEF (...), voltage(8, 1)), output(npnEF (...), voltage(9, 1)), cause(voltage(8, 1), voltage(9, 1)), high(input impedance(npnEF (...))), low(output impedance(npnEF (...))), equ(voltage gain(npnEF (...)), 1), state(q6, active), ... ] (14)

6

Conclusion

We have introduced semantic terms into grammar rules to extend DCSG. The semantic terms define relationships between syntactic structures and their meanings. We also changed the notation of terminal symbol. Different from DCSG, terminal symbols are not distinguished from non-terminal symbols, but defined by grammar rules lacking right-hand side. This enables us to treat the meaning of words same as meaning of structures. That is, Extended-DCSG defines not only the surface structure of language, but also their meanings hidden from the surface.

714

Takushi Tanaka opAmp

sdAmp

ecup

al

pnpCE

sink

sink q9

dtr q1

q2

npnEF

q6

r2

vreg q7

q3

q5

dtr q8

r1 q4

Fig. 5. Parse tree of cd42

We assumed circuit functions as meaning of circuit structures, and extended circuit grammars to define relationships between structure and function. The grammar rules were translated into Prolog clauses, and derived structures and functions from the given circuits. Now we have had a method of encoding knowledge for electronic circuits. If we encode more knowledge, more circuits can be parsed. Currently, our system outputs circuit structures and functions in logical sentence. In order to improve readability for user, we are developing natural language interface. Circuit simulators such as SPICE [5] derive voltages and currents from given circuits. Such simulator replaces measurements of experimental circuits with computer. While our system derives structures and functions from given circuits as a simulator of understanding. It helps user to understand how circuits work in design, analysis, and trouble shooting. These two different systems will work complementary.

References [1] Tanaka, T.: Parsing Circuit Topology in A Logic Grammar, IEEE-Trans. Knowledge and Data Eng., Vol.5, No.2, pp. 225–239, 1993. 707, 710, 711, 712 [2] Tanaka, T.: Definite Clause Set Grammars: A Formalism for Problem Solving, J. Logic Programming, Vol.10, pp. 1–17, 1991. 707, 708 [3] Pereira, F. C. N. and Warren, D. H. D.: Definite Clause Grammars for Language Analysis, Artificial Intell., Vol.13, pp. 231–278, 1980. 707 [4] Tanaka, T. and Bartenstein, O.: DCSG-Converters in Yacc/Lex and Prolog, Proc. 12th International Conference on Applications of Prolog, pp. 44–49, 1999. [5] Tuinenga, P. W.: SPICE - A Guide to circuit Simulation & Analysis using PSpice, Prentice Hall (1988) 714

A Web-Based System for Transformer Design J.G. Breslin and W.G. Hurley Power Electronics Research Centre, National University of Ireland Galway, Ireland {john.breslin,ger.hurley}@nuigalway.ie http://perc.nuigalway.ie/

Abstract. Despite the recent use of computer software to aid in the design of power supply components such as transformers and inductors, there has been little work done on investigating the usefulness of a webbased environment for the design of these magnetic components. Such an environment would offer many advantages, including the potential to share and view previous designs easily along with platform/OS independence. This paper presents a web-based transformer design system whereby users can create new optimised transformer designs and collaborate or comment on previous designs through a shared information space.

1

Introduction

Despite the recent use of computer software to aid in the design of magnetic components [3], to date there has been little work done on investigating the usefulness of a web-based environment for magnetic component design. Such an environment would offer many advantages, including the potential to share and view previous designs easily along with platform and operating system independence. This paper proposes a web-based prototype for magnetic component design. The system is based on an existing transformer design methodology [1], and is implemented using a web programming language, PHP. Magnetic component material information and designs created by students and instructors are stored in a MySQL database on the system server. It will therefore be possible for users of this system to collaborate and comment on previous designs through a shared information space. We will begin with a summary of existing methods for transformer design, followed by details of the system design, and finally an overview of the web-based implementation.

2

Related Research and Other Systems

2.1

Transformer Design Methodologies

The basic area product methodology [4] often results in designs that are not optimal in terms of either losses or size since the method is orientated towards low frequency V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 715-721, 2003.  Springer-Verlag Berlin Heidelberg 2003

716

J.G. Breslin and W.G. Hurley

transformers. A revised arbitrary waveform methodology was proposed [1] that allows designs at both low and high frequencies, and is suitable for integration with high frequency winding resistance minimisation techniques [2]. The selection of the core is optimised in this methodology to minimise both the core and winding losses. The design process can take different paths depending on whether a flux density constraint is violated or not. If the flux density exceeds a set limit (the saturation value for a particular core material), it is reset to be equal to this limit and one path is used to calculate the core size; otherwise the initial flux density is used and a different path is taken to find the core size. 2.2

Previous Windows-Based Packages

Many magnetic design companies have used computer spreadsheets to satisfy their own design needs and requirements. They thus tend to be solely linked to a company and its products, and remain unpublished as their content is only of interest to their direct users and competitors. Some of the limitations of these spreadsheets include: difficulties in incorporating previous designer expertise due to limited decision or case statements; non-conformity with professional database formats used by manufacturers; problems with implementing most optimisation routines due to slow iterative capabilities and a reduced set of mathematical functions; basic spreadsheet input mechanisms that lack the features possible with a customised GUI. A computer aided design package has previously been developed [3] for the Windows environment based on the arbitrary waveform methodology [1]. This package allows the design of transformers by both novice and expert users for multiple application types through “try-it-and-see” design iterations. As well as incorporated design knowledge and high frequency winding optimisation, the system allows access to customisable or pre-stored transformer geometries which were usually only available by consulting catalogues. Other systems such as [6] provide more detailed information on winding layouts and SPICE models, but lack the high frequency proximity and skin effects details of [3]. Another Windows-based package has recently been released [7] which allows the comparison of designs based on different parameters. Some companies have also advertised web-based selection of magnetic components [8]; however these tend to be online spreadsheets and therefore suffer from the problems previously mentioned.

3

System Design

All of the information and associated programs of the system reside on a web server, and any user can access the system using a web browser through the appropriate URL. The inputs to the system are in the form of specifications such as desired voltage, current, frequency, etc. The output from the system is an optimised design for the specifications given. Fig. 1 shows the steps taken in creating and managing a design. The system is comprised of the following elements: a relational database management system (RDBMS); a web-based graphical user interface (GUI); optimisation

A Web-Based System for Transformer Design

717

techniques; a repository of knowledge; and a shared information space. We will now describe these in some more detail. Start

Enter Specifications, Choose Materials

Opt. Flux Density > Sat. Value?

Y

N Use Opt. Flux Density, Calc. Area Product

Use Sat. Value, Calc. Alternative Area Product

N Satisified?

Y N Satisified?

Select Desired Core Geometry

Y

Calculate Turns Information N Satisified?

Y

Optimise Winding Thickness?

Y

N Select Desired Winding Geometries

Calculate and Use Opt. Winding Geometries

N Satisified?

Y

Predict Losses, Efficiency, Calc. Leakage Inductance N Satisified?

Y Stop

Fig. 1. Flow chart of design steps

3.1

Relational Database Management System

In transformer design, huge amounts of core and winding data have to be managed for effective exchange of information between various design stages. The magnetic designer should not get lost in the data management process as their concentration on the real design problem may be affected. A RDBMS can be used to save designers from having to search through books with manufacturer's data or from manipulating

718

J.G. Breslin and W.G. Hurley

data themselves with lower level external programs. All transformer core and winding data is accessible using the sophisticated database storage and retrieval capabilities of a relational database engine incorporated into the design application. The database contains the following tables of data: designs (where each saved design is identified by a unique design_id); designers (users of the system); areas (consisting of parent areas which contain child areas for related sets of designs); comments (each relating to a particular design_id); cores and windings (either read-only items added by an administrator or modifiable items created by a particular designer); and tables for core and winding materials, shapes, manufacturers and types. 3.2

Web-Based Graphical User Interface

In our system, we require over 250 HTML objects and form controls for interaction with the user; these include text boxes for both inputs and calculated outputs, labels describing each text box, radio buttons and checkboxes for selection of discrete or Boolean variables, option group menus, graphs of waveforms, etc. Proper categorisation and presentation of data in stages is our solution to the problem of organising this data in a meaningful way, whereby images identify links to the distinct steps in the design process, and only information related to a particular step is shown at any given time. Some of the main GUI features incorporated in the system are: numbered and boxed areas for entry of related data in “sub-steps”; input and output boxes colourcoded according to whether data entry is complete, required, or just not permitted; scrollbars for viewing large tables of data in small areas; and pop-up message windows for recommendations and errors. 3.3

Optimisation Techniques

The performance level of an engineering design is a very important criterion in evaluating the design. Optimisation techniques based on mathematical routines provide the magnetic designer with robust analytical tools, which help them in their quest for a better design. The merits of a design are judged on the basis of a criterion called the measure of merit function (or the figure of merit if only a single measure exists). Methods for optimising AC resistance for foil windings [2] and total transformer loss [1] are implemented in the web-based system; these variables are our measures of merit. 3.4

Repository of Knowledge

A “repository of knowledge” is incorporated into our system, to allow a program design problem to be supplemented by rules of thumb and other designer experience. In the early design stages, the designer generates the functional requirements of the transformer to be designed, and the expertise of previous users can play a very important role at this stage. For example, on entry of an incompatible combination of transformer specifications, the designer will be notified by a message informing them that a design error is imminent. The system will also suggest recommended “expert” values for certain

A Web-Based System for Transformer Design

719

variables. Although such a system is useful for novices, it can also be used by experts who may already know of certain recommended values and who want to save time setting them up in the first place. 3.5

Shared Information Space

The system allows collaboration between users working on a design through a shared information space, with features similar to those of a discussion forum. Designs are filed in folders, where each folder may be accessed by a restricted set of users. To accommodate this, user and group permissions are managed through an administration panel. Access to design folders is controlled by specifying either the users or the groups that have permission to view and add designs to that folder. Users can comment on each design, and can also send private messages to other users.

4

Implementation and Testing

A popular combination for creating data-driven web sites is the PHP language with a MySQL database, and this was chosen for the implementation of the system. PHP is a server-side scripting language that allows code to be embedded within an HTML page. The web server processes the PHP script and returns regular HTML back to the browser. MySQL is a RDBMS that can be run as a small, compact database server and is ideal for small to medium-sized applications. MySQL also supports standard ANSI SQL statements. The PHP / MySQL combination is cross-platform; this allowed the development of the system in Windows while the server runs on a stable BSD Unix platform. A typical PHP / MySQL interaction in the system is as follows. After initial calculations based on the design specifications to find the optimum core size (as mentioned in section 2.1 and detailed in [1]), a suitable core geometry is obtained from the cores database table using the statement: $suitable_core_array = mysql_query("SELECT * FROM cores WHERE (core_ap >= $optimum_ap AND corematerial_id = $chosen_corematerial_id AND coretype_id = $chosen_coretype_id) ORDER BY core_ap LIMIT 1"); where the names of calculated and user-entered values are prefixed by the dollar symbol ($), and fields in the database table have no prefix symbol. Fig. 2 shows the user interface, with each of the design steps clearly marked at the top and the current step highlighted (i.e. “Specifications”). An area at the bottom of the screen is available for designer comments. The underlying methodologies have previously been tested by both the authors [1] and external institutions [5]. Design examples carried out using the system produce identical results to those calculated manually. The system is being tested as a computer-aided instruction tool for an undergraduate engineering class.

J.G. Breslin and W.G. Hurley

Designs > Push-Pull Group > Dudley Device ES2d 1.1 Specifications

Comments 12

New Comment

Output Voltage (V): Output Current (A): Input Voltage Lower (V): Input Voltage Upper (V): Frequency (Hz): Temperature Rise (°C): Ambient Temp (°C): Efficiency (%): Calculated Area Product (cm^4): Calculated Flux Density (T):

Core Material: Winding Material:

36 72

John Smith

Circuit Diagram 11

Custom Addition 10

Leak Inductance 9

Welcome, John Smith! You have 2 PMs.

Modifications: 10 | Last: 22:01, 11-FEB-03

1.2 Core and Winding Info

24 12.5

Optim Thickness 8

Total Losses 7

Magnetic Component Designer 2.0.0b Core Losses 6

Wire Data 4

Turns Info 3

Core Data 2

Specifications 1

Login and Files 0

MaCoDe

Copper Losses 5

720

N27 CU

Transformer Type:

1.3 Expert Options Primary Power Factor: 2ndary Power Factor: Duty Cycle: Optimum Constant: Output Power (W): Volt Ampere Rating (VA): Waveform Factor: Current Density Factor: Turns Ratio: Window Utilisation Factor:

1:1 0.4

Perhaps it would be possible to integrate this particular design with the existing spec for the Allen application.

Emb PSU Team

If we limit the temperature rise to 25 degrees, we can meet that spec. E-Mail | Priv Msg

Matt Jones

13:23, 10-FEB-03

Edit | Quote

Yes.

Fig. 2. Screenshot of web-based system

5

Conclusions and Future Work

With the current trend towards miniaturisation in power converters, the magnetics designer should now expect accurate computer aided techniques that will allow the design of any magnetic component while incorporating existing techniques in the area of web-based collaboration. This paper presented a web-based transformer design system based on a previous methodology. This system is an improvement on previous automated systems because: previous designer expertise and optimisation routines are incorporated into the design method; database integration avoids the need for consultation of catalogues; and a user-friendly interface, with advanced input mechanisms, allows for collaboration among users where designs can be shared and analysed. This system can be further developed for more transformer applications, and with a revised methodology the system could also be updated to incorporate inductors and integrated planar magnetics.

A Web-Based System for Transformer Design

721

References [1] [2] [3] [4] [5] [6] [7] [8]

Hurley, W.G., Wlö fle, W., Breslin, J.G. : Optimized Transformer Design: Inclusive of High Frequency Effects. In: IEEE Trans. on Power Electronics, Vol. 13, No. 4 (1998) 651-659 Hurley, W.G., Gath, E., Breslin, J.G.: Optimizing the AC Resistance of Multilayer Transformer Windings with Arbitrary Current Waveforms. In: IEEE Trans. on Power Electronics, Vol. 15, No. 2 (2000) 369-376 Hurley, W.G., Breslin, J.G.: Computer Aided High Frequency Transformer Design Using an Optimized Methodology. In: IEEE Proc. of COMPEL 2000 (2000) 277-280 McLyman, W.T.: Transformer and Inductor Design Handbook. Marcel Dekker, New York (1978) Lavers, J.D., Bolborici, V.: Loss Comparison in the Design of High Frequency Inductors and Transformers. IEEE Transactions on Magnetics, Vol. 35, No. 5 (1999) 3541-3543 Intusoft Magnetics Designer: http://www.intusoft.com/mag.htm Ansoft PExprt: http://www.ansoft.com/products/em/pexprt/ Cooper Electronic Technologies Press Release for Coiltronics Versa Pac: WebBased Selection for Transformers and Inductors. http://www.electronicstalk.com/news/hnt/hnt105.html and http://www.cooperet.com/products_magnetics.asp

A Fuzzy Control System for a Small Gasoline Engine S.H. Lee, R.J. Howlett, and S.D. Walters Intelligent Systems & Signal Processing Laboratories Engineering Research Centre, University of Brighton Moulsecoomb, Brighton, BN2 4GJ, UK {S.H.Lee,R.J.Howlett,S.D.Walters}@Brighton.ac.uk

Abstract.Small spark-ignition gasoline-fuelled internal-combustion engines can be found all over the world performing in a variety of roles including power generation, agricultural applications and motive power for small boats. To attain low cost, these engines are typically aircooled, use simple carburettors to regulate the fuel supply and employ magneto ignition systems. Electronic control, of the sort found in automotive engines, has seldom proved cost-effective for use with small engines. The future trend towards engines that have low levels of polluting exhaust emissions will make electronic control necessary, even for small engines where previously this has not been economic. This paper describes a fuzzy control system applied to a small engine to achieve regulation of the fuel injection system. The system determines the amount of fuel required from a fuzzy algorithm that uses the engine speed and manifold air pressure as input values. A major advantage of the fuzzy control technique is that it is relatively simple to set up and optimise compared to the labour-intensive process necessary when the conventional "mapped" engine control method is used. Experimental results are presented to show that a considerable improvement in fuel regulation was achieved compared to the original carburettor-based engine configuration, with associated improvements in emissions. It is also demonstrated that the system produces improved output power and torque curves compared to those achieved when the original mechanical fuel regulation system was used. 1

Introduction

Electronic control of the air-fuel ratio (AFR) and ignition timing of a spark ignition (SI) engine is an effective way to achieve improved combustion efficiency and performance, as well as the reduction of exhaust emissions. The AFR essentially sets the operating point of the engine, and in conjunction with the ignition timing angle, determines the output power and the resulting levels of emissions. In an engine with electronic control, the amount of fuel that is supplied to the engine is controlled by an engine control unit (ECU). This is a micro-processor based V. Palade, R.J. Howlett, and L.C. Jain (Eds.): KES 2003, LNAI 2773, pp. 722-732, 2003.  Springer-Verlag Berlin Heidelberg 2003

A Fuzzy Control System for a Small Gasoline Engine

723

system that controls the frequency and width of the control pulse supplied to the fuel injector. The AFR is important in the combustion and calibration processes. If there is too much fuel, not all of it is burnt, causing high fuel consumption and increased emissions of HC and CO. Too little fuel can result in overheating and engine damage such as burnt exhaust valves. Conventional ECUs use three-dimensional mappings (3-D maps), in the form of look-up tables, to represent the non-linear behaviour of the engine in real-time [1]. A modern vehicle ECU can contain up to 50 or more of these maps to realise complex functions. In addition the engine will be equipped with a wide range of sensors to gather input data for the control system. A major disadvantage of the look-up table representation is the time taken to determine the values it should contain for optimal engine operation; a process known as calibration of the ECU. These 3-D maps are typically manually calibrated, or tuned, by a skilled technician using an engine dynamometer to obtain desired levels of power, emissions and efficiency. The calibration process is an iterative one that requires many cycles of engine measurements and is very time consuming. Techniques that reduce the time and effort required for the calibration process are of considerable interest to engine manufacturers. This is especially the case where the engine is a small capacity non-automotive engine. These engines are particularly price sensitive and any additional cost, including the cost of extended calibration procedures, is likely to make the engine un-economic to manufacture. For similar economic reasons, any control strategy intended for application to a small engine has to be achievable using only a small number of low-cost sensors. This paper makes a useful contribution to research in this area by proposing a technique for the rapid calibration of a small-engine ECU, requiring only sensors for speed and manifold air pressure.

2

Fuzzy Control

Fuzzy logic is a ‘soft computing' technique, which mimics the ability of the human mind to learn and make rational decisions in an uncertain and imprecise environment [2]. Fuzzy control has the potential to decrease the time and effort required in the calibration of engine control systems by easily and conveniently replacing the 3-D maps used in conventional ECUs. Fuzzy logic provides a practicable way to understand and manually influence the mapping behaviour. In general, a fuzzy system contains three main components, the fuzzification, the rule base and the defuzzification. The fuzzification is used to transform the so-called crisp values of the input variables into fuzzy membership values. Afterwards, these membership values are processed within the rule base using conditional ‘if-then' statements. The outputs of the rules are summed and def