Learning with recurrent neural networks

Lecture Notes in Control and Information Sciences Editor: M. Thoma

254

Springer London Berlin Heidelberg New York Barcelona Hong Kong Milan Paris Singapore

Tokyo

BarbaraHammer

Learning with Recurrent Neural Networks With 24 Figures

~ Springer

Series Advisory Board A. B e n s o u s s a n • M.J. G r i m b l e • P. K o k o t o v i c • A.B. K u r z h a n s k i • H. K w a k e r n a a k • J.L. M a s s e y • M. M o r a r i

Author Barbara Hammer, Dr rer. naL, DiplMath

Department of Mathematics and Computer Science,Universityof Osnabriick, D-49069 O s n a b r t i c k , G e r m a n y

ISBN 1-85233-343-X Springer-Verlag London Berlin Heidelberg British Library Cataloguing in Publication Data Hammer, Barbara Learning with recurrent neural networks, - (Lecture notes in control and information sciences) 1.Neural networks (Computer science) 2.Machine learning l.Tifie 006.3'2 ISBN 185233343X Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior perm~ion in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. © Springer-Verlag London Limited 2000 Printed in Great Britain The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore f~ee for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal respous~ility or liability for any errors or omissions that may be made. Typesetting: Camera ready by author Printed and bound at the Athenaeum Press Ltd., Gateshead, Tyne & Wear 69/3830-543210 Printed on acid-free paper SPIN 10771572

To Sagar and Volker.

Preface

A challenging question in machine learning is the following task: Is it possible to combine symbolic and connectionistic systems in some mechanism such that it contains the benefits of both approaches? A satisfying answer to this question does not exist up to now. However, approaches which tackle small parts of the problem exist. This monograph constitutes another piece in the puzzle which eventually may become a running system. It investigates so-called folding networks - neural networks dealing with structured, i.e., symbolic inputs. Classifications of symbolic data in a connectionistic way can be learned with folding neural networks which are the subject of this monograph. This capability is obtained in a very natural way via enlarging standard neural networks with appropriate recurrent connections. Several successful applications in classical symbolic areas exist - some of which are presented in the second chapter of this monograph. However, the main aim of this monograph is a precise mathematical foundation of the ability to learn with folding networks. Apart from the in-principle guarantee that folding networks can succeed in concrete tasks, this investigation yields practical consequences: Bounds on the number of neurons which are sufficient to represent the training data are obtained. Furthermore, explicit bounds on the generalization error in a concrete learning scenario can be derived. Finally, the complexity of training is investigated. Moreover, several results of general interest in the context of neural networks and learning theory are included in this monograph since they form the basis of the results for folding networks: Approximation results for discrete time recurrent neural networks, in particular explicit bounds on the number of neurons and a short proof of the super-Turing universality of sigmoidal recurrent networks, are presented. Several contributions to distribution-dependent learnability, an answer to an open question posed by Vidyasagar, and a generalization of the luckiness framework are included. The complexity of standard feed-forward networks is investigated and several new results on the so-called loading problem are derived in this context. Large parts of the research reported in this monograph were performed while I was preparing my Ph.D. thesis in Theoretical Computer Science at the Universiy of Osnabriick. I am pleased to acknowledge Thomas Elsken,

VIII Johann Hurink, Andreas Kfichler, Michael Schmitt, Jochen Steil, and Matei Toma for valuable scientific discussions on the topics presented in this volume. Besides, I had the opportunity to present and improve parts of the results at several institutes and universities, in particular during visits at Rutgers University (U.S.A.) and the Center for Artificial Intelligence and Robotics (Bangalore, India). I am deeply indebted to Bhaskar Dasgupta, Eduardo Sontag, and Mathukumalli Vidyasagar for their hospitality during my stay at Rutgers University and CAIR, respectively. Furthermore, I gratefully acknowledge the people whose encouragement was of crucial support, in particular Grigoris Antoniou, Ute Matecki, and, of course, Manfred. Special thanks go to Teresa Gehrs who improved the English grammar and spelling at the same rate as I introduced new errors, to Prof. Manfred Thoma as the Editor of this series, and Hannah Ransley as the engineering editorial assistant at Springer-Verlag. Finally, I would like to express my gratitude to my supervisor, Volker Sperschneider, who introduced me to the field of Theoretical Computer Science. He and Mathukumalli Vidyasagar, who introduced me to the field of learning theory - first through his wonderful textbook and subsequently in person - laid the foundation for the research published in this volume. I would like to dedicate this monograph to them.

Osnabriick, April 2000.

Table of C o n t e n t s

..............................................

lo

Introduction

2.

Recurrent and Folding Networks .......................... 2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 T r a i n i n g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 B a c k g r o u n d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 A p p l i c a t i o n s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 T e r m Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 L e a r n i n g Tree A u t o m a t a . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 C o n t r o l of Search H e u r i s t i c s for A u t o m a t e d D e d u c t i o n 2.4.4 Classification of C h e m i c a l D a t a . . . . . . . . . . . . . . . . . . . . 2.4.5 L o g o Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.1 3.2

3.3

3.4 .

Ability .................................... Foundations ........................................... A p p r o x i m a t i o n in P r o b a b i l i t y . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 I n t e r p o l a t i o n of a F i n i t e Set of D a t a . . . . . . . . . . . . . . . . 3.2.2 A p p r o x i m a t i o n of a M a p p i n g in P r o b a b i l i t y . . . . . . . . . . 3.2.3 I n t e r p o l a t i o n w i t h a = H . . . . . . . . . . . . . . . . . . . . . . . . . . A p p r o x i m a t i o n in t h e M a x i m u m N o r m . . . . . . . . . . . . . . . . . . . . 3.3.1 N e g a t i v e E x a m p l e s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 A p p r o x i m a t i o n on U n a r y Sequences . . . . . . . . . . . . . . . . 3.3.3 Noisy C o m p u t a t i o n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.4 A p p r o x i m a t i o n on a F i n i t e T i m e I n t e r v a l . . . . . . . . . . . . Discussion a n d O p e n Q u e s t i o n s . . . . . . . . . . . . . . . . . . . . . . . . . .

Approximation

Learnability .............................................. 4.1 T h e L e a r n i n g S c e n a r i o . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 D i s t r i b u t i o n - d e p e n d e n t , M o d e l - d e p e n d e n t L e a r n i n g . . . 4.1.2 D i s t r i b u t i o n - i n d e p e n d e n t , M o d e l - d e p e n d e n t L e a r n i n g . 4.1.3 M o d e l - f r e e L e a r n i n g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 D e a l i n g w i t h Infinite C a p a c i t y . . . . . . . . . . . . . . . . . . . . . 4.1.5 V C - D i m e n s i o n of N e u r a l N e t w o r k s . . . . . . . . . . . . . . . . . . 4.2 P A C L e a r n a b i l i t y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 5 5 11 13 15 15 15 16 16 18 19 20 25 25 30 35 36 36 39 44 46 48 51 53 54 57 58 60 62 63

x

Table of Contents

4.3

4.4 4.5 4.6 .

6.

4.2.1 D i s t r i b u t i o n - d e p e n d e n t L e a r n i n g . . . . . . . . . . . . . . . . . . . 4.2.2 Scale Sensitive T e r m s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Noisy D a t a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.4 M o d e l - f r e e L e a r n i n g . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.5 D e a l i n g w i t h Infinite C a p a c i t y . . . . . . . . . . . . . . . . . . . . . B o u n d s on t h e V C - D i m e n s i o n of F o l d i n g N e t w o r k s . . . . . . . . . . 4.3.1 T e c h n i c a l D e t a i l s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 E s t i m a t i o n o f t h e V C - D i m e n s i o n . . . . . . . . . . . . . . . . . . . 4.3.3 Lower B o u n d s on fat~ ( F ) . . . . . . . . . . . . . . . . . . . . . . . . . . C o n s e q u e n c e s for L e a r n a b i l i t y . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lower B o u n d s for t h e L R A A M . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion a n d O p e n Q u e s t i o n s . . . . . . . . . . . . . . . . . . . . . . . . . .

Complexity ............................................... 5.1 T h e L o a d i n g P r o b l e m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 T h e P e r c e p t r o n Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 P o l y n o m i a l S i t u a t i o n s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 N P - R e s u l t s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 T h e S i g m o i d a l C a s e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Discussion a n d O p e n Q u e s t i o n s . . . . . . . . . . . . . . . . . . . . . . . . . .

103 105 110 110 113 122 130

Conclusion

133

Bibliography Index

63 66 70 74 75 79 79 83 89 93 97 98

...............................................

..................................................

.........................................................

137 145

Chapter 1

Introduction

In many areas of application, computers far outperform human beings as regards both speed and accuracy. In the field of science this includes numerical computations or computer algebra, for example, and in industrial applications hardly a process runs without an automatic control of the machines involved. However, some tasks exist which turn out to be extremely difficult for computers, whereas a human being can manage them easily: comprehension of a spoken language, recognition of people or objects from their image, driving cars, playing football, answering incomplete or fuzzy questions, to mention just a few. The factor linking these tasks is that there is no exact description of how humans solve the problems. In contrast to the area in which computers are used very successfully, here a detailed and deterministic program for the solution does not exist. Humans apply several rules which are fuzzy, incomplete, or even partially contradictory. Apart from this explicit knowledge, they use implicit knowledge which they have learned from a process of trial and error or from a number of examples. The lack of exact rules makes it difficult to tell a robot or a machine to behave in the same way. Exact mathematical modeling of the situation seems impossible or at least very difficult and time-consuming. The available rules are incomplete and not useful if they are not accompanied by concrete examples. Such a situation is addressed in the field of machine learning. Analogously to the way in which humans solve these tasks, machines should be able to learn a desired behavior if they are aware of incomplete information and a number of examples instead of an exact model of the situation. Several methods exist to enable learning to take place: Frequently, the learning problem can be formulated as the problem of estimating an unknown mapping f if several examples (x ~ f(x)) for this mapping are given. If the input and output values are discrete, e.g., symbolic values, a symbolic approach like a decision tree can be applied. Here the input for f consists of several features which are considered in a certain order depending on the actual i n p u t . The exact number and order of the considered attributes is stored in a tree structure, the actual output of a special input can be found at the leaves of such a decision tree. Learning means finding a decision tree which represents a mapping such that the examples are mapped correctly and the entire tree has the most simple structure. The learning process is successful if not only

2

I. Introduction

the known examples but even the underlying regularity is represented by the decision tree. In a stochastic learning method, the function class which is represented by the class of decision trees in the previous learning method is substituted by functions which correspond to a probability distribution on the set of possible events. Here the output in a certain situation may be the value such that the probability of this output, given the actual input, is maximized. Learning means estimating an appropriate probability which mirrors the training data. Neural networks constitute another particularly successful approach. Here the function class is formed by so-cailed neural networks which are motivated by biological neural networks, that means the human brain. They compute complex mappings within a network of single elements, the neurons, which each implement only very simple functions. Different global behavior results from a variation of the network structure, which usually forms an arbitrary acyclic graph, and a variation of the single parameters which describe the exact behavior of the single neurons. Since the function class of neural networks is parameterized, a learning algorithm is a method which chooses an appropriate number of parameters and appropriate values for these parameters such that the function specified in this way fits to the given examples. Artificial neural networks mirror several aspects of biological networks like a massive parallelism improving the computation speed, redundancy in the representation in order to allow fault tolerance, or the implementation of a complex behavior by means of a network of simple elements, but they remain a long way away from the power of biological networks. Artificial neural networks are very successful if they can classify vectors of real numbers which represent appropriately preprocessed data. In image processing the data may consist of feature vectors which are obtained via Fourier transformation, for example. But when confronted with the raw image data instead, neural networks have little chance of success. (From a theoretical point of view they should even succeed with such complex data assuming that enough training examples are available - but it is unlikely that success will come about in practice in the near future.) The necessity of intense preprocessing so that networks can solve the learning task in an efficient way is of course a drawback of neural networks. Preprocessing requires specific knowledge about the area of application. Additionally, it must be fitted to neural networks, which makes a time-consuming trial and error process necessary in general. But this problem is not limited to the field of neural networks: Any other learning method also requires preprocessing and adaption of the raw data. Up to now, no single method is known which can solve an entire complex problem automatically instead of only a small part of the problem. This difficulty suggests the use of several approaches simultaneously, each solving just one simple part of the learning problem. If different methods are used for the different tasks because various approaches seem suitable for the different subproblems, a unified interface between the single methods

1. Introduction

3

is necessary such that they can exchange their respective data structures. Additionally, the single methods should be able to deal with data structures which are suitable to represent the entire learning methods involved. Then they can control parts of the whole learning process and partially automate the modularization of the original task. Here a problem occurs if neural networks are to be combined with sym: bolic approaches. Standard neural networks deal with real vectors of a fixed dimension, whereas symbolic methods work with symbolic data, e.g., terms, formulas, . . . , i.e., data structures of an in principle unbounded length. If standard networks process these data, either because they classify the output of another learning method or because they act as global control structures of the learning algorithms involved, it becomes necessary to encode the structured data in a real vector. But universal encoding is not generally fitted to the specific learning task and to the structure of the artificial network which uses the encoded values as inputs. In an encoding process an important structure may be hidden in such a way that the network can hardly use the information; on the contrary, redundant encoding can waste space with superfluous information and slow down the training process and generalization ability of the network. An alternative to encoding structured data is to alter the network architecture. Recurrent connections can be introduced such that the network itself selects the useful part of the structured data. Indeed, recurrent networks are able to work directly on data with a simple structure: lists of a priori unlimited length. Therefore, encoding such data into a vector of fixed dimension becomes superfluous if recurrent networks are used instead of standard feedforward neural networks. A generalization of the recurrence to arbitrary tree structures leads to so-called folding networks. These networks can be used directly in connection with symbolic learning methods, and are therefore a natural tool in any scenario where both symbolic data and real vectors occur. They form a very promising approach concerning the integration of symbolic and subsymbolic learning methods. In the following, we will consider recurrent and folding networks in more detail. Despite successful practical applications of neural networks - which also exist for recurrent and folding networks - the theoretical proof of their ability to learn has contributed to the fact that neural networks are accepted as standard learning tools. This theoretical investigation implies two tasks: a mathematical formalization of learnability and a proof that standard neural networks fit into this definition. Learnability can be formalized by Valiant's paradigm of 'probably approximately correct' or PAC learnability. The possibility of learning with standard neural networks in this formalism is based on three properties: They are universal approximators, which means that they are capable of approximating anything that they are intended to approximate; they can generalize from the training examples to unseen data such that a trained network mirrors the underlying regularity that has to be

4

1. Introduction

learned, too; and fitting the network to the training examples is possible with an efficient training algorithm. In this monograph, we will examine the question of approximation capability, learnability, and complexity for folding networks. A positive answer to these questions is a necessary condition for the practical use of this learning approach. In fact, these questions have not yet been entirely solved for recurrent or standard feed-forward networks and even the term of 'probably approximately correct' learnability leaves some questions concerning the formalization of learnability unsolved. Apart from the results which show the theoretical possibility of learning with folding networks we will obtain results which are of interest in the field of recurrent networks, standard feed-forward networks, or learnability in principle, too. The volume consists of 5 chapters which are as independent as possible of each other. Chapter 2 introduces the general notation and the formal definition of folding networks. A brief description of their in-principle use and some concrete applications are mentioned. Chapters 3 to 5 each examine one of the above mentioned topics: the approximation ability, learnability, and complexity of training, respectively. They each start with a specification of the problems that will be considered in the respective chapter and some bibliographical references and end with a summary of the respective chapter. The monograph is concluded with a discussion of several open questions.

Chapter 2

Recurrent and Folding Networks

This chapter contains the formal definition of folding networks, the standard training algorithm, and several applications. Folding networks constitute a general learning mechanism which enables us to learn a function from a set of labeled k-trees with labels in some finite dimensional vector space into a finite dimensional vector space. By labeled k-trees we address trees where the single nodes are equipped with labels in some alphabet and every node has at most k successors. This sort of data occurs naturally in symbolic areas, for example. Here the objects one is dealing with are logical formulas or terms, the latter consisting of variables, constants, and function symbols, the former being representable by terms over some enlarged signature with function symbols corresponding to the logical symbols. Terms have a natural representation as k-trees, k being the maximum arity of the function symbols which occur: The single symbols are enumerated and the respective values can be found in the labels of a tree. The tree structure mirrors the structure of a term such that subtrees of a node correspond to subterms of the function symbol which the respective node represents. Hence folding networks enable us to use connectionistic methods in classical symbolic areas. Other areas of application are the classification of chemical data, graphical objects, or web data, to name just a few. We will show how these data can be encoded by tree structures in the following. But first we formally define folding networks which constitute a generalization of standard feed-forward and recurrent networks.

2.1 Definitions One characteristic of a neural network is that a complex mapping is implemented via a network of single elements, the neurons, which in each case implement a relatively simple function. D e f i n i t i o n 2.1.1. A feed-forward neural network consists of a tuplc F = (N, - r , w, 8, f, I, 0 ) . N is a finite set, the set of neurons or units, and (N, ~ ) is an acyclic graph with vertices in N x N . We write i --+ j if neuron i is connected to neuron j in this graph. Each connection i -+ j is equipped with

6

2. Recurrent and Folding Networks input neurons

hidden n e u r o n s

t

o1 w I

/

~'9

02 03 / w 3

output neurons

f(w

o)

activation function

Fig. 2.1. Example for a feed-forward network with a multilayered structure; given an input, the neurons compute successively their respective activation.

a weight wij E R, w is the vector of all weights. The neurons without predecessor are called input neurons and constitute the set I. All other neurons are called c o m p u t a t i o n units. A nonempty subset of the computation units is specified and called o u t p u t units, denoted by O. All computation units, which are not output neurons, are called hidden neurons. Each computation unit i is equipped with a bias 0i E R and an activation function fi : IR --+ IR. 0 and f are the corresponding vectors. We assume w.l.o.g, that N C N and that the input neurons are { 1 , . . . , m } . A network with m inputs and n outputs computes the function

f : ~ m _+ Rn, I(zl

.... , z,,)

= (oil,...,

o~.)

where il, . . . , in are the output units and oi is defined reeursivcly for any neuron i by

oi =

xi f i ( Y ~ j ~ i wjioj + Oi)

if i is an input unit, otherwise.

The term ~ j - ~ i wjioj + Oi is called the activation of neuron i. A n architecture is a tuple Y = ( N , ~ , w ' , O ' , f , I , O ) as above, with the difference that we require w ' and O' to specify a weight w~j for only some of the connections i ~ j , or a bias O~ for some of the neurons i, respectively.

2.1 Definitions H(x)

l sgd(x)

~X

7

~ lin(x)

mX

~X

Fig. 2.2. Several activation functions: perceptron activation H, standard sigmoidal function sgd, and semilinear activation lin; all three functions are squashing functions. In an obvious way, an architecture stands for the set of networks that results from the above tuple by completing the specification of the weight vector and biases. By an abuse of notation, the set of functions that result from these networks is sometimes called an architecture, too. Since these functions all have the same form, we sometimes denote the architecture by a symbol which typically denotes such a function. For a concrete network F the corresponding architecture refers to the architecture where no wii or 0i are specified. A network computes a mapping which is composed of several simple functions computed by the single neurons. See Fig.2.1 as an example. The activation functions fi of the single computation units are often identical for all units or all but the o u t p u t units, respectively. We drop the subscript i in these cases. The following activation functions will be considered (see Fig.2.2): The identity id : I~ -~ R, id(x) = x, which is often used as an output activation function in order to implement a scaling or shifting of the desired output domain; the perceptron activation H : R ~ R, H(x)=/

0 ( 1

x<0 x_>0

'

which is a binary valued function occurring mostly in theoretical investigations of networks. In practical applications the activation functions are similar to the perceptron activation for large or small values and have a smooth transition between the two binary values. The standard sigmoidal ]unction sgd(x) = (1 + e-Z) -1 and the scaled version tanh(x) = 2. sgd(2x) - 1 are common functions of this type. A more precise approximation of these sigmoidal activations which is sometimes considered in theoretical investigations is the semilinear ]unction lin(x)=

1 x 0

x>l xE]0,1[ x_<0

,

8

2. Recurrent and Folding Networks

which fits the asymptotic behavior of sgd and the linearity at the point 0. For technical reasons we will consider the square activation x ~ x 2. By a squashing activation we mean any function which is monotonous with function values that tend to 1 for large inputs x and to 0 for small input values x, respectively. A function is C n if it is n times continuously differentiable. A p r o p e r t y of a function holds locally if it is valid in the neighborhood of at least one point. For technical reasons we will consider multiplying units, which simply compute a product of the output values of their predecessing units instead of a weighted sum, i.e., Oi = Hj--~i Oj for multiplying units i. The connection structure --+ often has a special form: the neurons decompose into several groups No . . . . , Nh+l, where No are the input neurons, Nh+l are the output neurons, and i ~ j if and only if i E Nk, j E Nk+l for some k. Such a network is called a multilayer ]eed-forward network, or M L P for short, with h hidden layers; the neurons in Ni constitute the hidden layer number i for i E { 1 , . . . , h } . Feed-forward networks can handle real vectors of a fixed dimension. More complex objects are trees with labels in a real vector space. We consider trees where any node has a fixed fan-out k, which means t h a t any n o n e m p t y node has at most k successors, some of which m a y be the e m p t y tree. Hence, a tree with labels in a set 57 is either the e m p t y tree which we denote by _L or it consists of a root which is labeled with some value a E 2: and k subtrees tl, .. 9 tk some of which may be empty. In the latter case we denote the tree by a(tl . . . . , tk). The set of trees which can be defined as above is denoted by 57~. In the following, 57 is a finite set or a real vector space. One can use the recursive nature of trees to construct an induced mapping which deals with trees as inputs from any vector valued mapping with a p p r o p r i a t e arity:

Definition 2.1.2. Assume R is a set. Any mapping g : 57 x R k ~ R and initial context y E R induces a mapping ~ : 57~, -r R, which is defined recursively as follows: ~v(a(tl,...,tk))

=

y,

=

g(a,~(tl),...,#~(tk)).

This definition can be used to formally define recurrent and folding networks: D e f i n i t i o n 2.1.3. A folding network consists of two feed-forward networks which compute g : 1~'~+h't ~ ~ and h : ~ ~ R n , respectively, and an initial context y e I~. It computes the mapping

h o ~ : (Xm)~ -~ X". A folding architecture is given by two feed-forward architectures 3: with m + k 9 l inputs and l outputs, and ~ with l inputs and n outputs and an only partially defined initial context y~ in ~ .

2.1 Definitions folding n e t w o r k g i v e n by the networks:

c o m p a c t notation:

g

', ,_ . . . . . . . . . . . . . .

e n c o d i n g layer , ..... i

context-

thea input tree

"

..L---*

neurons .... ' : i ~',

b/N,c I A de

~

recurrentg f

d

,; ....................

-,

i~'l

I

;o'

,,

. . . . . . :i . .". . . L . . . .Y ............

"'

feed-forwardpart h

~=

J-

b

-L

leads to the computation

9

a

.L Q

e _1_

c

J_ f .1_

~ /

computationinduced by g

A_ Fig. 2.3. Example for the unfolding process if a concrete value is computed with a very simple folding network; the folding network is formally unfolded according to the structure of the input tree; for every subtree one copy of the recursive part of the folding network can be found in the computation.

The input neurons m + 1, . . . , m + k . l of g are called context neurons. g is referred to as the recursive part of the network, h is the feed-forward part. The input neurons of a folding network or architecture are the neurons "1, . . . , m of g, the output units are the output neurons of h. I l k = 1 we call folding networks recurrent networks.

Specifying all weights and biases in an architecture given by j r and ~ and all coefficients in y ' leads to a class of folding networks. As before, we sometimes identify the function class which can be c o m p u t e d by these folding networks with the folding architecture. To understand how a folding network computes a function value one can think of the recursive p a r t as an encoding part: a tree is encoded recursively into a real vector in 1~. Starting at the e m p t y tree Z, which is encoded by the initial context y, a leaf a ( l , . . . , • is encoded via ] as f ( a , y , . . . , y ) using the code of • Proceeding in the same way, a subtree a ( t l , . . . ,tk) is

I0

2. Recurrent and Folding Networks

recurrent network

input

_ ~

a ~ . ~ f ~ ~ _ , ~

C

• Fig. 2.4. A simple recurrent neural network: here the recurrent connection is a one-to-one correspondence due to the linearity of the input structure.

encoded via f using the already computed codes of the k subtrees tl, . . . , tk. The feed-forward part maps the encoded tree to the desired output value. We refer to l as the encoding dimension. A folding network which takes binary trees with real valued nodes as inputs is depicted in Fig. 2.3. In a concrete computation, part g is applied several times according to the structure of the input. Recurrent links in part g indicate the implicit recurrence which depends on the input structure. The input tree a(b(d(.J_, • .J_),c(e(J_, .l_), f(_l_, .1_))), for example, is mapped to the value h(g(a, g(b, g(d, y, y), y), g(c, g(e, y, y), g(f, y, y)))). Note that for each fixed structure of input trees one can find an equivalent feed-forward network which computes the same output by simply unfolding the folding network. According to the recurrence, several weights are identical in this unfolded network. For k = 1, the inputs consist of sequences of real vectors in IRrn . Here the recurrent connections have a one-to-one correspondence since the unfolding process is linear (see Fig. 2.4). In the case k = 1 we will drop the subscript k. Then a tree (al(a2(a3(... (a,(.J_))...)))) is denoted by [an,.-. ,a2,al]. Any mappings we consider are total mappings unless stated otherwise. A mapping between real vector spaces is measurable or continuous, respectively, if it is measurable with respect to the standard Borel a-algebra or continuous with respect to the standard topology, respectively. A mapping f : (R'n)~ --+ R n is measurable or continuous if and only if any restriction of f to trees of a fixed structure is measurable or continuous as a mapping between real vector spaces. This latter requirement defines the topology and a-algebra, respectively, on the set of trees. In particular, the set which consists of all trees of one fixed structure is open and measurable. If we restrict our consideration to trees of a fixed structure, a set is open or measurable if and only if it is

2.2 Training

11

open or measurable, respectively, with respect to the standard topology or a-algebra on a real vector space. In the following, we denote by ,U
2.2 Training

In a concrete learning task several examples (xi, f(xi))~= l, the training set, of an unknown function are available. Here the xi E ( ~ ) ~ are vectors, sequences, or trees, depending on the learning task, and f : (R 'n)~ ~ R n is a vector valued function which is to be learned. For a fixed folding architecture h o ~y with m inputs and n outputs, where all weights and biases are unspecified, we can define the quadratic error P

E(w) = Z

If(x~) - h o ~y(Xi)] 2

i----1

which depends on the weights and biases w. Often, the initial context y is chosen as some fixed vector, the origin, for example. Minimizing E ( w ) with respect to the weights yields a folding network which outputs on data xi some value as similar as possible to the desired output f(xi). Since the activation functions used in practice are usually differentiable, the error minimization can be performed with a simple gradient descent method, i.e., some procedure of the form: w

:= s o m e

initial

values

while E(w) is large w

:= w - ~ . V E ( w )

with some value 0 < 7/. For appropriate 7/and appropriate starting point w, this converges to a local optimum of the function E with positive definite Hessian. The gradient V E ( w ) can be computed very efficiently, i.e., in linear time with respect to the number of weights, using a method called backpropagation [105, 136, 138] for feed-forward architectures, back-propagation through time for recurrent architectures [90, 106, 137], and back-propagation through structure for folding architectures [41, 42, 73]. In order to speed up the above procedure one can adapt factor ~/ appropriately or take secondorder information into account. To avoid local minima one can use heuristic modifications of the above rule or add some form of randomization to the process [53, 103, 141]. Hence this algorithm allows us to find weights for a specific architecture such that the examples are fitted. Commonly, the architecture is chosen by a trial and error process. More precisely: One fixes several different architectures which seem appropriate - the proceeding chapters will give us hints of how this can be made more precise. Roughly, they yield upper bounds on the

12

2. Recurrent and Folding Networks

number of neurons such that the capability of bringing the quadratic error to zero or the in-principle possibility of valid generalization to unseen examples, respectively, is guaranteed. Hence the search space for an appropriate architecture becomes finite. After choosing several architectures, the quality of each architecture is estimated as follows: Each architecture is trained on one part of the training set with (a modification of) the above training method. Afterwards, the quadratic error on the remaining part of the training set is used to estimate the quality of the architecture. Obviously, one cannot use the quadratic error on the training set because in general, it will be smaller than the value one is interested in, the deviation of the network function from the function that is to be learned. Hence one uses the quadratic error on a set not used for training as an estimation of f If(x) - h o ~ y ( X ) [ 2 d p , P denoting the probability measure on the input trees in accordance to which the input patterns are chosen. Unfortunately, this method yields a large variance. The variance can be reduced if the single architectures are trained and tested several times on different divisions of the training data using so-called cross-validation [125]. Hence we get a ranking of the different architectures and can use the best architecture for the final network training as described above. This roughly describes the in-principle learning method. Note that this yields a training method, but up to now no theoretical justification of its quality. For feed-forward networks the training method is well founded because their universal approximation capability and information theoretical and complexity theoretical learnability are (with some limitations) well understood. These theoretical properties do not transfer to folding networks immediately. The necessary investigations are the topic of this volume and altogether establish the in-principle possibility of learning with the method as described above. However, a couple of refinements and modifications of the training algorithm exist for folding networks as for standard feed-forward networks. We mention just a few methods, the ideas of which can be transferred immediately: integration of prior knowledge via a penalty term in the quadratic error which penalizes solutions contradicting the prior information or regularization of the network via weight restriction [21, 96]; modification of the architecture during the training or simplification of the final network via pruning less important weights and neurons [77, 92]. One problem is to be dealt with, while training recurrent and folding architectures, which does not occur while training feed-forward networks: the problem of long-term dependencies [16]. Roughly, the problem tells us that the training scenario is numerically ill-behaved if a function is to be learned where entries at the very beginning of long sequences or labels near the leaves of deep trees determine the output of the function. Hence several approaches try to modify the architecture, the training algorithm, or both

2.3 Background

13

[14, 40, 54, 67]. This problem has not yet been satisfactorily solved; we will consider some correlated theoretical problems in Chapter 5.

2.3 B a c k g r o u n d The above argumentation tells us that a large variety of training methods are available in order to use folding networks in practice. Before describing several areas of application in more detail, we want to mention the origin of the approach. Despite the success of neural networks in several different areas dealing with continuous data [25, 121], their ability to process symbolic data was doubted in [34], for example. One of the main criticisms is that a natural representation of symbolic data in a finite dimensional vector space does not exist because symbolic data is highly structured and consists of a n a priors unlimited length. Since standard networks only deal with distributed representations of the respective data in a finite dimensional vector space, they cannot be used for processing symbolic data. In reaction to the criticism [34], several approaches including the RAAM and LRAAM [101, 122, 124], tensor constructions [117], BoltzCONS [128], and holographic reduced representations [100} were proposed as mechanisms to deal with some kind of structured data. One key problem is to find a mechanism that maps the structured data into a subsymbolic representation, i.e., a real vector of fixed dimension. For this purpose, connectionistic models are equipped with an appropriate recurrent dynamics in the above approaches. Actually, the RAAM and LRAAM already contain the same dynamics as folding networks, but training proceeds in a different way. They consist of two parts, the first of which is used for recursive encoding of trees into a finite dimensional vector space, the second of which performs a recursive decoding of finite dimensional vectors to trees. Formally, the LRAAM consists of two networks ] and g with input dimension m + k. n or n, respectively, and output dimension n or m + k -n, respectively. It computes h y o ~ where h r : •n __+ (Rm)~ is defined by hy(x) = { _l_ ho(x) (hy (hi ( x ) ) , . . . , hy (hk (x)))

x e Y otherwise,

where Y C R '~ a n d h = ( h o , h l , . . . , h j , ) . Hence ho outputs the label, hi, 9.., hk compute codes for the k subtrees, h performs a dual computation compared to ~. The LRAAM performs some kind of universal encoding of tree structures. The two networks g and h are trained simultaneously such that the composition yields the identity on k-trees. For this purpose, some kind of truncated gradient descent method is used [101]. Afterwards, one can address any learning task that deals with tree structures as inputs or outputs with a standard network, since the composition of a standard network with one or two parts of the LRAAM reduces the learning task to the learning of a mapping between finite dimensional vector spaces. See Fig.2.5 as an illustration.

14

2. Recurrent and Folding Networks encoding

decoding

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

.!i

a

A b

A

d

c

e

\

i

a

b f

c

I i

d

I

~

e

f

-

may be composed with a standard network Fig. 2.5. The LRAAM performs both encoding and decoding. Hence composition of these parts with a standard network can be used to learn arbitrary mappings with trees as input or output. Folding networks deal with only one part of the problem: mappings from tree structures into a real vector space. Hence they use only the encoding part of the LRAAM and combine it directly with a standard network. The training is fitted to the specific learning task. This method allows them to store only the valuable information that is necessary for the specific task and makes the efficient processing of the desired information possible. Additionally, it will turn out in Chapter 4 that folding networks are more likely to succeed from a theoretical point of view compared to the LRAAM. The other approaches mentioned above have in common the fact that complex data is encoded into a representation as a real vector which is universal and not fitted to the specific learning problem, either. The tensor construction, BoltzCONS, and the holographic reduced representation deal with a fixed encoding which is not learned or modified at all. As already mentioned, RAAM and LRAAM learn the encoding, but only the structure of the input data is relevant for this process - the classification task is not taken into account. In contrast, recurrent and folding networks fit the encoding to the specific learning task. Actually, this method is already used very successfully in the special case of linear trees, i.e., lists. Recurrent networks are a natural tool in any domain where time plays a role, such as speech recognition and production, control, or time series prediction, to name just a few [39, 43, 80, 87, 91,126, 140]. They are also used for the classification of symbolic data as DNA sequences [102]. Elman networks are a special case [29, 30], with the main difference that the learning algorithm is only a truncated gradient descent and hence training is commonly less efficient compared to standard methods used for training recurrent networks [54]. Both architectures compete with feed-forward net-

2.4 Applications

15

works which cut the maximum length of an input sequence to a fixed value and can therefore only deal with an a priori limited time context [15, 57, 87]. Folding networks have been designed to enable neural networks to classify classical terms, logic formulas, and tree structures so that they can be used in symbolic domains, for the control of search heuristics in automated deduction [41, 111], for example. Furthermore, successful applications exist in various areas.

2.4 Applications In the following, we have a closer look at some of these applications focusing on the method of how the respective data is encoded by a tree structure.

2.4.1 Term Classification The set of terms over a finite signature and finite set of variables can be represented by trees in the following way: Denote by k the maximum arity of the function symbols, by c some injective function which maps the symbols, variables, constants, and function symbols, to real numbers. A variable X is encoded by the k-tree C ( X ) = c ( X ) ( • • a term f ( t l , . . . ,tkl) (kl <_ k) is encoded by the k-tree C ( f ( t l , . . . , fk~)) = c ( f ) ( C ( t l ) , . . . , C(tkl), • • Reasonable training tasks are to learn mappings f : {t E I~ It = C ( T ) for some term T} --~ {0,1} where f ( C ( T ) ) = 1 ~:~ T ~ is a subterm of T for some fixed term T', or f ( C ( T ) ) = 1 r T is an instance of a fixed term T' containing variables, or f ( C ( T ) ) = 1 ~ a Boolean combination of similar characteristics as above holds, . . . . Results for these kinds of training problems can be found in [42, 73]. Most of the considered tasks can be solved with folding networks h o ~y where h and g do not possess a hidden layer, with standard training methods, and more than 93% correct classified data on a test set.

2.4.2 Learning Tree A u t o m a t a A (finite, bottom-up, deterministic) tree automaton is a tuple (,U, Q, b, F, 6) where ,U, the input alphabet, and Q, (Q n ~: = 0), the set of states, are finite sets, b E Q is the initial state, F C Q is the nonempty set of final states, and 6 : ~7 • Qk ~ Q is the transition function. A tree t E Z~ is accepted by the automaton if and only if $b(t) E F. Assuming ,U C ~ natural tasks for a folding architecture are to learn the function f : (,Uk)* ~ {0, 1} with f ( t ) = 1 if and only if the tree t is accepted by some specified tree automaton. In [72], folding networks are successfully trained on tasks of this kind with standard training methods; in all cases an accuracy of nearly 100% is obtained on a test set. Furthermore, their in-principle ability of simulating

i6

2. Recurrent and Folding Networks

tree a u t o m a t a is established. Moreover, training turns out to be successful (i.e., more than 94% accuracy on a test set) for languages that are not recognizable by tree a u t o m a t a either. [72] reports very good results for the recognition of the language {t E {f, a, b}~ I t contains an instance of the term .f(X, X) as subterm}. We will deal with correlated problems concerning the computational capability of folding networks in Chapter 3. 2.4.3 C o n t r o l o f S e a r c h H e u r i s t i c s for A u t o m a t e d D e d u c t i o n Several calculi in a u t o m a t e d theorem proving, for example, the resolution calculus or model elimination [81, 104], try to resolve a goal ~ successively with a set of formulas M in order to see whether ~ follows from M or not. The proof process can be modeled by a search tree with nodes labeled with the actual states of the theorem prover and sons of the respective nodes which represent the possible proof steps. If ~ follows from M, then at least one path in this tree leads to a valid proof. See Fig. 2.6 as an example for such a proof tree. Obviously, the success of the theorem prover depends on the order according to which the single possible states are explored. Since the fan-out of the nodes in the search tree is typically larger than one at every proof step, a naive search leads to an exponential amount of time. In order to prevent this fact, one can try to find heuristics telling us which state is to be visited first. Formally, we want to find a mapping from the single nodes of the search tree into the real numbers such that a search visiting the states in accordance with the induced ranking leads to a proof in a short timeThe single states of the theorem prover consist of a finite set of logical formulas and hence can be represented by a finite set of trees in a natural way. Therefore folding networks can be used in order to learn the mapping of finite sets of trees into the real numbers which represents an appropriate ranking. In [41], this method is applied to several word problems from group theory. The training d a t a is obtained from several different (truncated) proof trees where the states on a proof path are mapped to high values and states on a failure path are mapped to low values in a first approximation. Since the same state may occur on a proof path as well as on a failure path, this naive approach does not work. Hence in [41] the quadratic error is modified such that the error is small if at least one state on every failure path is ranked with a lower value than all states on at least one proof path. The use of this modified error function training yields very good results: 17 of 19 proofs which could not be obtained within reasonable time without a ranking could be performed with the ranking which wa~ learned with a folding network and standard training methods. 2.4.4 Classification o f C h e m i c a l D a t a In [110], folding networks are used compared to standard feed-forward networks for the task of mapping chemical data (here: triazines) to their activity

2.4 Applications

Set of formulas: l(a,b), l(b,c), l(e,c). L(X,X). L(X,Y):-L(Y,X).

17

L(X,Y):-I(X,Z),L(Z,Y).

The goal L(a,e) leads to the following search tree: ?-L(a,e). 7-h(a,e). 9 ..

?-h(c,a). o ' " a - "

infinite path! . ~

.~L(c,a). ?-L(a,c). "'"

'

'

~s2~'

?-L(e,c). / ?-L(c,e). failure!

?-L(b,c).

/ ?-L(c,b). "'"

~ ?-L(c,c).

L(c,c).

L(e,e).

[]

?-L(/).~,c []

success! F i g . 2.6. Example for a search tree of an a u t o m a t e d theorem prover: The question whether L(a, e) follows from the given set leads to a search tree with several proof paths, but several failure and infinite paths as well.

(here: their ability to inhibit the reproduction of cancer cells).For this purpose the chemical data is represented by terms in an appropriate way. A typical structure is depicted in Fig. 2.7. T w o basic phenyl rings can be found in every structure of the training set, the second of which possesses at positions 3 and 4 variable structures: atoms, basic molecules, another ring, or a bridge, as an example. Hence a representation by a binary symbol ring(_,_) which captures the basic ring structure is possible. The two places indicate that at positions 3 and 4 variable structures may be found. These are represented by real values ifwe deal with atoms or basic molecules, by the symbol ring(_,_),ifanother ring isfound, or bridge(_, _),ifa bridge takes place. In the latter case, the firstplace represents the structure of the bridge itselfand the second place represents the free position. Hence we obtain a term representation, for example, the term ring(CL,bridge(CH2CH2, _)) for the structure in Fig. 2.7. In addition to the single symbols, physio-chemical attributes likepolarizabilitymay be encoded in the labels of the corresponding tree. Training on these data with standard methods yields a folding network such that the spear-man's correlation (i.e.,the correlation of the ranking induced by the network) is improved from 0.48 for feed-forward networks to 0.57 for folding networks on a test set. Note that both the labels and the structure encode valuable information in this example if physio-chemical attributes are added in the trees representation.

18

2. Recurrent and Folding Networks

CL

variable ~176176

C H 2C H

N Fig. 2.7. Structure of a triazine: Two phenyl rings form the basic structure, the second of which may possess different structures at positions 3 and 4.

2.4.5 Logo Classification Images of logos that are subject to noise are to be recognized in an approach presented in [24]. The logos are represented by trees in the following way: First, classical edge detection algorithms extract edges. These are encoded via features like their curvature, length, center point, ... and constitute the labels of the nodes. The nodes are arranged in a tree such that a node becomes the son of another node if the edge represented by the latter node surrounds the former edge by at least 270 degrees. Since this method yields trees with a large fan-out, the approach in [24] applies a method which substitutes the trees by binary trees first. For this data, the approach in [24] reports a test set accuracy of at least 88% for several different settings which are trained with folding networks and standard training algorithms. Other possible areas of application are reported in [36, 38]. Hence folding networks have turned out to be successful in practice in several different areas. In the following, we investigate a theoretical justification of this fact.

Chapter 3

Approximation Ability

In this chapter we deal with the ability of folding networks to approximate a function from structured data into a real vector space in principle. In a concrete learning task some empirical data (xi, .f(xi)) of an unknown function f which is to be learned is presented. We want to find a folding network that maps the data correctly. T h a t is, the inputs xi are mapped by the network to values approximating .f(xi). Furthermore, this approximating folding network should represent the entire function ] underlying the d a t a correctly. T h a t is, it should map any tree x to a value approximating f ( x ) even if x is different from all training examples. But we have no chance of finding such a network unless any finite set of d a t a can be correctly approximated by a folding network and, more significantly, unless any function we want to approximate can be approximated in some way by a folding network with an appropriate number of neurons. Of course, the notation of approximation is to be made more precise. A function can be approximated in the maximum norm, i.e., the network's output differs for any input at most a small value e from the desired output. In particular, trees of arbitrary height are to be predicted correctly. This means that we want to classify any tree correctly even if the tree is so high such that it will rarely occur. However, situations exist where approximation in the maximum norm seems appropriate, e.g., if the long-term behavior of a dynamic system is to be approximated. Alternatively, we could restrict the approximation to trees that seem reasonable, excluding, for example, trees of a large height. Formally, we demand only that the probability of trees where the network's output differs significantly from the desired o u t p u t can be made arbitrarily small. This demand corresponds to an approximation in probability. Since trees with more than a certain height become nearly impossible in any probability measure, this notation of approximation restricts the consideration to inputs with only a restricted recurrence. It would be nice to not only ensure the approximation capability for some maybe very large - network but to find explicit bounds on the number of hidden layers and neurons that are necessary in such an architecture. Bounds on the number of hidden layers and neurons limit a priori the architectures -

20

3. Approximation Ability

we have to consider unless a correct representation of the empirical data is found. The following questions are worth considering as a consequence: 1. Can any finite set of data be approximated or interpolated, respectively, by a folding network? 2. Can any reasonable function be approximated in probability? 3. Can any reasonable function be approximated in the maximum norm? 4. Can the number of neurons and hidden layers that ensure the approximation capability be limited? The first two questions are answered with 'yes' in section 3: Any finite set of data can be interpolated and any measurable function can be approximated in probability. Furthermore, to interpolate a set of p patterns, it is sufficient to use a number of neurons which is quadratic in p for an activation function like the sigmoidal function. It is sufficient to consider folding networks with only a constant number of layers to approximate a measurable function. On the contrary, the third question is answered with 'no' in general in section 4. But first we introduce some notation.

3.1 Foundations We start with a formal definition of the term 'approximation' that we consider here. D e f i n i t i o n 3.1.1. A s s u m e X is a set and Y C R n /or some n. A function f : X - r Y is approximated by a function 9 : X ~ Y in the maximum norm w/th accuracy e if sup If(z) - g(z)[ < e. zEX

A s s u m e X is equipped with a a-algebra and P is a probability measure on X . A measurable function f : X ~ Y is approximated by a measurable function g : X --r Y in probability with accuracy e and confidence ~ if P(x E X [If(x)-g(x)l

> e) < 5.

The situation where a finite set of data, i.e., X C Rn is finite, is approximated or even interpolated, i.e., e = 0, is of particular interest when dealing with a learning algorithm which tries to fit a network such that it represents some empirical data. In the following, X C Rm will be an arbitrary subset or X C (R m)~ will contain sequences or trees if f is a recursive mapping. Y will be the real vector space Rn or a compact subset of this space. Any result that leads to a positive or negative fact concerning these two formalisms is called an approximation result in the following. For feed-forward networks it is shown in [59] that any continuous function R m ~ Rn with compact domain can be approximated arbitrarily well

3.1 Foundations

21

in the maximum norm, and any Borel-measurable function can be approximated arbitrarily well in probability with a network with only one hidden layer with squashing activation function and linear output. Additionally, it is shown in [58] that it is possible to approximate a continuous function in the maximum norm or a measurable function in probability, respectively, with single hidden layer networks and bounded weights if the activation function of the hidden nodes is locally Riemann integrable and nonpolynomial. Furthermore, to interpolate p points with outputs in R exactly by such a network with unbounded weights it is sufficient to use only p neurons in the hidden layer [119]. For an input dimension m, o u t p u t dimension n, and continuously differentiable activation with some other conditions, this bound is improved to 2 p m / ( n + m ) hidden neurons, which are sufficient for the interpolation of p points in general position [28]. We will use these results in our constructions in order to show some universal approximation property for folding networks. The respective feed-forward networks will be accompanied by a recursive network. The recursive network encodes the respective data in a finite dimensional vector space and hence reduces the approximation problem to a problem for feed-forward networks. However, as a consequence of these results folding networks with appropriate activation functions can approximate any function of the special form g o ]y : K