Subseries of Lecture Notes in Computer Science
3
Berlin Heidelberg New York Hong Kong London Milan Paris Tokyo
Algorithmic Learning Theory 14th International Conference, ALT 2003 Sapporo, Japan, October 17-19, 2003 Proceedings
13
Volume Editors Ricard Gavaldà Technical University of Catalonia Department of Software (LSI) Jordi Girona Salgado 1-3, 08034 Barcelona, Spain E-mail:
[email protected] Klaus P. Jantke Deutsches Forschungszentrum für Künstliche Intelligenz GmbH Im Stadtwald, Geb. 43.8, 66125 Saarbrücken, Germany E-mail:
[email protected] Eiji Takimoto Tohoku University Graduate School of Information Sciences Sendai 980-8579, Japan E-mail:
[email protected] Cataloging-in-Publication Data applied for A catalog record for this book is available from the Library of Congress. Bibliographic information published by Die Deutsche Bibliothek Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data is available in the Internet at
.
CR Subject Classification (1998): I.2.6, I.2.3, F.1, F.2, F.4.1, I.7 ISSN 0302-9743 ISBN 3-540-20291-9 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2003 Printed in Germany Typesetting: Camera-ready by author, data conversion by PTP-Berlin GmbH Printed on acid-free paper SPIN: 10963852 06/3142 543210
Preface
This volume contains the papers presented at the 14th Annual Conference on Algorithmic Learning Theory (ALT 2003), which was held in Sapporo (Japan) during October 17–19, 2003. The main objective of the conference was to provide an interdisciplinary forum for discussing the theoretical foundations of machine learning as well as their relevance to practical applications. The conference was co-located with the 6th International Conference on Discovery Science (DS 2003). The volume includes 19 technical contributions that were selected by the program committee from 37 submissions. It also contains the ALT 2003 invited talks presented by Naftali Tishby (Hebrew University, Israel) on “Efficient Data Representations that Preserve Information,” by Thomas Zeugmann (University of L¨ ubeck, Germany) on “Can Learning in the Limit be Done Efficiently?”, and by Genshiro Kitagawa (Institute of Statistical Mathematics, Japan) on “Signal Extraction and Knowledge Discovery Based on Statistical Modeling” (joint invited talk with DS 2003). Furthermore, this volume includes abstracts of the invited talks for DS 2003 presented by Thomas Eiter (Vienna University of Technology, Austria) on “Abduction and the Dualization Problem” and by Akihiko Takano (National Institute of Informatics, Japan) on “Association Computation for Information Access.” The complete versions of these papers were published in the DS 2003 proceedings (Lecture Notes in Artificial Intelligence Vol. 2843). ALT has been awarding the E. Mark Gold Award for the most outstanding paper by a student author since 1999. This year the award was given to Sandra Zilles for her paper “Intrinsic Complexity of Uniform Learning.” This conference was the 14th in a series of annual conferences established in 1990. Continuation of the ALT series is supervised by its steering committee, consisting of: Thomas Zeugmann (Univ. of L¨ ubeck, Germany), Chair, Arun Sharma (Univ. of New South Wales, Australia), Co-chair, Naoki Abe (IBM T.J. Watson Research Center, USA), Klaus Peter Jantke (DFKI, Germany), Phil Long (National Univ. of Singapore), Hiroshi Motoda (Osaka Univ., Japan), Akira Maruoka (Tohoku Univ., Japan), Luc De Raedt (Albert-Ludwigs-Univ., Germany), Takeshi Shinohara (Kyushu Institute of Technology, Japan), and Osamu Watanabe (Tokyo Institute of Technology, Japan). We would like to thank all individuals and institutions who contributed to the success of the conference: the authors for submitting papers, the invited speakers for accepting our invitation and lending us their insight into recent developments in their research areas, as well as the sponsors for their generous financial support. Furthermore, we would like to express our gratitude to all program committee members for their hard work in reviewing the submitted papers and participating in on-line discussions. We are also grateful to the external referees whose reviews made a considerable contribution to this process.
VI
Preface
We are also grateful to the DS 2003 Chairs Yuzuru Tanaka (Hokkaido University, Japan), Gunter Grieser (Technical University of Darmstadt, Germany) and Akihiro Yamamoto (Hokkaido University, Japan) for their efforts in coordinating with ALT 2003, and to Makoto Haraguchi and Yoshiaki Okubo (Hokkaido University, Japan) for their excellent work on the local arrangements. Last but not least, Springer-Verlag provided excellent support in preparing this volume.
August 2003
Ricard Gavald` a Klause P. Jantke Eiji Takimoto
Organization
Conference Chair Klaus P. Jantke
DFKI GmbH Saarbr¨ ucken, Germany
Program Committee Ricard Gavald` a (Co-Chair) Eiji Takimoto (Co-Chair) Hiroki Arimura Shai Ben-David Nicol` o Cesa-Bianchi Nello Cristianini Fran¸cois Denis Kouichi Hirata Sanjay Jain Stephen Kwek Phil Long Yasubumi Sakakibara Rocco Servedio Hans-Ulrich Simon Frank Stephan Christino Tamon
Tech. Univ. of Catalonia, Spain Tohoku Univ., Japan Kyushu Univ., Japan Technion, Israel Univ. di Milano, Italy UC Davis, USA LIF, Univ. de Provence, France Kyutech, Japan Nat. Univ. Singapore, Singapore Univ. Texas, San Antonio, USA Genome Inst. Singapore, Singapore Keio Univ., Japan Columbia Univ., USA Ruhr-Univ. Bochum, Germany Univ. Heidelberg, Germany Clarkson Univ., USA
Local Arrangements Makoto Haraguchi (Chair) Yoshiaki Okubo
Hokkaido Univ., Japan Hokkaido Univ., Japan
Subreferees Kazuyuki Amano Dana Angluin Tijl De Bie Laurent Brehelin Christian Choffrut Pedro Delicado Claudio Gentile R´emi Gilleron Sally Goldman
Joshua Goodman Colin de la Higuera Hiroki Ishizaka Jeffrey Jackson Satoshi Kobayashi Jean-Yves Marion Andrei E. Romashchenko Hiroshi Sakamoto Kengo Sato
VIII
Organization
Dale Schuurmans Chema Sempere Shinichi Shimozono Takeshi Shinohara Robert Sloan
Lee Wee Sun Hisao Tamaki Marc Tommasi Takashi Yokomori
Sponsoring Institutions The Japanese Ministry of Education, Culture, Sports, Science and Technology The Suginome Memorial Foundation, Japan
Table of Contents
Invited Papers Abduction and the Dualization Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas Eiter
1
Signal Extraction and Knowledge Discovery Based on Statistical Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Genshiro Kitagawa
3
Association Computation for Information Access . . . . . . . . . . . . . . . . . . . . . . Akihiko Takano
15
Efficient Data Representations That Preserve Information . . . . . . . . . . . . . . Naftali Tishby
16
Can Learning in the Limit Be Done Efficiently? . . . . . . . . . . . . . . . . . . . . . . . Thomas Zeugmann
17
Regular Contributions Inductive Inference Intrinsic Complexity of Uniform Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sandra Zilles
39
On Ordinal VC-Dimension and Some Notions of Complexity . . . . . . . . . . . Eric Martin, Arun Sharma, Frank Stephan
54
Learning of Erasing Primitive Formal Systems from Positive Examples . . . Jin Uemura, Masako Sato
69
Changing the Inference Type – Keeping the Hypothesis Space . . . . . . . . . . Frank Balbach
84
Learning and Information Extraction Robust Inference of Relevant Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan Arpe, R¨ udiger Reischuk
99
Efficient Learning of Ordered and Unordered Tree Patterns with Contractible Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Yusuke Suzuki, Takayoshi Shoudai, Satoshi Matsumoto, Tomoyuki Uchida, Tetsuhiro Miyahara
X
Table of Contents
Learning with Queries On the Learnability of Erasing Pattern Languages in the Query Model . . . 129 Steffen Lange, Sandra Zilles Learning of Finite Unions of Tree Patterns with Repeated Internal Structured Variables from Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Satoshi Matsumoto, Yusuke Suzuki, Takayoshi Shoudai, Tetsuhiro Miyahara, Tomoyuki Uchida
Learning with Non-linear Optimization Kernel Trick Embedded Gaussian Mixture Model . . . . . . . . . . . . . . . . . . . . . 159 Jingdong Wang, Jianguo Lee, Changshui Zhang Efficiently Learning the Metric with Side-Information . . . . . . . . . . . . . . . . . . 175 Tijl De Bie, Michinari Momma, Nello Cristianini Learning Continuous Latent Variable Models with Bregman Divergences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 Shaojun Wang, Dale Schuurmans A Stochastic Gradient Descent Algorithm for Structural Risk Minimisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 Joel Ratsaby
Learning from Random Examples On the Complexity of Training a Single Perceptron with Programmable Synaptic Delays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 ˇıma Jiˇr´ı S´ Learning a Subclass of Regular Patterns in Polynomial Time . . . . . . . . . . . 234 John Case, Sanjay Jain, R¨ udiger Reischuk, Frank Stephan, Thomas Zeugmann Identification with Probability One of Stochastic Deterministic Linear Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Colin de la Higuera, Jose Oncina
Online Prediction Criterion of Calibration for Transductive Confidence Machine with Limited Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Ilia Nouretdinov, Vladimir Vovk Well-Calibrated Predictions from Online Compression Models . . . . . . . . . . 268 Vladimir Vovk
Table of Contents
XI
Transductive Confidence Machine Is Universal . . . . . . . . . . . . . . . . . . . . . . . . 283 Ilia Nouretdinov, Vladimir V’yugin, Alex Gammerman On the Existence and Convergence of Computable Universal Priors . . . . . 298 Marcus Hutter
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
XII
Table of Contents
Abduction and the Dualization Problem Thomas Eiter Institut f¨ur Informationssysteme, Technische Universit¨at Wien, Favoritenstraße 9-11, A-1040 Wien, Austria [email protected]
Abduction is a fundamental mode of reasoning which was extensively studied by C.S. Peirce, who also introduced the term for inference of explanations for observed phenomena. Abduction has taken on increasing importance in Artificial Intelligence (AI) and related disciplines, where it has been recognized as an important principle of commonsense reasoning. It has applications in many areas of AI and Computer Science including diagnosis, database updates, planning, natural language understanding, learning, to number some of them. In a logic-based setting, abduction can be seen as the task to find, given a set of formulas Σ (the background theory), a formula χ (the query), and a set of formulas A (the abducibles or hypotheses), a minimal subset E of A such that Σ plus E is satisfiable and logically entails χ (i.e., an explanation). In many application scenarios T is a propositional Horn theory, χ is a literal or a conjunction of literals, and the abducibles A are certain literals of interest. For use in practice, computing abductive explanations in this setting is an important problem, for which well-known early systems such as Poole’s Theorist or assumption-based Truth Maintenance Systems have been devised in the 1980s. Since then, there has been a growing literature on this subject. Besides computing some arbitrary explanation for a query, the problem of generating several or all explanations has received more attention in the last years. This problem is important since often one would like to select one out of a set of alternative explanations according to a preference or plausibility relation; this relation may be based on subjective intuition which is difficult to formalize and thus can not be implemented by an algorithm. In general, a query may have exponentially many explanations, and thus generating all explanations inevitably requires exponential time in general, even in propositional logic. It is then of interest to know whether generating all explanations is feasible in polynomial total time (aka output-polynomial time), i.e., in time polynomial in the combined size of the input and the output. Furthermore, if exponential resources are prohibitive, it is of interest to know whether a few explanations (e.g., polynomially many) can be generated in polynomial time. In recent and ongoing work, we have investigated the computational complexity of generating all abductive explanations, and compiled a number of interesting results for charting the tractability / intractability frontier of this problem. In this talk, we shall recall some of the results and then focus on abduction from Horn theories represented by their characteristic models. In this setting, the background theory T is represented by a set of so called characteristic models, char(T ), rather than by formulas. The benefit R. Gavald`a et al. (Eds.): ALT 2003, LNAI 2842, pp. 1–2, 2003. c Springer-Verlag Berlin Heidelberg 2003
2
T. Eiter
is that for certain formulas, logical consequence from T efficiently reduces to deciding consequence from char(T ) (which is easy) and thus admits tractable inference. In fact, finding some abductive explanation for a query literal is polynomial in this setting, while this is well-known to be NP-hard under formula-based representation. Computing all abductive explanations for a query literal, which rises in different contexts, is known to be polynomial-time equivalent (in a precise sense) to the problem of dualizing a Boolean function given by a monotone CNF. The latter problem, Monotone Dualization, is with respect to complexity a somewhat mysterious problem which since more than 20 years resists to a precise classification in terms of well-established complexity classes. Currently, no polynomial total-time algorithm solving this problem is known; on other hand, there is also no stringent evidence that such an algorithm is unlikely to exist (like, e.g., coNP-hardness of the associated decision problem whether, given two monotone CNFs ϕ and ψ, they represent dual functions). On the contrary, results in the 1990’s provided some hints that the problem is closer to polynomial totaltime, since as shown by Fredman and Khachyian, the decisional variant can be solved in quasi-polynomial time, i.e., in time O(nlog n ). This was recently refined to solvability in polynomial time with limited nondeterminism, i.e., using a poly-logarithmic number of bit guesses. Apart from this peculiarity, Monotone Dualization has been recognized as an important problem since there are a large number of other problems in Computer Science which are known to be polynomial-time equivalent to this problem. It has a role similar to the one of SAT for the class NP: A polynomial total-time algorithm for Monotone Dualization implies polynomial total-time algorithms for all the polynomial-time equivalent problems. We will consider some possible extensions of the results for abductive explanations which are polynomial-time equivalent to Monotone Dualization. Besides generating all abductive explanations for a literal, there are many other problems in Knowledge Discovery and Data Mining which are polynomial-time equivalent or closely related to Monotone Dualization, including learning with oracles, computation of infrequent and frequent sets, and key generation. We shall give a brief account of such problems, and finally will conclude with some open problems and issues for future research. The results presented are joint work with Kazuhisa Makino, Osaka University.
Association Computation for Information Access Akihiko Takano National Institute of Informatics Hitotsubashi, Chiyoda, Tokyo 101-8430 Japan [email protected]
Abstract. GETA (Generic Engine for Transposable Association) is a software that provides efficient generic computation for association. It enables the quantitative analysis of various proposed methods based on association, such as measuring similarity among documents or words. Scalable implementation of GETA can handle large corpora of twenty million documents, and provides the implementation basis for the effective information access of next generation. DualNAVI is an information retrieval system which is a successful example to show the power and the flexibility of GETA-based computation for association. It provides the users with rich interaction both in document space and in word space. Its dual view interface always returns the retrieved results in two views: a list of titles for document space and “Topic Word Graph” for word space. They are tightly coupled by their cross-reference relation, and inspires the users with further interactions. The two-stage approach in the associative search, which is the key to its efficiency, also facilitates the content-based correlation among databases. In this paper we describe the basic features of GETA and DualNAVI.
The full version of this paper is published in the Proceedings of the 6th International Conference on Discovery Science, Lecture Notes in Artificial Intelligence Vol. 2843.
R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, p. 15, 2003. c Springer-Verlag Berlin Heidelberg 2003
Efficient Data Representations That Preserve Information Naftali Tishby School of Computer Science and Engineering and Center for Neural Computation The Hebrew University, Jerusalem 91904, Israel [email protected]
Abstract. A fundamental issue in computational learning theory, as well as in biological information processing, is the best possible relationship between model representation complexity and its prediction accuracy. Clearly, we expect more complex models that require longer data representation to be more accurate. Can one provide a quantitative, yet general, formulation of this trade-off? In this talk I will discuss this question from Shannon’s Information Theory perspective. I will argue that this trade-off can be traced back to the basic duality between source and channel coding and is also related to the notion of “coding with side information”. I will review some of the theoretical achievability results for such relevant data representations and discuss our algorithms for extracting them. I will then demonstrate the application of these ideas for the analysis of natural language corpora and speculate on possibly-universal aspects of human language that they reveal. Based on joint works with Ran Bacharach, Gal Chechik, Amir Globerson, Amir Navot, and Noam Slonim.
R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, p. 16, 2003. c Springer-Verlag Berlin Heidelberg 2003
Can Learning in the Limit Be Done Efficiently? Thomas Zeugmann Institut f¨ ur Theoretische Informatik, Universit¨ at zu L¨ ubeck, Wallstraße 40, 23560 L¨ ubeck, Germany [email protected]
Abstract. Inductive inference can be considered as one of the fundamental paradigms of algorithmic learning theory. We survey results recently obtained and show their impact to potential applications. Since the main focus is put on the efficiency of learning, we also deal with postulates of naturalness and their impact to the efficiency of limit learners. In particular, we look at the learnability of the class of all pattern languages and ask whether or not one can design a learner within the paradigm of learning in the limit that is nevertheless efficient. For achieving this goal, we deal with iterative learning and its interplay with the hypothesis spaces allowed. This interplay has also a severe impact to postulates of naturalness satisfiable by any learner. Finally, since a limit learner is only supposed to converge in the limit, one never knows at any particular learning stage whether or not the learner did already succeed. The resulting uncertainty may be prohibitive in many applications. We survey results to resolve this problem by outlining a new learning model, called stochastic finite learning. Though pattern languages can neither be finitely inferred from positive data nor PAC-learned, our approach can be extended to a stochastic finite learner that exactly infers all pattern languages from positive data with high confidence.
1
Introduction
Inductive inference can be considered as one of the fundamental paradigms of algorithmic learning theory. In particular, inductive inference of recursive functions and of recursively enumerable languages have been studied intensively within the last four decades (cf., e.g., [3,4,30,16]). The basic model considered within this framework is learning in the limit which can be informally described as follows. The learner receives more and more data about the target and maps these data to hypotheses. Of special interest is the investigation of scenarios in which the sequence of hypotheses stabilizes to an accurate and finite description (e.g. a grammar, a program) of the target. Clearly, then some form of learning must have taken place. Here by data we mean either any infinite sequence of pairs argument-value (in case of learning recursive functions) such that all arguments appear eventually or any infinite sequence of all members of the target language R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, pp. 17–38, 2003. c Springer-Verlag Berlin Heidelberg 2003
18
T. Zeugmann
(in case of language learning from positive data). Alternatively, one can also study language learning from both positive and negative data. Most of the work done in the field has been aimed at the following goals: showing what general collections of function classes or language classes are learnable, characterizing those collections of classes that can be learned, studying the impact of several postulates on the behavior of learners to their learning power, and dealing with the influence of various parameters to the efficiency of learning. However, defining an appropriate measure for the complexity of learning in the limit has turned out to be quite difficult (cf. Pitt [31]). Moreover, whenever learning in the limit is done, in general one never knows whether or not the learner has already converged. This is caused by the fact that it is either undecidable at all whether or not convergence already occurred. But even if it is decidable, it is practically infeasible to do so. Thus, there is always an uncertainty which may not be tolerable in many applications of learning. Therefore, different learning models have been proposed. In particular, Valiant’s [46] model of probably approximately correct (abbr. PAC) learning has been very influential. As a matter of fact, this model puts strong emphasis on the efficiency of learning and avoids the problem of convergence at all. In the PAC model, the learner receives a finite labeled sample of the target concepts and outputs, with high probability, a hypothesis that is approximately correct. The sample is drawn with respect to an unknown probability distribution and the error of as well as the confidence in the hypothesis are measured with respect to this distribution, too. Thus, if a class is PAC learnable, one obtains nice performance guarantees. Unfortunately, many interesting concept classes are not PAC learnable. Consequently, one has to look for other models of learning or one is back to learning in the limit. So, let us assume that learning in the limit is our method of choice. What we would like to present in this survey is a rather general way to transform learning in the limit into stochastic finite learning. It should also be noted that our ideas may be beneficial even in case that the considered concept class is PAC learnable. Furthermore, we aim to outline how a thorough study of limit learnability of concept classes may nicely contribute to support our new approach. We exemplify the research undertaken by mainly looking at the class of all pattern languages introduced by Angluin [1]. As Salomaa [37] has put it “Patterns are everywhere” and thus we believe that our research is worth the effort undertaken. There are several problems that have to be addressed when dealing with the learnability of pattern languages. First, the nice thing about patterns is that they are very intuitive. Therefore, it seems desirable to design learners outputting pattern as their hypotheses. Unfortunately, membership is known to be N P complete for the pattern languages (cf. [1]). Thus, many of the usual approaches used in machine learning will directly lead to infeasible learning algorithms. As a consequence, we shall ask what kind of appropriate hypothesis spaces can be used at all to learn the pattern languages, and what are the appropriate learning strategies.
Can Learning in the Limit Be Done Efficiently?
19
In particular, we shall deal with the problem of redundancy in the hypothesis space chosen, with consistency, conservativeness, and iterative learning. Here consistency means that the intermediate hypotheses output by the learner do correctly reflect the data seen so far. Conservativeness addresses the problem to avoid overgeneralization, i.e., preventing the learner from guessing a proper superset of the target language. These requirements are naturally arising desiderata, but this does not mean that they can be fulfilled. With iterative learning, the learning machine, in making a conjecture, has access to its previous conjecture and the latest data item coming in. Iterative learning is also a natural requirement whenever learning in the limit is concerned, since no practical learner can process at every learning stage all examples provided so far, it may even not be able to store them. Finally, we address the question how efficient the overall learning process can be performed, and how we can get rid of the uncertainty of not knowing whether or not the learner has already converged.
2
Preliminaries
Unspecified notation follows Rogers [35]. By N = {0, 1, 2, . . .} we denote the set of all natural numbers. We set N+ = N \ {0} . The cardinality of a set S is denoted by |S| . Let ∅, ∈, ⊂, ⊆ , ⊃ , and ⊇ , denote the empty set, element of, proper subset, subset, proper superset, and superset, respectively. Let ϕ0 , ϕ1 , ϕ2 , . . . denote any fixed acceptable programming system for all (and only) the partial recursive functions over N (cf. Rogers [35]). Then ϕk is the partial recursive function computed by program k . In the following subsection we define the main learning models considered within this paper. 2.1
Learning in the Limit
Gold’s [12] model of learning in the limit allows one to formalize a rather general class of learning problems, i.e., learning from examples. For defining this model we assume any recursively enumerable set X and refer to it as the learning domain. By ℘(X ) we denote the power set of X . Let C ⊆ ℘(X ) , and let c ∈ C be non-empty; then we refer to C and c as a concept class and a concept, respectively. Let c be a concept, and let t = (xj )j∈N be any infinite sequence of elements xj ∈ c such that range(t) =df {xj j ∈ N} = c . Then t is said to be a positive presentation or, synonymously, a text for c . By text(c) we denote the set of all positive presentations for c . Moreover, let t be a positive presentation, and let y ∈ N . Then, we set ty = x0 , . . . , xy , i.e., ty is the initial segment of t + of length y + 1 , and t+ y =df {xj j ≤ y} . We refer to ty as the content of ty . Furthermore, let σ = x0 , . . . , xn−1 be any finite sequence. Then we use |σ| to denote the length n of σ , and let content(σ) and σ + , respectively, denote
20
T. Zeugmann
the content of σ . Additionally, let t be a text and let τ be a finite sequence; then we use σ t and σ τ to denote the sequence obtained by concatenating σ onto the front of t and τ , respectively. Alternatively, one can also consider complete presentations or, synonymously, informants. Let c be a concept; then any sequence i = (xj , bj )j∈N of labeled examples, where bj ∈ {+, −} such that {xj j ∈ N} = X and i+ = {xj (xj , bj ) = (xj , +), j ∈ N} = c and i− = {xj (xj , bj ) = (xj , −), j ∈ N} = X \ c is called an informant for c . For the sake of presentation, the following definitions are only given for the text case, the generalization to the informant case should be obvious. We sometimes use the term data sequence to refer to both text and informant, respectively. An inductive inference machine (abbr. IIM) is an algorithm that takes as input larger and larger initial segments of a text and outputs, after each input, a hypothesis from a prespecified hypothesis space H = (hj )j∈N . The indices j are regarded as suitable finite encodings of the concepts described by the hypotheses. A hypothesis is said to describe a concept c iff c = h . Definition 1. Let C be any concept class, and let H = (hj )j∈N be a hypothesis space for it. C is called learnable in the limit from text iff there is an IIM M such that for every c ∈ X and every text t for c , (1) for all n ∈ N+ , M (tn ) is defined, (2) there is a j such that c = hj and for all but finitely many n ∈ N+ , M (tn ) = j . By LimTxt we denote the collection of all concepts classes C that are learnable in the limit from text1 . Note that instead of LimTxt sometimes TxtEx is used. Note that Definition 1 does not contain any requirement concerning efficiency. Before we are going to deal with efficiency, we want to point to another crucial parameter of our learning model, i.e., the hypothesis space H . Since our goal is algorithmic learning, we can consider the special case that X = N and let C be any subset of the collection of all recursively enumerable sets over N . Let Wi = domainϕi . In this case, (Wj )j∈N is the most general hypothesis space. Within this setting many learning problems can be described. Moreover, this setting has been used to study the general capabilities of different learning models which can be obtained by suitable modifications of Definition 1. There are numerous papers performing studies along this line of research (cf., e.g., [16,30] and the references therein). On the one hand, the results obtained considerably broaden our general understanding of algorithmic learning. On the other hand, one has also to ask what kind of consequences one may derive from these results for practical learning problems. This is a non-trivial question, since the setting of learning recursively enumerable languages is very rich. Thus, it is conceivable 1
If learning from informant is considered we use LimInf to denote the collection of all concepts classes C that are learnable in the limit from informant.
Can Learning in the Limit Be Done Efficiently?
21
that several of the phenomena observed hold in this setting due to the fact too many sets are recursively enumerable and there are no counterparts within the world of efficient computability. As a first step to address this question we mainly consider the scenario that indexable concept classes with uniformly decidable membership have to be learned (cf. Angluin [2]). A class of non-empty concepts C is said to be an indexable class with uniformly decidable membership provided there are an effective enumeration c0 , c1 , c2 , ... of all and only the concepts in C and a recursive function f such that for all j ∈ N and all elements x ∈ X we have 1, if x ∈ cj , f (j, x) = 0, otherwise. In the following we refer to indexable classes with uniformly decidable membership as to indexable classes for short. Furthermore, we call any enumeration (cj )j∈N of C with uniformly decidable membership problem an indexed family. Since the paper of Angluin [2] learning of indexable concept classes has attracted much attention (cf., e.g., Zeugmann and Lange [51]). Let us shortly provide some-well known indexable classes. Let Σ be any finite alphabet of symbols, and let X be the free monoid over Σ , i.e., X = Σ ∗ . We set Σ + = Σ ∗ \ {λ} , where λ denotes the empty string. As usual, we refer to subsets L ⊆ X as languages. Then the set of all regular languages, context-free languages, and context-sensitive languages are indexable classes. n Next, let Xn = {0, 1} be the set of all n -bit Boolean vectors. We consider X = n≥1 Xn as learning domain. Then, the set of all concepts expressible as a monomial, a k -CNF, a k -DNF, and a k -decision list form indexable classes.
When learning indexable classes C , it is generally assumed that the hypothesis space H has to be an indexed family, too. We distinguish class preserving learning and class comprising learning defined by C = range(H) and C ⊆ range(H) , respectively. When dealing with class preserving learning, one has the freedom to choose as hypothesis space a possibly different enumeration of the target class C . In contrast, when class comprising learning is concerned, the hypothesis space may enumerate, additionally, languages not belonging to C . Note that, in general, one has to allow class comprising hypothesis spaces to obtain the maximum possible learning power (cf. Lange and Zeugmann [20,22]). Finally, we call an hypothesis space redundant if it is larger than necessary, i.e., there is at least one hypothesis in H not describing any concept from the target class or one concept possesses at least two different descriptions in H . Thus, non-redundant hypothesis spaces are as small as possible. Formally, a hypothesis space H = (hj )j∈N is non-redundant for some target concept class C iff range(H) = C and hi = hj for all i, j ∈ N with i = j . Otherwise, H is a redundant hypothesis space for C . Next, let us come back to the issue of efficiency. Looking at Definition 1 we see that an IIM M has always access to the whole history of the learning process, i.e., in order to compute its actual guess M is fed all examples seen so
22
T. Zeugmann
far. In contrast to that, next we define iterative IIMs. An iterative IIM is only allowed to use its last guess and the next element in the positive presentation of the target concept for computing its actual guess. Conceptionally, an iterative IIM M defines a sequence (Mn )n∈N of machines each of which takes as its input the output of its predecessor. Definition 2 (Wiehagen [47]). Let C be a concept class, let c be a concept, let H = (hj )j∈N be a hypothesis space, and let a ∈ N ∪ {∗} . An IIM M ItLimTxt H -infers c iff for every t = (xj )j∈N ∈ text(c) the following conditions are satisfied: (1) for all n ∈ N , Mn (T ) is defined, where M0 (T ) =df M (x0 ) and for all n ≥ 0 : Mn+1 (T ) =df M (Mn (T ), xn+1 ) , (2) the sequence (Mn (T ))n∈N converges to a number j such that c = hj . Finally, M ItLimTxt H -infers C iff, for each c ∈ C , M ItLimTxt H -infers c . In the latter definition Mn (t) denotes the (n+1) th hypothesis output by M when successively fed the text t . Thus, it is justified to make the following convention. Let σ = x0 , . . . , xn be any finite sequence of elements over the relevant learning domain. Moreover, let C be any concept class over X , and let M be any IIM that iteratively learns C . Then we denote by My (σ) the (y + 1) th hypothesis output by M when successively fed σ provided y ≤ n , and there exists a concept c ∈ C with σ + ⊆ c . Furthermore, we let M∗ (σ) denote M|σ|−1 (σ) . Moreover, whenever learning a concept class from text, a major problem one has to deal with is avoiding or detecting overgeneralization. An overgeneralization occurs if the learner is guessing a superconcept of the target concept. Clearly, such an overgeneralized guess cannot be detected by using the incoming positive data only. Therefore, one may be tempted to disallow overgeneralized guesses at all. Learners behaving thus are called conservative. Intuitively speaking a conservative IIM maintains its actual hypothesis at least as long as it has not seen data contradicting it. More formally, an IIM M is said to be conservative iff for all concepts c in the target class C and all texts t for c the condition M (ty ) = M (ty+z ) then t+ y+z ⊆ hM (ty ) is fulfilled. Another property of learners quite often found in the literature is consistency. Informally, a learner is called consistent if all its intermediate hypotheses do correctly reflect the data seen so far. More formally, an IIM M is said to be consistent iff t+ x ⊆ hM (tx ) for all x ∈ N and every text t for every concept c in the target class C . Whenever one talks about the efficiency of learning besides the storage needed by the learner one has also to consider the time complexity of the learner. When talking about the time complexity of learning, it does not suffice to consider the time needed to compute the actual guess. What really counts in applications is the overall time needed until successful learning. Therefore, following Daley and Smith [10] we define the total learning time as follows.
Can Learning in the Limit Be Done Efficiently?
23
Let C be any concept class, and let M be any IIM that learns C in the limit. Then, for every c ∈ C and every text t for c , let Conv (M, t) =df the least number m ∈ N+ such that for all n ≥ m, M (tn ) = M (tm ) denote the stage of convergence of M on t (cf. [12]). Note that Conv (M, t) = ∞ if M does not learn the target concept from its text t . Moreover, by TM (tn ) we denote the time to compute M (tn ) . We measure this time as a function of the length of the input and call it the update time. Finally, the total learning time taken by the IIM M on successive input t is defined as Conv (M,t)
T T (M, t) =df
TM (tn ).
n=1
Clearly, if M does not learn the target concept from text t then the total learning time is infinite. Two more remarks are in order here. First, it has been argued elsewhere that within the learning in the limit paradigm a learning algorithm is invoked only when the current hypothesis has some problem with the latest observed data. However, such a viewpoint implicitly assumes that membership in the target concept is decidable in time polynomial in the length of the actual input. This may be not case. Thus, directly testing consistency would immediately lead to a non-polynomial update time provided membership is not known to be in P . Second, Pitt [31] addresses the question with respect to what parameter one should measure the total learning time. In the definition given above this parameter is the length of all examples seen so far. Clearly, now one could try to play with this parameter by waiting for a large enough input before declaring success. However, when dealing with the learnability of non-trivial concept classes, in the worst-case the total learning time will be anyhow unbounded. Thus, it does not make much sense to deal with the worst-case. Instead, we shall study the expected total learning time. In such a setting one cannot simply wait for long enough inputs. Therefore, using the definition of total learning time given above seems to be reasonable. Next, we define important concept classes which we are going to consider throughout this survey. 2.2
The Pattern Languages
Following Angluin [1] we define patterns and pattern languages as follows. Let A = {0, 1, . . .} be any non-empty finite alphabet containing at least two elements. By A∗ we denote the free monoid over A . The set of all finite non-null strings of symbols from A is denoted by A+ , i.e., A+ = A∗ \ {λ} , where
24
T. Zeugmann
λ denotes the empty string. Let X = {xi i ∈ N} be an infinite set of variables such that A ∩ X = ∅ . Patterns are non-empty strings over A ∪ X , e.g., 01, 0x0 111, 1x0 x0 0x1 x2 x0 are patterns. The length of a string s ∈ A∗ and of a pattern π is denoted by |s| and |π| , respectively. A pattern π is in canonical form provided that if k is the number of different variables in π then the variables occurring in π are precisely x0 , . . . , xk−1 . Moreover, for every j with 0 ≤ j < k − 1 , the leftmost occurrence of xj in π is left to the leftmost occurrence of xj+1 . The examples given above are patterns in canonical form. In the sequel we assume, without loss of generality, that all patterns are in canonical form. By Pat we denote the set of all patterns in canonical form. If k is the number of different variables in π then we refer to π as to a k -variable pattern. By Pat k we denote the set of all k -variable patterns. Furthermore, let π ∈ Pat k , and let u0 , . . . , uk−1 ∈ A+ ; then we denote by π[x0 /u0 , . . . , xk−1 /uk−1 ] the string w ∈ A+ obtained by substituting uj for each occurrence of xj , j = 0, . . . , k − 1 , in the pattern π . For example, let π = 0x0 1x1 x0 . Then π[x0 /10, x1 /01] = 01010110 . The tuple (u0 , . . . , uk−1 ) is called a substitution. Furthermore, if |u0 | = · · · = |uk−1 | = 1 , then we refer to (u0 , . . . , uk−1 ) as to a shortest substitution. Let π ∈ Pat k ; we define the language generated by pattern π by L(π) = {π[x0 /u0 , . . . , xk−1 /uk−1 ] u0 , . . . , uk−1 ∈ A+ } . By PAT k we denote the set of all k -variable pattern languages. Finally, PAT = k∈N PAT k denotes the set of all pattern languages over A . Furthermore, we let Q range over finite sets of patterns and define L(Q) = π∈Q L(π) , i.e., the union of all pattern languages generated by patterns from Q . Moreover, we use Pat(k) and PAT (k) to denote the family of all unions of at most k canonical patterns and the family of all unions of at most k pattern languages, respectively. That is, Pat(k) = {Q Q ⊆ Pat, |Q| ≤ k} and PAT (k) = {L (∃Q ∈ Pat(k))[L = L(Q)]} . Finally, let L ⊆ A+ be a language, and let k ∈ N+ ; we define Club(L, k) = {Q |Q| ≤ k, L ⊆ L(Q), (∀Q )[Q ⊂ Q ⇒ L ⊆ L(Q )]} . Club stands for consistent least upper bounds. The pattern languages have been intensively investigated (cf., e.g., Salomaa [37,38], and Shinohara and Arikawa [43] for an overview). Nix [29] as well as Shinohara and Arikawa [43] outlined interesting applications of pattern inference algorithms. For example, pattern language learning algorithms have been successfully applied for solving problems in molecular biology (cf., e.g., Shimozono et al. [39], Shinohara and Arikawa [43]). As it turned out, pattern languages and finite unions of pattern languages are subclasses of Smullyan’s [45] elementary formal systems (Abbr. EFS). Arikawa et al. [5] have shown that EFS can also be treated as a logic programming language over strings. Recently, the techniques for learning finite unions of pattern languages have been extended to show the learnability of various subclasses of EFS (cf. Shinohara [42]). The investigations of the learnability of subclasses of EFSs are interesting because they yield corresponding results about the learnability of subclasses of logic programs. Hence, these results are also of relevance for Inductive Logic Programming (ILP) [28,23,8,24]. Miyano et al. [26] intensively studied the polynomial-time learnability of EFSs.
Can Learning in the Limit Be Done Efficiently?
25
Therefore, we may consider the learnability of pattern languages and of unions thereof as a nice test bed for seeing what kind of results one may obtain by considering the corresponding learning problems within the setting of learning in the limit.
3
Results
Within this section we ask whether or not the pattern languages and finite unions thereof can be learned efficiently. The principal learnability of the pattern languages from text with respect to the hypothesis space Pat has been established by Angluin [1]. However, her algorithm is based on computing descriptive patterns for the data seen so far. Here a pattern π is said to be descriptive (for the set S of strings contained in the input provided so far) if π can generate all strings contained in S and no other pattern with this property generates a proper subset of the language generated by π . Since no efficient algorithm is known for computing descriptive patterns, and finding a descriptive pattern of maximum length is N P -hard, its update time is practically intractable. There are also serious difficulties when trying to learn the pattern languages within the PAC model introduced by Valiant [46]. In the original model, the sample complexity depends exclusively on the VC dimension of the target concept class and the error and confidence parameters ε and δ , respectively. Recently, Mitchell et al. [25] have shown that even the class of all one-variable pattern languages has infinite VC dimension. Consequently, even this special subclass of PAT is not uniformly PAC learnable. Moreover, Schapire [40] has shown that pattern languages are not PAC learnable in the generalized model provided P/poly = N P/poly with respect to every hypothesis space for PAT that is uniformly polynomially evaluable. Though this result highlights the difficulty of PAC learning PAT it has no clear application to the setting considered in this paper, since we aim to learn PAT with respect to the hypothesis space Pat . Since the membership problem for this hypothesis space is N P -complete, it is not polynomially evaluable (cf. [1]). In contrast, Kearns and Pitt [18] have established a PAC learning algorithm for the class of all k -variable pattern languages. Positive examples are generated with respect to arbitrary product distributions while negative examples are allowed to be generated with respect to any distribution. In their algorithm the length of substitution strings is required to be polynomially related to the length of the target pattern. Finally, they use as hypothesis space all unions of polynomially many patterns that have k or fewer variables2 . The overall learning time of their PAC learning algorithm is polynomial in the length of the target 2
More precisely, the number of allowed unions is at most poly(|π|, s, 1/ε, 1/δ, |A|) , where π is the target pattern, s the bound on the length on substitution strings, ε and δ are the usual error and confidence parameter, respectively, and A is the alphabet of constants over which the patterns are defined.
26
T. Zeugmann
pattern, the bound for the maximum length of substitution strings, 1/ε , 1/δ , and |A| . The constant in the running time achieved depends doubly exponential on k , and thus, their algorithm becomes rapidly impractical when k increases. Finally, Lange and Wiehagen [19] have proposed an inconsistent but iterative and conservative algorithm that learns PAT with respect to Pat . We shall study this algorithm below in much more detail. But before doing it, we aim to figure out under which circumstances iterative learning of PAT is possible at all. A first answer is given by the following theorems from Case et al. [9]. Note that Pat is a non-redundant hypothesis space for PAT . Theorem 1 (Case et al. [9]). Let C be any concept class, and let H = (hj )j∈N be any non-redundant hypothesis space for C . Then, every IIM M that ItLimTxt H -infers C is conservative. Proof. Suppose the converse, i.e., there are a concept c ∈ C , a text t = (xj )j∈N ∈ text(c) , and a y ∈ N such that, for j = M∗ (ty ) and k = M∗ (ty+1 ) = M (j, xy+1 ) , both j = k and t+ y+1 ⊆ hj are satisfied. The latter implies xy+1 ∈ hj , and thus we may consider the following text t˜ ∈ text(hj ) . Let tˆ = (ˆ xj )j∈N be any text for hj and let t˜ = x ˆ0 , xy+1 , x ˆ1 , xy+1 , x ˆ2 , . . . Since M has to learn hj from t˜ there must be a z ∈ N such that M∗ (t˜z+r ) = j for all r ≥ 0 . But M∗ (t˜2z+1 ) = M (j, xy+1 ) = k , a contradiction. Next, we point to another peculiarity of PAT , i.e., it meets the superset condition defined as follows. Let C be any indexable class. C meets the superset condition if, for all c, c ∈ C , there is some cˆ ∈ C being a superset of both c and c . Theorem 2. (Case et al. [9]). Let C be any indexable class meeting the superset condition, and let H = (hj )j∈N be any non-redundant hypothesis space for C . Then, every consistent IIM M that ItLimTxt H -infers C may be used to decide the inclusion problem for H . Proof. Let X be the underlying learning domain, and let (wj )j∈N be an effective enumeration of all elements in X . Then, for every i ∈ N , ti = (xij )j∈N is the following computable text for hi . Let z be the least index such that wz ∈ hi . Recall that, by definition, hi = ∅ , since H is an indexed family, and thus wz must exist. Then, for all j ∈ N , we set xij = wj , if wj ∈ hi , and xij = wz , otherwise. We claim that the following algorithm Inc decides, for all i, k ∈ N , whether or not hi ⊆ hk . Algorithm Inc: “On input i, k ∈ N do the following: ⊆ hk . Determine the least y ∈ N with i = M∗ (tiy ) . Test whether or not ti,+ y In case it is, output ‘Yes,’ and stop. Otherwise, output ‘No,’ and stop.” Clearly, since H is an indexed family and ti is a computable text, Inc is an algorithm. Moreover, M learns hi on every text for it, and H is a nonredundant hypothesis space. Hence, M has to converge on text ti to i , and therefore Inc has to terminate.
Can Learning in the Limit Be Done Efficiently?
27
It remains to verify the correctness of Inc . Let i, k ∈ N . Clearly, if Inc outputs ‘No,’ a string s ∈ hi \hk has been found, and hi ⊆ hk follows. Next, consider the case that Inc outputs ‘Yes.’ Suppose to the contrary that hi ⊆ hk . Then, there is some s ∈ hi \ hk . Now, consider M when fed the text t = tiy tk . Since ti,+ ⊆ hk , t is a text for hk . Since M learns hk , there is y some r ∈ N such that k = M∗ (tiy tkr ) . By assumption, there are some cˆ ∈ C with hi ∪ hk ⊆ cˆ , and some text tˆ for cˆ having the initial segment tiy s tkr . By Theorem 1, M is conservative. Since s ∈ hi and i = M∗ (tˆy ) , we obtain M∗ (tˆy+1 ) = M (i, s) = i . Consequently, M∗ (tiy s tkr ) = M∗ (tiy tkr ) . Finally, i k since s ∈ tˆ+ / hk , M fails to consistently learn y+r+2 , k = M∗ (ty tr ) , and s ∈ cˆ from text tˆ , a contradiction. This proves the theorem. Taking into account that the inclusion problem for Pat is undecidable (cf. Jiang et al. [17] and that PAT meets the superset condition, since L(x0 ) = A+ , by Theorem 2, we immediately arrive at the following corollary. Corollary 3 (Case et al. [9]). If an IIM M ItLimTxt Pat -learns PAT then M is inconsistent. As a matter of fact, the latter corollary generalizes to all non-redundant hypothesis spaces for PAT . All the ingredients to prove this can be found in Zeugmann et al. [52]. Consequently, if one wishes to learn the pattern languages or unions of pattern languages iteratively, then either redundant hypothesis spaces or inconsistent learners cannot be avoided. As for unions, the first result goes back to Shinohara [41] who proved the class of all unions of at most two pattern languages to be in LimTxt Pat(2) . Wright [49] extended this result to PAT (k) ∈ LimTxt Pat(k) for all k ≥ 1 . Moreover, Theorem 4.2 in Shinohara and Arimura’s [44] together with a lemma from Blum and Blum [6] shows that k∈N PAT (k) is not LimTxt H -inferable for every hypothesis space H . The iterative learnability of PAT (k) has been established by Case et al. [9]. Our learner is also consistent. Thus, the hypothesis space used had to be designed to be redundant. We only sketch the proof here. Theorem 4. (1) Club(L, k) is finite for all L ⊆ A+ and all k ∈ N+ , (2) If L ∈ PAT (k) , then Club(L, k) is non-empty and contains a set Q , such that L(Q) = L . Proof. Part (2) is obvious. Part (1) is easy for finite L . For infinite L , it follows from the lemma below. Lemma 1. Let k ∈ N+ , let L ⊆ A+ be any language, and suppose t = (sj )j∈N ∈ text(L). Then, + (1) Club(t+ 0 , k) can be obtained effectively from s0 , and Club(tn+1 , k) is effec+ tively obtainable from Club(tn , k) and sn+1 (* note the iterative nature *).
28
T. Zeugmann
+ (2) The sequence Club(t+ 0 , k), Club(t1 , k), . . . converges to Club(L, k).
Putting it all together, one directly gets the following theorem. Theorem 5. For all k ≥ 1 , PAT (k) ∈ ItLimTxt . Proof. Let can(·) , be some computable bijection from finite classes of finite sets of patterns onto N . Let pad be a 1–1 padding function such that, for all x, y ∈ N , Wpadx,y = Wx . For a finite class S of sets of patterns, let g(S) denote a grammar obtained, effectively from S , for Q∈S L(Q) . Let L ∈ PAT (k) , and let t = (sj )j∈N ∈ text(L) . The desired IIM M is de+ fined as follows. We set M0 (t) = M (s0 ) = padg(Club(t+ 0 , k)), can(Club(t0 , k)) , and for all n > 0 , let Mn+1 (t) = M (Mn (t), sn+1 ) + = padg(Club(t+ n+1 , k)), can(Club(tn+1 , k)) Using Lemma 1 it is easy to verify that Mn+1 (t) = M (Mn (t), sn+1 ) can be obtained effectively from Mn (t) and sn+1 . Therefore, M ItLimTxt -identifies PAT (k) . So far, the general theory provided substantial insight into the iterative learnability of the pattern languages. But still, we do not know anything about the number of examples needed until successful learning and the total amount of time to process them. Therefore, we address this problem in the following subsection. 3.1
Stochastic Finite Learning
As we have already mentioned, it does not make much sense to study the worstcase behavior of learning algorithms with respect to their total learning time. The reason for this phenomenon should be clear, since an arbitrary text may provide the information needed for learning very late. Therefore, in the following we always assume a class D of admissible probability distributions over the relevant learning domain. Ideally, this class should be parameterized. Then, the data fed to learner are generated randomly with respect to one of the probability distributions from the class D of underlying probability distributions. Furthermore, we introduce a random variable CONV for the stage of convergence. Note that CONV can be also interpreted as the total number of examples read by the IIM M until convergence. The first major step to be performed consists now in determining the expectation E[CONV ] . Clearly, E[CONV ] should be finite for all concepts c ∈ C and all distributions D ∈ D . Second, one has to deal with tail bounds for E[CONV ] . The easiest way to perform this step is to use Markov’s inequality, i.e., we always know that Pr(CONV ≥ t · E[CONV ]) ≤
1 for all t ∈ N+ . t
However, quite often one can obtain much better tail bounds. If the underlying learner is known to be conservative and rearrangement-independent we always
Can Learning in the Limit Be Done Efficiently?
29
get exponentially shrinking tail bounds. A learner is said to be rearrangementindependent if its output depends exclusively on the range and length of its input (cf. [21] and the references therein). These tail bounds are established by the following theorem. Theorem 6. (Rossmanith and Zeugmann [36].) Let CONV be the sample complexity of a conservative and rearrangement-independent learning algorithm. Then Pr(CONV ) ≥ 2t · E[CONV ]) ≤ 2 −t for all t ∈ N . Theorem 6 puts the importance of rearrangement-independent and conservative learners into the right perspective. As long as the learnability of indexed families is concerned, these results have a wide range of potential applications, since every conservative learner can be transformed into a learner that is both conservative and rearrangement-independent provided the hypothesis space is appropriately chosen (cf. Lange and Zeugmann [21]). Furthermore, since the distribution of CONV decreases geometrically for all conservative and rearrangement-independent learning algorithms, all higher moments of CONV exist in this case, too. Thus, instead of applying Theorem 6 directly, one can hope for further improvements by applying even sharper tail bounds using for example Chebyshev’s inequality. Additionally, the learner takes a confidence parameter δ as input. But in contrast to learning in the limit, the learner itself decides how many examples it wants to read. Then it computes a hypothesis, outputs it and stops. The hypothesis output is correct for the target with probability at least 1 − δ . The explanation given so far explains how it works, but not why it does. Intuitively, the stochastic finite learner simulates the limit learner until an upper bound for twice the expected total number of examples needed until convergence has been met. Assuming this to be true, by Markov’s inequality the limit learner has now converged with probability 1/2 . All what is left, is to decrease the probability of failure. This is done by using the tail bounds for CONV . Applying Theorem 6, one easily sees that increasing the sample complexity by a factor of O(log 1δ ) results in a probability of 1 − δ for having reached the stage of convergence. If Theorem 6 is not applicable, one can still use Markov’s inequality but then the sample complexity needed will increase by a factor of 1/δ . It remains to explain how the stochastic finite learner can calculate the upper bound for E[CONV ] . This is precisely the point where we need the parameterization of the class D of underlying probability distributions. Since in general, it is not known which distribution from D has been chosen, one has to assume a bit of prior knowledge or domain knowledge provided by suitable upper and/or lower bounds for the parameters involved. A more serious difficulty is to incorporate the unknown target concept into this estimate. This step depends on the concrete learning problem on hand, and requires some extra effort. We shall exemplify it below. Now we are ready to formally define stochastic finite learning.
30
T. Zeugmann
Definition 3 ([33,34,36]). Let D be a set of probability distributions on the learning domain, C a concept class, H a hypothesis space for C , and δ ∈ (0, 1) . (C, D) is said to be stochastically finitely learnable with δ -confidence with respect to H iff there is an IIM M that for every c ∈ C and every D ∈ D performs as follows. Given any random data sequence θ for c generated according to D , M stops after having seen a finite number of examples and outputs a single hypothesis h ∈ H . With probability at least 1−δ (with respect to distribution D ) h has to be correct, that is c = h . If stochastic finite learning can be achieved with δ -confidence for every δ > 0 then we say that (C, D) can be learned stochastically finite with high confidence. Note that there are subtle differences between our model and PAC learning. By its definition, stochastic finite learning is not completely distribution independent. A bit of additional knowledge concerning the underlying probability distributions is required. Thus, from that perspective, stochastic finite learning is weaker than the PAC-model. On the other hand, we do not measure the quality of the hypothesis with respect to the underlying probability distribution. Instead, we require the hypothesis computed to be exactly correct with high probability. Note that exact identification with high confidence has been considered within the PAC paradigm, too (cf., e.g., Goldman et al. [13]). Conversely, we also can easily relax the requirement to learn probably exactly correct but whenever possible we shall not do it. Furthermore, in the uniform PAC model as introduced in Valiant [46] the sample complexity depends exclusively on the VC dimension of the target concept class and the error and confidence parameters ε and δ , respectively. This model has been generalized by allowing the sample size to depend on the concept complexity, too (cf., e.g., Blumer et al. [7] and Haussler et al. [15]). Provided no upper bound for the concept complexity of the target concept is given, such PAC learners decide themselves how many examples they wish to read (cf. [15]). This feature is also adopted to our setting of stochastic finite learning. However, all variants of PAC learning we are aware of require that all hypotheses from the relevant hypothesis space are uniformly polynomially evaluable. Though this requirement may be necessary in some cases to achieve (efficient) stochastic finite learning, it is not necessary in general as we shall see below. Next, let us exemplify our model by looking at the concept class of all pattern languages. The results presented below have been obtained by Zeugmann [50] and Rossmanith and Zeugmann [36]. Our stochastic finite learner uses Lange and Wiehagen’s [19] pattern language learner as a main ingredient. We consider here learning from positive data only. Recall that every string of a particular pattern language is generated by at least one substitution. Therefore, it is convenient to consider probability distributions over the set of all possible substitutions. That is, if π ∈ Pat k , then it suffices to consider any probability distribution D over A+ × · · · × A+ . For k−times
(u0 , . . . , uk−1 ) ∈ A+ × · · · × A+ we denote by D(u0 , . . . , uk−1 ) the probability
Can Learning in the Limit Be Done Efficiently?
31
that variable x0 is substituted by u0 , variable x1 is substituted by u1 , . . . , and variable xk−1 is substituted by uk−1 . In particular, we mainly consider a special class of distributions, i.e., product distributions. Let k ∈ N+ , then the class of all product distributions for Pat k is defined as follows. For each variable xj , 0 ≤ j ≤ k − 1 , we assume an arbitrary probability distribution Dj over A+ on substitution strings. Then we call D = D0 × · · · × Dk−1 product distribution over A+ × · · · × A+ , i.e., k−1 D(u0 , . . . , uk−1 ) = j=0 Dj (uj ) . Moreover, we call a product distribution regular if D0 = · · · = Dk−1 . Throughout this paper, we restrict ourselves to deal with regular distributions. We therefore use d to denote k−1 the distribution over A+ on substitution strings, i.e, D(u0 , . . . , uk−1 ) = j=0 d(uj ) . We call a regular distribution admissible if d(a) > 0 for at least two different elements a ∈ A . As a special case of an admissible distribution we consider the uniform distribution over A+ , i.e., d(u) = 1/(2 · |A|) for all strings u ∈ A+ with |u| = . We will express all estimates with the help of the following parameters: E[Λ] , α and β , where Λ is a random variable for the length of the examples drawn. α and β are defined below. To get concrete bounds for a concrete implementation one has to obtain c from the algorithm and has to compute E[Λ] , α , and β from the admissible probability distribution D . Let u0 , . . . , uk−1 be independent random variables with distribution d for substitution strings. Whenever the index i of ui does not matter, we simply write u or u . The two parameters α and β are now defined via d . First, α is simply the probability that u has length 1, i.e., α = Pr(|u| = 1) =
d(a).
a∈A
Second, β is the conditional probability that two random strings that get substituted into π are identical under the condition that both have length 1 , i.e.,
2
β = Pr u = u |u| = |u | = 1 = d(a)2 d(a) . a∈calA
a∈A
Note that we have omitted the assumption of a text to exhaust the target language. Instead, we only demand the data sequence fed to the learner to contain “enough” information to recognize the target pattern. The meaning of “enough” is mainly expressed by the parameter α . The model of computation as well as the representation of patterns we assume is the same as in Angluin [1]. In particular, we assume a random access machine that performs a reasonable menu of operations each in unit time on registers of length O(log n) bits, where n is the input length. Lange and Wiehagen’s [19] algorithm (abbr. LWA) works as follows. Let hn be the hypothesis computed after reading s1 , . . . , sn , i.e., hn = M (s1 , . . . , sn ) .
32
T. Zeugmann
Then h1 = s1 and for all n > 1 : if |hn−1 | < |sn | hn−1 , if |hn−1 | > |sn | hn = sn , hn−1 ∪ sn , if |hn−1 | = |sn | The algorithm computes the new hypothesis only from the latest example and the old hypothesis. If the latest example is longer than the old hypothesis, the example is ignored, i.e., the hypothesis does not change. If the latest example is shorter than the old hypothesis, the old hypothesis is ignored and the new example becomes the new hypothesis. If, however, |hn−1 | = |sn | the new hypothesis is the union of hn−1 and sn . The union = π ∪ s of a canonical pattern π and a string s of the same length is defined as π(i), if π(i) = s(i) xj , if π(i) = s(i) & ∃k < i : [(k) = xj , s(k) = s(i), (i) = π(k) = π(i)] xm , otherwise, where m = #var((1) . . . (i − 1)) where (0) = λ for notational convenience. Note that the resulting pattern is again canonical. If the target pattern does not contain any variable then the LWA converges after having read the first example. Hence, this case is trivial and we therefore assume in the following always k ≥ 1 , i.e., the target pattern has to contain at least one variable. Our next theorem analyzes the complexity of the union operation. Theorem 7 (Rossmanith and Zeugmann [36]). The union operation can be computed in linear time. Furthermore, the following bound for the stage of convergence for every target pattern from Pat k can be shown. Theorem 8(Rossmanith and Zeugmann [36]). 1 E[CONV ] = O · log1/β (k) for all k ≥ 2 . αk Hence, by total learning time can be estimated by Theorem 7, the expected 1 E[Λ] log1/β (k) for all k ≥ 2 . E[T T ] = O αk For a better understanding of the bound obtained we evaluate it for the uniform distribution and compare it to the minimum number of examples needed for learning a pattern language via the LWA. Theorem 9 (Rossmanith and Zeugmann [36]). E[T T ] = O(2k |π| log|A| (k)) for the uniform distribution and all k ≥ 2 . Theorem 10 (Zeugmann [50]). To learn a pattern π ∈ Pat k the LWA needs exactly log|A| (|A| + k − 1) + 1 examples in the best case.
Can Learning in the Limit Be Done Efficiently?
33
The main difference between the two bounds just given is the factor 2k which precisely reflects the time the LWA has to wait until it has seen the first shortest string from the target pattern language. Moreover, in the best-case the LWA is processing shortest examples only. Thus, we introduce MC to denote the number of minimum length examples read until convergence. Then, one can show that 2 ln(k) + 3 E[MC ] ≤ +2 . ln(1/β) Note that Theorem 8 is shown by using the bound for E[MC ] just given. More precisely, we have E[CONV ] = (1/αk )E[MC ] . Now, we are ready to transform the LWA into a stochastic finite learner. Theorem 11 (Rossmanith and Zeugmann [36]). Let α∗ , β∗ ∈ (0, 1) . Assume D to be a class of admissible probability distributions over A+ such that α ≥ α∗ , β ≤ β∗ and E[Λ] finite for all distributions D ∈ D . Then (PAT , D) is stochastically finitely learnable with high confidence from text. Proof. Let D ∈ D , and let δ ∈ (0, 1) be arbitrarily fixed. Furthermore, let t = s1 , s2 , s3 , . . . be any randomly generated text with respect to D for the target pattern language. The wanted learner M uses the LWA as a subroutine. Additionally, it has a counter for memorizing the number of examples already seen. Now, we exploit the fact that the LWA produces a sequence (τn )n∈N+ of hypotheses such that |τn | ≥ |τn+1 | for all n ∈ N+ . The learner runs the LWA until for the first time C many examples have been processed, where |τ | 2 ln(|τ |) + 3 +2 (A) C = α1∗ · ln(1/β∗ ) and τ is the actual output made by the LWA. Finally, in order to achieve the desired confidence, the learner sets γ = log 1δ and runs the LWA for a total of 2 · γ · C examples. This is the reason we need the counter for the number of examples processed. Now, it outputs the last hypothesis τ produced by the LWA, and stops thereafter. Clearly, the learner described above is finite. Let L be the target language and let π ∈ Pat k be the unique pattern such that L = L(π) . It remains to argue that L(π) = L(τ ) with probability at least 1 − δ . First, the bound in (A) is an upper bound for the expected number of examples needed for convergence by the LWA that has been established in Theorem 8 (via the reformulation using E[MC ] given above). On the one hand, this follows from our assumptions about the allowed α and β as well as from the fact that |τ | ≥ |π| for every hypothesis output. On the other hand, the learner does not know k , but the estimate #var (π) ≤ |π| is sufficient. Note that we have to use in (A) the bound for E[MC ] given above, since the target pattern may contain zero or one different variables.
34
T. Zeugmann
Therefore, after having processed C many examples the LWA has already converged on average. The desired confidence is then an immediate consequence of Corollary 6. The latter theorem allows a nice corollary which we state next. Making the same assumption as done by Kearns and Pitt [18], i.e., assuming the additional prior knowledge that the target pattern belongs to Pat k , the complexity of the stochastic finite learner given above can be considerably improved. The resulting learning time is linear in the expected string length, and the constant depending on k grows only exponentially in k in contrast to the doubly exponentially growing constant in Kearns and Pitt’s [18] algorithm. Moreover, in contrast to their learner, our algorithm learns from positive data only, and outputs a hypothesis that is correct for the target language with high probability. Again, for the sake of presentation we shall assume k ≥ 2 . Moreover, if the prior knowledge k = 1 is available, then there is also a much better stochastic finite learner for PAT 1 (cf. [34]). Corollary 12. Let α∗ , β∗ ∈ (0, 1) . Assume D to be a class of admissible probability distributions over A+ such that α ≥ α∗ , β ≤ β∗ and E[Λ] finite for all distributions D ∈ D . Furthermore, let k ≥ 2 be arbitrarily fixed. Then there exists a learner M such that (1) M learns (PAT k , D) stochastically finitely with high confidence from text, and
k (2) The running time of M is O α ˆ ∗ E[Λ] log1/β∗ (k) log2 (1/δ) . (* Note that α ˆ ∗k and log1/β∗ (k) now are constants. *)
4
Conclusions
The present paper surveyed results recently obtained concerning the iterative learnability of the class of all pattern languages and finite unions thereof. In particular, it could be shown that there are strong dependencies between iterative learning, the class of admissible hypothesis spaces and additional requirements to the learner such as consistency, conservativeness and the decidability of the inclusion problem for the hypothesis space chosen. Looking at these results, we have seen that the LWA is in some sense optimal. Moreover, by analyzing the average-case behavior of Lange and Wiehagen’s pattern language learning algorithm with respect to its total learning time and by establishing exponentially shrinking tail bounds for a rather rich class of limit learners, we have been able to transform the LWA into a stochastic finite learner. The price paid is the incorporation of a bit prior knowledge concerning the class of underlying probability distributions. When applied to the class of all k -variable pattern languages, where k is a priori known, the resulting total learning time is linear in the expected string length.
Can Learning in the Limit Be Done Efficiently?
35
Thus, the present paper provides evidence that analyzing the average-case behavior of limit learners with respect to their total learning time may be considered as a promising path towards a new theory of efficient algorithmic learning. Recently obtained results along the same path as outlined in Erlebach et al.[11] as well as in Reischuk and Zeugmann [32,34] provide further support for the fruitfulness of this approach. In particular, in Reischuk and Zeugmann [32,34] we have shown that onevariable pattern languages are learnable for basically all meaningful distributions within an optimal linear total learning time on the average. Furthermore, this learner can also be modified to maintain the incremental behavior of Lange and Wiehagen’s [19] algorithm. Instead of memorizing the pair (PRE, SUF) , it can also store just the two or three examples from which the prefix PRE and the suffix SUF of the target pattern has been computed. While it is no longer iterative, it is still a bounded example memory learner. A bounded example memory learner is essentially an iterative learner that is additionally allowed to memorize an a priori bounded number of examples (cf. [9] for a formal definition). While the one-variable pattern language learner from [34] is highly practical, our stochastic finite learner for the class of all pattern languages is still not good enough for practical purposes. But our results surveyed point to possible directions for potential improvements. However, much more effort seems necessary to design a stochastic finite learner for PAT (k) . Additionally, we have applied our techniques to design a stochastic finite learner for the class of all concepts describable by a monomial which is based on Haussler’s [14] Wholist algorithm. Here we have assumed the examples to be binomially distributed. The sample size of our stochastic finite learner is mainly bounded by log(1/δ) log n , where δ is again the confidence parameter and n is the dimension of the underlying Boolean learning domain. Thus, the bound obtained is exponentially better than the bound provided within the PAC model. Our approach also differs from U-learnability introduced by Muggleton [27]. First of all, our learner is fed with positive examples only, while in Muggleton’s [27] model examples labeled with respect to their containment in the target language are provided. Next, we do not make any assumption concerning the distribution of the target patterns. Furthermore, we do not measure the expected total learning time with respect to a given class of distributions over the targets and a given class of distributions for the sampling process, but exclusively in dependence on the length of the target. Finally, we require exact learning and not approximately correct learning.
References 1. D. Angluin, Finding Patterns common to a Set of Strings, Journal of Computer and System Sciences 21, 1980, 46–62. 2. D. Angluin, Inductive inference of formal languages from positive data, Information and Control 45, 1980, 117–135.
36
T. Zeugmann
3. D. Angluin and C.H. Smith. Inductive inference: Theory and methods. Computing Surveys 15, No. 3, 1983, 237–269. 4. D. Angluin and C.H. Smith. Formal inductive inference. “Encyclopedia of Artificial Intelligence” (St.C. Shapiro, Ed.), Vol. 1, pp. 409–418, Wiley-Interscience Publication, New York. 5. S. Arikawa, T. Shinohara and A. Yamamoto, Learning elementary formal systems, Theoretical Computer Science 95, 97–113, 1992. 6. L. Blum and M. Blum, Toward a mathematical theory of inductive inference, Information and Control 28, 125–155, 1975. 7. A. Blumer, A. Ehrenfeucht, D. Haussler and M. Warmuth, Learnability and the Vapnik-Chervonenkis Dimension, Journal of the ACM 36 (1989), 929–965. 8. I. Bratko and S. Muggleton, Applications of inductive logic programming, Communications of the ACM, 1995. 9. J. Case, S. Jain, S. Lange and T. Zeugmann, Incremental Concept Learning for Bounded Data Mining, Information and Computation 152, No. 1, 1999, 74–110. 10. R. Daley and C.H. Smith. On the Complexity of Inductive Inference. Information and Control 69 (1986), 12–40. 11. T. Erlebach, P. Rossmanith, H. Stadtherr, A. Steger and T. Zeugmann, Learning one-variable pattern languages very efficiently on average, in parallel, and by asking queries, Theoretical Computer Science 261, No. 1–2, 2001, 119–156. 12. E.M. Gold, Language identification in the limit, Information and Control 10 (1967), 447–474. 13. S.A. Goldman, M.J. Kearns and R.E. Schapire, Exact identification of circuits using fixed points of amplification functions. SIAM Journal of Computing 22, 1993, 705–726. 14. D. Haussler, Bias, version spaces and Valiant’s learning framework, “Proc. 8th National Conference on Artificial Intelligence,” pp. 564–569, San Mateo, CA: Morgan Kaufmann, 1987. 15. D. Haussler, M. Kearns, N. Littlestone and M.K. Warmuth, Equivalence of models for polynomial learnability. Information and Computation 95 (1991), 129–161. 16. S. Jain, D. Osherson, J.S. Royer and A. Sharma, “Systems That Learn: An Introduction to Learning Theory,” MIT-Press, Boston, Massachusetts, 1999. 17. T. Jiang, A. Salomaa, K. Salomaa and S. Yu, Inclusion is undecidable for pattern languages, in “Proceedings 20th International Colloquium on Automata, Languages and Programming,” (A. Lingas, R. Karlsson, and S. Carlsson, Eds.), Lecture Notes in Computer Science, Vol. 700, pp. 301–312, Springer-Verlag, Berlin, 1993. 18. M. Kearns L. Pitt, A polynomial-time algorithm for learning k –variable pattern languages from examples. in “Proc. Second Annual ACM Workshop on Computational Learning Theory” (pp. 57–71). San Mateo, CA: Morgan Kaufmann, 1989. 19. S. Lange and R. Wiehagen, Polynomial-time inference of arbitrary pattern languages. New Generation Computing 8 (1991), 361–370. 20. S. Lange and T. Zeugmann, Language learning in dependence on the space of hypotheses. in “Proc. of the 6th Annual ACM Conference on Computational Learning Theory,” (L. Pitt, Ed.), pp. 127–136, ACM Press, New York, 1993. 21. S. Lange and T. Zeugmann, Set-driven and Rearrangement-independent Learning of Recursive Languages, Mathematical Systems Theory 29 (1996), 599–634. 22. S. Lange and T. Zeugmann, Incremental Learning from Positive Data, Journal of Computer and System Sciences 53(1996), 88–103. 23. N. Lavraˇc and S. Dˇzeroski, “Inductive Logic Programming: Techniques and Applications,” Ellis Horwood, 1994.
Can Learning in the Limit Be Done Efficiently?
37
24. T. Mitchell. “Machine Learning,” McGraw Hill, 1997. 25. A. Mitchell, A. Sharma, T. Scheffer and F. Stephan, The VC-dimension of Subclasses of Pattern Languages, in “Proc. 10th International Conference on Algorithmic Learning Theory,” (O. Watanabe and T. Yokomori, Eds.), Lecture Notes in Artificial Intelligence, Vol. 1720, pp. 93–105, Springer-Verlag, Berlin, 1999. 26. S. Miyano, A. Shinohara and T. Shinohara, Polynomial-time learning of elementary formal systems, New Generation Computing, 18:217–242, 2000. 27. S. Muggleton, Bayesian Inductive Logic Programming, in “Proc. 7th Annual ACM Conference on Computational Learning Theory” (M. Warmuth, Ed.), pp. 3–11, ACM Press, New York, 1994. 28. S. Muggleton and L. De Raedt, Inductive logic programming: Theory and methods, Journal of Logic Programming, 19/20:669–679, 1994. 29. R.P. Nix, Editing by examples, Yale University, Dept. Computer Science, Technical Report 280, 1983. 30. D.N. Osherson, M. Stob and S. Weinstein, “Systems that Learn, An Introduction to Learning Theory for Cognitive and Computer Scientists,” MIT-Press, Cambridge, Massachusetts, 1986. 31. L. Pitt, Inductive Inference, DFAs and Computational Complexity, in “Proc. 2nd Int. Workshop on Analogical and Inductive Inference” (K.P. Jantke, Ed.), Lecture Notes in Artificial Intelligence, Vol. 397, pp. 18–44, Springer-Verlag, Berlin, 1989. 32. R. Reischuk and T. Zeugmann, Learning One- Variable Pattern Languages in Linear Average Time, in “Proc. 11th Annual Conference on Computational Learning Theory - COLT’98,” July 24th - 26th, Madison, pp. 198–208, ACM Press 1998. 33. R. Reischuk and T. Zeugmann, A Complete and Tight Average-Case Analysis of Learning Monomials, in “Proc. 16th International Symposium on Theoretical Aspects of Computer Science,” (C. Meinel and S. Tison, Eds.), Lecture Notes in Computer Science, Vol. 1563, pp. 414–423, Springer-Verlag , Berlin 1999. 34. R. Reischuk and T. Zeugmann, An Average-Case Optimal One-Variable Pattern Language Learner, Journal of Computer and System Sciences 60, No. 2, 2000, 302–335. 35. H. Rogers, Jr., “Theory of Recursive Functions and Effective Computability,” McGraw–Hill, New York, 1967. 36. P. Rossmanith and T. Zeugmann. Stochastic Finite Learning of the Pattern Languages, Machine Learning 44, No. 1-2, 2001, 67–91. 37. Patterns (The Formal Language Theory Column), EATCS Bulletin 54, 46–62, 1994. 38. Return to patterns (The Formal Language Theory Column), EATCS Bulletin 55, 144–157, 1994. 39. S. Shimozono, A. Shinohara, T. Shinohara, S. Miyano, S. Kuhara and S. Arikawa, Knowledge acquisition from amino acid sequences by machine learning system BONSAI, Trans. Information Processing Society of Japan 35, 2009–2018, 1994. 40. R.E. Schapire, Pattern languages are not learnable, In M.A. Fulk & J. Case (Eds.), Proceedings of the Third Annual ACM Workshop on Computational Learning Theory, (pp. 122–129). San Mateo, CA: Morgan Kaufmann, (1990). 41. T. Shinohara, Inferring unions of two pattern languages, Bulletin of Informatics and Cybernetics 20, 83–88, 1983. 42. T. Shinohara, Inductive inference of monotonic formal systems from positive data, New Generation Computing 8, 371–384, 1991. 43. T.Shinohara and S. Arikawa, Pattern inference, in “Algorithmic Learning for Knowledge-Based Systems,” (K.P. Jantke and S. Lange, Eds.), Lecture Notes in Artificial Intelligence, Vol. 961, pp. 259–291, Springer-Verlag, Berlin, 1995.
38
T. Zeugmann
44. T. Shinohara and H. Arimura, Inductive inference of unbounded unions of pattern languages from positive data, in “Proceedings 7th International Workshop on Algorithmic Learning Theory,” (S. Arikawa and A.K. Sharma, Eds.), Lecture Notes in Artificial Intelligence, Vol. 1160, pp. 256–271, Springer-Verlag, Berlin, 1996. 45. R. Smullyan, “Theory of Formal Systems,” Annals of Mathematical Studies, No. 47. Princeton, NJ, 1961. 46. L.G. Valiant, A Theory of the Learnable, Communications of the ACM 27 (1984), 1134–1142. 47. R. Wiehagen. Limes-Erkennung rekursiver Funktionen durch spezielle Strategien. Journal of Information Processing and Cybernetics (EIK) 12, 1976, 93–99. 48. R. Wiehagen and T. Zeugmann, Ignoring Data may be the only Way to Learn Efficiently, Journal of Experimental and Theoretical Artificial Intelligence 6 (1994), 131–144. 49. K. Wright, Identification of unions of languages drawn from an identifiable class, in “Proceedings of the 2nd Workshop on Computational Learning Theory,” (R. Rivest, D. Haussler, and M. Warmuth, Eds.), pp. 328–333, San Mateo, CA: Morgan Kaufmann, 1989. 50. T. Zeugmann, Lange and Wiehagen’s Pattern Language Learning Algorithm: An Average-case Analysis with respect to its Total Learning Time, Annals of Mathematics and Artificial Intelligence 23, No. 1–2, 1998, 117–145. 51. T. Zeugmann and S. Lange, A guided tour across the boundaries of learning recursive languages, in “Algorithmic Learning for Knowledge-Based Systems,” (K.P. Jantke and S. Lange, Eds.), Lecture Notes in Artificial Intelligence, Vol. 961, pp. 190–258, Springer-Verlag, Berlin, 1995. 52. T. Zeugmann, S. Lange and S. Kapur, Characterizations of monotonic and dual monotonic language learning, Information and Computation 120, 155–173, 1995.
Intrinsic Complexity of Uniform Learning Sandra Zilles Universit¨ at Kaiserslautern, FB Informatik, Postfach 3049, 67653 Kaiserslautern, Germany, [email protected]
Abstract. Inductive inference is concerned with algorithmic learning of recursive functions. In the model of learning in the limit a learner successful for a class of recursive functions must eventually find a program for any function in the class from a gradually growing sequence of its values. This approach is generalized in uniform learning, where the problem of synthesizing a successful learner for a class of functions from a description of this class is considered. A common reduction-based approach for comparing the complexity of learning problems in inductive inference is intrinsic complexity. In this context, reducibility between two classes is expressed via recursive operators transforming target functions in one direction and sequences of corresponding hypotheses in the other direction. The present paper is the first one concerned with intrinsic complexity of uniform learning. The relevant notions are adapted and illustrated by several examples. Characterizations of complete classes finally allow for various insightful conclusions. The connection to intrinsic complexity of non-uniform learning is revealed within several analogies concerning firstly the role and structure of complete classes and secondly the general interpretation of the notion of intrinsic complexity.
1
Introduction
Inductive inference is concerned with algorithmic learning of recursive functions. In the model of learning in the limit, cf. [7], a learner successful for a class of recursive functions must eventually find a correct program for any function in the class from a gradually growing sequence of its values. The learner is understood as a machine – called inductive inference machine or IIM – reading finite sequences of input-output pairs of a target function, and returning programs as its hypotheses, see also [2]. The underlying programming system is then called a hypothesis space. Studying the potential of such IIMs in general leads to the question whether – given a description of a class of functions – a corresponding successful IIM can be synthesized computationally from this description. This idea is generalized in the notion of uniform learning: we consider a collection C0 , C1 , . . . of learning problems – which may be seen as a decomposition of a class C = C0 ∪ C1 ∪ . . . – and ask for some kind of meta-IIM tackling the whole collection of learning problems. As an input, such a meta-IIM gets a description of one of the learning R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, pp. 39–53, 2003. c Springer-Verlag Berlin Heidelberg 2003
40
S. Zilles
problems Ci (in our context a class Ci of recursive functions) in the collection. The meta-IIM is then supposed to develop a successful IIM for Ci . Besides studies on uniform learning of classes of recursive functions, cf. [12,16], this topic has also been investigated in the context of learning formal languages, see in particular [1,13,14]. Since we consider IIMs as tackling a given problem, namely the problem of identifying all elements in a particular class of recursive functions, the complexity of such IIMs might express, how hard a learning problem is. For instance, the class of all constant functions allows for a simple and straightforward identification method; for other classes successful methods might seem more complicated. But this does not involve any rule allowing us to compare two learning problems with respect to their difficulty. So a formal approach for comparing the complexity of learning problems (i. e. of classes of recursive functions) is desirable. Different aspects have been analysed in this context. One approach is, e. g., mind change complexity measured by the maximal number of hypothesis changes a machine needs to identify a function in the given class, see [3]. But since in general this number of mind changes is unbounded, other notions of complexity might be of interest. Various subjects in theoretical computer science deal with comparing the complexity of decision problems, e. g. regarding decidability as such, see [15], or the possible efficiency of decision algorithms, see [5]. In general Problem A is at most as hard as Problem B, if A is reducible to B under a given reduction. Each such reduction involves a notion of complete (hardest solvable) problems. Besides studies concerning language learning, see [9,10,11], in [4] an approach for reductions in the context of learning recursive functions is introduced. This subject, intrinsic complexity, has been further analysed in [8] with a focus on complete classes. It has turned out that, for learning in the limit, a class is complete, iff it contains a dense r. e. subclass. Here the aspect of high topological complexity (density), contrasts with the aspect of low algorithmic complexity of r. e. sets, which is somehow striking and has caused discussions on whether this particular approach of intrinsic complexity is adequate. The present paper deals with intrinsic complexity in the context of uniform learning. Assume some new reduction expresses such an idea of intrinsic complexity. If a class C of functions is complete in the initial sense, natural questions are (i) whether C can be decomposed into a uniformly learnable collection C0 , C1 , . . . , which is not a hardest problem in uniform learning, and (ii) whether there are also inappropriate decompositions of C, i. e. collections of highest complexity in uniform learning. Below a notion of intrinsic complexity for uniform learning is developed and the corresponding complete classes are characterized. The obtained structure of degrees of complexity matches recent results on uniform learning: it has been shown that even decompositions into singleton classes can yield problems too hard for uniform learning in Gold’s model. This suggests that collections representing singleton classes may sometimes form hardest problems in uniform learning. Indeed, the notion developed below expresses this intuition, i. e. collec-
Intrinsic Complexity of Uniform Learning
41
tions of singleton sets may constitute complete classes in uniform learning. Still, the characterization of completeness here reveals a weakness of the general idea of intrinsic complexity, namely – as in the non-uniform case – complete classes have a low algorithmic complexity (see Theorem 7). All in all, this shows that intrinsic complexity, as in [4], is on the one hand a useful approach, because it can be adapted to match the intuitively desired results in uniform learning. On the other hand, the doubts in [8] are corroborated.
2 2.1
Preliminaries Notations
Knowledge of basic notions used in mathematics and computability theory is assumed, cf. [15]. N is the set of natural numbers. The cardinality of a set X is denoted by card X. Partial-recursive functions always operate on natural numbers. If f is a function, f (n) ↑ indicates that f (n) is undefined. Our target objects for learning will always be recursive functions, i. e. total partial-recursive functions. R denotes the set of all recursive functions. If α is a finite tuple of numbers, then |α| denotes its length. Finite tuples are coded, i. e. if f (0), . . . , f (n) are defined, a number f [n] represents the tuple (f (0), . . . , f (n)), called an initial segment of f . f [n] ↑ means that f (x) ↑ for some x ≤ n. For convenience, a function may be written as a sequence of values or as a set of input-output pairs. A sequence σ = x0 , x1 , x2 , . . . converges to x, iff xn = x for all but finitely many n; we write lim(σ) = x. For example let f (n) = 7 for n ≤ 2, f (n) ↑ otherwise; g(n) = 7 for all n. Then f = 73 ↑∞ = {(0, 7), (1, 7), (2, 7)}, g = 7∞ = {(n, 7) | n ∈ N}; lim(g) = 7, and f ⊆ g. For n ∈ N, the notion f =n g means that for all x ≤ n either f (n) ↑ and g(n) ↑ or f (n) = g(n). A set C of functions is dense, iff for any f ∈ C, n ∈ N there is some g ∈ C satisfying f =n g, but f = g. Recursive functions – our target objects for learning – require appropriate representation schemes, to be used as hypothesis spaces. Partial-recursive enumerations serve for that purpose: any (n + 1)-place partial-recursive function ψ enumerates the set Pψ := {ψi | i ∈ N} of n-place partial-recursive functions, where ψi (x) := ψ(i, x) for all x = (x1 , . . . , xn ). Then ψ is called a numbering. Given f ∈ Pψ , any index i satisfying ψi = f is a ψ-program of f . Following [6], we call a family (di )i∈N of natural numbers limiting r. e., iff there is a recursive numbering d such that lim(di ) = di for all i ∈ N. 2.2
Learning in the Limit and Intrinsic Complexity
Below, let τ be a fixed acceptable numbering, serving as a hypothesis space. The learner is a total computable device called IIM (inductive inference machine) working in steps. The input of an IIM M in step n is an initial segment f [n] of some f ; the output M (f [n]) is interpreted as a τ -program. In learning in the limit, M is successful for f , if the sequence M (f ) := (M (f [n]))n∈N of hypotheses is admissible for f :
42
S. Zilles
Definition 1 [4] Let f, σ ∈ R. σ is admissible for f , iff σ converges and lim(σ) is a τ -program for f . Now a class of recursive functions is learnable in the limit (Ex -learnable; Ex is short for explanatory), if a single IIM is successful for all functions in the class. Definition 2 [7,2] A class C ⊆ R is Ex -learnable (C ∈ Ex ), iff there is an IIM M such that, for any f ∈ C, the sequence M (f ) is admissible for f . M is then called an Ex -learner or an IIM for C. The class of constant functions and the class Cfsup = {α0∞ | α is an initial segment} of recursive functions of finite support are in Ex , but intuitively, the latter is harder to learn. A reduction-based approach for comparing the learning complexity is proposed in [4], using the notion of recursive operators. Definition 3 [15,8] Let Θ be a total function operating on functions. Θ is a recursive operator, iff for all functions f, g and all numbers n, y ∈ N: 1. if f ⊆ g, then Θ(f ) ⊆ Θ(g); 2. if Θ(f )(n) = y, then Θ(f )(n) = y for some initial segment f ⊆ f ; 3. if f is finite, then one can effectively (in f ) enumerate Θ(f ). Reducing a class C1 of functions to a class C2 of functions requires two operators: the first one maps C1 into C2 ; the second maps any admissible sequence for a mapped function in C2 to an admissible sequence for the associated original function in C1 . Definition 4 [4] Let C1 , C2 ∈ Ex . C1 is Ex -reducible to C2 , iff there are recursive operators Θ, Ξ such that all functions f ∈ C1 fulfil the following conditions: 1. Θ(f ) belongs to C2 , 2. if σ is admissible for Θ(f ), then Ξ(σ) is admissible for f . Note, if C1 is Ex -reducible to C2 , then an IIM for C1 can be deduced from any IIM for C2 ; e. g. by [4], each class in Ex is Ex -reducible to Cfsup . As usual, this reduction yields complete classes, i. e. learnable classes of highest complexity. Definition 5 [4] A class C ∈ Ex is Ex -complete, iff each class C ∈ Ex is Ex -reducible to C. By the remark above, the class Cfsup is Ex -complete. Note that Cfsup is r. e. and dense – a relevant property for characterizing Ex -complete classes: Theorem 1 [8] A class C ∈ Ex is Ex -complete iff it has an r. e. dense subset. Ex -complete classes have subsets, which are dense, i. e. topologically complex, but r. e., i. e. algorithmically non-complex. The latter is astonishing, since there are dense classes, which are not Ex -complete, cf. [8], so they do not contain r. e. dense subsets. These classes are algorithmically more complex than Cfsup , but belong to a lower degree of intrinsic complexity. R. e. subsets as in Theorem 1 are obtained by mapping r. e. Ex -complete classes – such as Cfsup – to C with the help of an operator Θ. So perhaps this approach of intrinsic complexity just makes a class complete, if it is a suitable ‘target’ for recursive operators. This may be considered as a weakness of the notion of intrinsic complexity.
Intrinsic Complexity of Uniform Learning
2.3
43
Uniform Learning in the Limit
Uniform learning views the approach of Ex -learning on a meta-level; it is not only concerned with the existence of methods solving specific learning problems, but with the problem to synthesize such methods. So the focus is on families of learning problems (here families of classes of recursive functions). Given a representation or description of a class of recursive functions, the aim is to effectively determine an adequate learner, i. e. to compute a program for a successful IIM learning the class. For a formal definition of uniform learning it is necessary to agree on a scheme for describing classes of recursive functions (i. e. describing learning problems). For that purpose we fix a three-place acceptable numbering ϕ. If d ∈ N, the numbering ϕd is the function resulting from ϕ, if the first input is fixed by d. Then any number d corresponds to a two-place numbering ϕd enumerating the set Pϕd of partial-recursive functions. Now it is conceivable to consider the subset of all total functions in Pϕd as a learning problem which is uniquely determined by the number d. Thus each number d acts as a description of the set Rd , where Rd := {ϕdi | i ∈ N and ϕdi is recursive} = Pϕd ∩ R for any d ∈ N . Rd is called recursive core of the numbering ϕd . So any set D = {d0 , d1 , . . . } can be regarded as a set of descriptions, i. e. a collection of learning problems Rd0 , Rd1 , . . . In this context, D is called a description set. A meta-IIM M is an IIM with two inputs: (i) a description d of a recursive core Rd , and (ii) an initial segment f [n] of some f ∈ R. Then Md is the IIM resulting from M , if the first input is fixed by d. A meta-IIM M can be seen as mapping descriptions d to IIMs Md ; it is a successful uniform learner for a set D, in case Md learns Rd for all d ∈ D; i. e. given any description in D, M develops a suitable learner for the corresponding recursive core. Definition 6 Let D ⊆ N. D is uniformly Ex -learnable (D ∈ UEx ), iff there is a meta-IIM M such that, for any d ∈ D, the IIM Md is an Ex -learner for Rd . As a numbering ϕd enumerates a superset of Rd , a meta-IIM might also use ϕd as a hypothesis space for Rd . This involves a new notion of admissible sequences. Definition 7 Let d ∈ N, f ∈ Rd , σ ∈ R. σ is r -admissible for d and f , iff σ converges and lim(σ) is a ϕd -program for f . This approach yields just a special (restricted ) case of uniform Ex -learning, because ϕd -programs can be uniformly translated into τ -programs. Definition 8 Let D ⊆ N. D is uniformly Ex -learnable restrictedly (D ∈ rUEx ), iff there is a meta-IIM M such that, for any d ∈ D and any function f ∈ Rd , the sequence Md (f ) is r -admissible for d and f . By the following result, special sets describing only singleton recursive cores are not uniformly Ex -learnable (restrictedly). For Claim 2 cf. a proof in [16].
44
S. Zilles
Theorem 2 1. [12,16] {d ∈ N | card Rd = 1} ∈ / UEx . 2. Fix s ∈ R. Then {d ∈ N | Rd = {s}} ∈ / rUEx . It has turned out, that even UEx -learnable subsets of these description sets are not in UEx (or rUEx ), if additional demands concerning the sequence of hypotheses are posed, see [17]. This suggests that description sets representing only singletons may form hardest problems in uniform learning; analogously description sets representing only a fixed singleton recursive core may form hardest problems in restricted uniform learning. Hopefully, this intuition can be expressed by a notion of intrinsic complexity of uniform learning.
3 3.1
Intrinsic Complexity of Uniform Learning Intrinsic Complexity of UEx -Learning
The crucial notion now concerns the reduction between description sets D1 and D2 . As in the non-uniform model, a meta-IIM for D1 should be computable from a meta-IIM for D2 , if D1 is reducible to D2 . We first focus on UEx -learning; the restricted variant will be discussed later on. A first idea for UEx -reducibility might be to demand the existence of operators Θ and Ξ such that for d1 ∈ D1 and f1 ∈ Rd1 Θ transforms (d1 , f1 ) into a pair (d2 , f2 ) with d2 ∈ D2 and f2 ∈ Rd2 ; where Ξ maps any admissible sequence for f2 to an admissible sequence for f1 . Unfortunately, this does not allow us to reduce every set in UEx to a set describing only singleton recursive cores: suppose Rd = Cfsup . As the set D1 = {d} is uniformly Ex -learnable, it should be reducible to a set D2 representing only singleton recursive cores, say via Θ and Ξ as above. Now for any initial segment α, there are d2 ∈ D2 and f2 ∈ Rd2 such that Θ(d, α0∞ ) = (d2 , f2 ). The usual notion of an operator yields an n > 0 and a subfunction σ ⊆ f2 such that Θ(d, α0n ) = (d2 , σ). As card Rd2 = 1, this implies Θ(d, α0n β0∞ ) = (d2 , f2 ) for all initial segments β. In particular, there are f, f ∈ Rd such that f = f , but Θ(d, f ) = Θ(d, f ) = (d2 , f2 ). By assumption, Ξ maps each admissible sequence for f2 to a sequence admissible for both f and f . The latter is of course impossible, so this approach does not meet our purpose. The problem above is that the description d2 , once it is output by Θ on input of (d1 , f1 [m]), can never be changed depending on the values of f1 to be read. Hence, Θ should be allowed to return a sequence of descriptions, when fed a pair (d1 , f1 ). As an improved approach, it is conceivable to demand, that for d1 ∈ D1 and f1 ∈ Rd1 Θ transforms (d1 , f1 ) into a pair (δ2 , f2 ) . Here δ2 is a sequence converging to some d2 ∈ D2 with f2 ∈ Rd2 . Moreover, Ξ maps any admissible sequence for f2 to an admissible sequence for f1 . Still this approach bears a problem. Intuitively, reducibility should be transitive. In general, such a transitivity is achieved by connecting the operators of a
Intrinsic Complexity of Uniform Learning
45
first reduction with the operators of a second reduction. The idea above cannot guarantee that: assume D1 is reducible to D2 via Θ1 and Ξ1 ; D2 is reducible to D3 via Θ2 and Ξ2 . If Θ1 maps (d1 , f1 ) to (δ2 , f2 ), then which description d in the sequence δ2 should form an input (d, f2 ) for Θ2 ? It is in general impossible to detect the limit d2 of the sequence δ2 , and any description d = d2 might change the output of Θ2 . So it is inevitable to let Θ operate on sequences of descriptions and on functions, i. e., Θ maps pairs (δ1 , f1 ), where δ1 is a sequence of descriptions, to pairs (δ2 , f2 ). Definition 9 Let Θ be a total function operating on pairs of functions. Θ is a recursive meta-operator, iff the following properties hold for all functions δ, δ , f, f : 1. if δ ⊆ δ , f ⊆ f , as well as Θ(δ, f ) = (γ, g) and Θ(δ , f ) = (γ , g ), then γ ⊆ γ and g ⊆ g ; 2. if n, y ∈ N, Θ(δ, f ) = (γ, g), and γ(n) = y (or g(n) = y, resp.), then there are initial segments δ0 ⊆ δ and f0 ⊆ f such that (γ0 , g0 ) = Θ(δ0 , f0 ) fulfils γ0 (n) = y (g0 (n) = y, resp.); 3. if δ, f are finite, Θ(δ, f ) = (γ, g), one can effectively (in δ, f ) enumerate γ, g. This finally allows for the following definition of UEx -reducibility. Definition 10 Let D1 , D2 ∈ UEx . Fix a recursive meta-operator Θ and a recursive operator Ξ. D1 is UEx -reducible to D2 via Θ and Ξ, iff for any d1 ∈ D1 , any f1 ∈ Rd1 , and any initial segment δ1 there are functions δ2 and f2 satisfying: 1. Θ(δ1 d∞ 1 , f1 ) = (δ2 , f2 ), 2. δ2 converges to some description d2 ∈ D2 such that f2 ∈ Rd2 , 3. if σ is admissible for f2 , then Ξ(σ) is admissible for f1 . D1 is UEx -reducible to D2 , iff D1 is UEx -reducible to D2 via some Θ and Ξ . Note that this definition expresses intrinsic complexity in the sense that a meta-IIM for D1 can be computed from a meta-IIM for D2 , if D1 is UEx reducible to D2 . Moreover, as has been demanded in advance, the resulting reducibility is transitive: Lemma 3 If D1 , D2 , D3 are description sets such that D1 is UEx -reducible to D2 and D2 is UEx -reducible to D3 , then D1 is UEx -reducible to D3 . The notion of completeness can be adapted from the usual definitions. Definition 11 A description set D ∈ UEx is UEx -complete, iff each description set D ∈ UEx is UEx -reducible to D. The question is, whether this notion of intrinsic complexity expresses the intuitions formulated in advance, e. g., that there are UEx -complete description sets representing only singleton recursive cores. Before answering this question consider an illustrative example.
46
S. Zilles
This example states that there is a single description d of an Ex -complete set such that the description set {d} is UEx -complete. On the one hand, this might be surprising, because a description set consisting of just one index representing an Ex -learnable class might be considered rather simple and thus not complete for uniform learning. But on the other hand, this result is not contrary to the intuition, that the hardest problems in non-uniform learning may remain hardest, when considered in the context of meta-learning. The reason is that the complexity is still of highest degree, if the corresponding class of recursive functions is not decomposed appropriately. Example 4 Let d ∈ N fulfil Rd = Cfsup . Then the set {d} is UEx -complete. Proof. Obviously, {d} ∈ UEx . To show that each description set in UEx is UEx reducible to {d}, fix D1 ∈ UEx and let M be a corresponding meta-IIM as in Definition 6. It remains to define a recursive meta-operator Θ and a recursive operator Ξ appropriately. Given initial segments δ1 and α, let Θ just modify the sequence of hypotheses returned by the meta-IIM M , if the first input parameter is gradually taken from the sequence δ1 and the second input parameter is gradually taken from the sequence α. The modification is to increase each hypothesis by 1 and to change each repetition of hypotheses into a zero output. A formal definition is omitted. Moreover, given an initial segment σ = (s0 , . . . , sn ), let Ξ(σ) look for the maximal m ≤ n such that at least one of the values τsm (x), x ≤ n, is defined within n steps and greater than 0. In case m does not exist, Ξ(σ) = Ξ(s0 , . . . , sn−1 ). Otherwise, let y ≤ n be maximal such that τsm (y) has already been computed and is greater than 0. Then Ξ(σ) = Ξ(s0 , . . . , sn−1 )τsm (y) − 1. Now D1 is UEx -reducible to {d} via Θ, Ξ; details are omitted. That decompositions of Ex -complete classes may also be not UEx -complete, is shown in Section 3.3. Example 4 moreover serves for proving the completeness of other sets, if Lemma 5 – an immediate consequence of Lemma 3 – is applied. Lemma 5 Let D1 , D2 ∈ UEx . If D1 is UEx -complete and UEx -reducible to D2 , then D2 is UEx -complete. Lemma 5 and Example 4 simplify the proofs of further examples, finally revealing that there are indeed UEx -complete description sets representing singleton recursive cores only. Example 6 1. Let (αi )i∈N be an r. e. family of all initial segments. Let g ∈ R g(i) g(i) fulfil ϕ0 = αi 0∞ and ϕx+1 =↑∞ for i, x ∈ N. Then the description set {g(i) | i ∈ N} is UEx -complete. g(i) g(i) 2. Let g ∈ R fulfil ϕ0 = τi and ϕx+1 =↑∞ for i, x ∈ N. Then the description set {g(i) | i ∈ N} is UEx -complete. Proof. ad 1. Obviously, {g(i) | i ∈ N} ∈ UEx . Now we reduce the UEx -complete set {d} from Example 4 to {g(i) | i ∈ N}. Lemma 5 then proves Assertion 1.
Intrinsic Complexity of Uniform Learning
47
It is easy to define Θ such that, if α does not end with 0, then Θ(δ1 , α0∞ ) = (δ2 , α0∞ ), where δ2 converges to some g(i) with αi = α. Let Ξ(σ) = σ for all σ. Then {d} is UEx -reducible to {g(i) | i ∈ N} via Θ and Ξ. Details are omitted. ad 2. Fix an r. e. family (αi )i∈N of all initial segments; fix h ∈ R with τh(i) = g(h(i)) g(h(i)) αi 0∞ for all i ∈ N. Then ϕ0 = αi 0∞ and ϕx+1 =↑∞ for i, x ∈ N. As above, the set {g(h(i)) | i ∈ N} is UEx -complete; so is its superset {g(i) | i ∈ N}. Just as the properties of Cfsup are characteristic for Ex -completeness, the properties of description sets representing decompositions of Cfsup are characteristic for UEx -completeness, as is stated in Theorem 7 and Corollary 8. Theorem 7 Let D ∈ UEx . D is UEx -complete, iff there are a recursive numbering ψ and a limiting r. e. family (di )i∈N of descriptions in D such that: 1. ψi belongs to Rdi for all i ∈ N; 2. Pψ is dense. Proof. Fix a description set D in UEx . Necessity. Assume D is UEx -complete. Fix any one-one recursive numbering χ such that Pχ = Cfsup . Moreover fix g ∈ R which, given any i, x ∈ N, fulfils g(i) g(i) ϕ0 = χi and ϕx =↑∞ , if x > 0. Then the description set {g(i) | i ∈ N} is UEx -complete, as can be verified similarly to Example 6. Lemma 5 then implies that {g(i) | i ∈ N} is UEx -reducible to D, say via Θ and Ξ. Fix a one-one r. e. family (αi )i∈N of all finite tuples over N. For i ∈ N, i coding the pair (x, y), define (δi , ψi ) := Θ(αy g(x)∞ , χx ). By definition, ψ is a recursive numbering and, for all i ∈ N, the sequence δi converges to some di ∈ D such that ψi ∈ Rdi . Hence (di )i∈N is a limiting r. e. family of descriptions in D. It remains to verify Property 2. For that purpose fix i, n ∈ N. By definition, if i encodes (x, y), we obtain Θ(αy g(x)∞ , χx ) = (δi , ψi ). The properties of Θ yield some m ∈ N such that Θ(αy g(x)m , χx [m]) = (δi , α ) for some δi , α with δi ⊆ δi and ψi [n] ⊆ α ⊆ ψi . Because of the particular properties of χ, there is some x ∈ N, x = x, such that χx =m χx , but χx = χx . Moreover, there is some y ∈ N such that αy = αy g(x)m . If j encodes (x , y ), this yields Θ(αy g(x)m g(x )∞ , χx ) = (δj , ψj ), where α ⊆ ψj . In particular ψj =n ψi . Assume ψi = ψj . Suppose σ is any admissible sequence for ψi . Then σ is admissible for ψj . This implies that Ξ(σ) is admissible for both χx and χx . As χx = χx , this is impossible. So ψi = ψj . Sufficiency. Assume D, ψ, and (di )i∈N fulfil the conditions of Theorem 7. Let d denote a numbering associated to the limiting r. e. family (di )i∈N . The results in the context of non-uniform learning help to show that D is UEx -complete: By assumption, Pψ is a dense r. e. subset of R. Theorem 1 then implies that Pψ is Ex -complete, so Cfsup is Ex -reducible to Pψ , say via Θ , Ξ . Using Θ and Ξ one can show that the UEx -complete set {d} from Example 4 is UEx -reducible to D. This implies that D is UEx -complete, too. Note that Rd = Cfsup .
48
S. Zilles
It remains to define a recursive meta-operator Θ and a recursive operator Ξ appropriately. If δ1 and α1 are finite tuples over N, define Θ(δ1 , α1 ) as follows. Compute Θ (α1 ) = α2 and n = |α2 |. For all x < n, let ix be minimal such that α2 [x] ⊆ ψix . Return Θ(δ1 α1 ) = ((di0 (0), di1 (1), . . . , din−1 (n − 1)), α2 ) (if n = 0, then the first component of Θ(δ1 α1 ) is the empty sequence). Clearly, if f1 ∈ R, then Θ(δ1 , f1 ) = (δ2 , Θ (f1 )) for some sequence δ2 . Moreover, let Ξ := Ξ . Finally, to verify that {d} is UEx -reducible to D, fix a sequence δ1 and a function f1 ∈ Rd . First, note that f2 = Θ (f1 ) ∈ Pψ . Let i be the minimal ψ-program of Θ (f1 ) = f2 . As ψ ∈ R, for all x ∈ N the minimal ix satisfying f2 [x] ⊆ ψix can be computed. Additionally, lim(ix )x∈N = i. Note that di converges to di . Hence Θ(δ1 , f1 ) = (δ2 , f2 ), where f2 ∈ Pψ and δ2 converges to di , given f2 = ψi . In particular, f2 ∈ Rdi . Second, if σ is admissible for f2 , then Ξ (σ) is admissible for f1 . So {d} is UEx -reducible to D via Θ and Ξ, and thus D is UEx -complete. Corollary 8 Let D ∈ UEx . D is UEx -complete, iff there are a recursive numbering ψ and a limiting r. e. family (di )i∈N of descriptions in D such that: 1. ψi belongs to Rdi for all i ∈ N; 2. Pψ is Ex -complete. Proof. Necessity. The assertion follows from Theorem 1 and Theorem 7. Sufficiency. Let D ∈ UEx . Assume ψ and (di )i∈N fulfil the conditions above. Let d be a recursive numbering corresponding to the limiting r. e. family (di )i∈N . By Property 2, Pψ is Ex -complete; thus, by Theorem 1, there exists a dense r. e. subclass C ⊆ Pψ . Let ψ be a one-one, recursive numbering with Pψ = C, in particular Pψ is dense. It remains to find a limiting r. e. family (di )i∈N of descriptions in D such that ψi ∈ Rdi for all i ∈ N. For that purpose define a corresponding numbering d . Given i, n ∈ N, define di (n) as follows. Let j ∈ N be minimal such that ψi =n ψj . (* Note that, for all but finitely many n, the index j will be the minimal ψ-program of ψi . *) Return di (n) := dj (n). (* lim(di ) = dj , for j minimal with ψi = ψj . *) Finally, let di be given by the limit of the function di , in case a limit exists. Fix i ∈ N. Then there is a minimal j with ψi = ψj . By definition, the limit di of di exists and di = dj ∈ D. Moreover, as ψj ∈ Rdj , the function ψi is in Rdi . As ψ and (di )i∈N allow us to apply Theorem 7, the set D is UEx -complete. Thus certain decompositions of Ex -complete classes remain UEx -complete, and UEx -complete description sets always represent decompositions of supersets of Ex -complete classes. Example 9 illustrates how to apply the above characterizations of UEx -completeness. A similar short proof may be given for Example 6.
Intrinsic Complexity of Uniform Learning
49
Example 9 Fix a recursive numbering χ such that Pχ is dense. Let g ∈ R fulfil g(i) g(i) ϕ0 = χi and ϕx+1 =↑∞ for i, x ∈ N. Then {g(i) | i ∈ N} is UEx -complete. Proof. (g(i))i∈N is a (limiting) r. e. family such that χi ∈ Rg(i) for all i ∈ N and Pχ is Ex -complete. Corollary 8 implies that {g(i) | i ∈ N} is UEx -complete. 3.2
Intrinsic Complexity of rUEx -Learning
Adapting the formalism of intrinsic complexity for restricted uniform learning, we have to be careful concerning the operator Ξ. In UEx -learning, the current description d has no effect on whether a sequence is admissible for a function or not. For restricted learning this is different. Therefore, to communicate the relevant information to Ξ, it is inevitable to include a description from D2 in the input of Ξ. That means, Ξ should operate on pairs (δ2 , σ) rather than on sequences σ only. Since only the limit of the function output by Ξ is relevant for the reduction, this idea can be simplified. It suffices, if Ξ operates correctly on the inputs d2 and σ, where d2 is the limit of δ2 . Then an operator on the pair (δ2 , σ) is obtained from Ξ by returning the sequence (Ξ(δ2 (0)σ[0]), Ξ(δ2 (1)σ[1]), . . . ). Its limit will equal the limit of Ξ(d2 σ). Definition 12 Let D1 , D2 ∈ rUEx . Fix a recursive meta-operator Θ and a recursive operator Ξ. D1 is rUEx -reducible to D2 via Θ and Ξ, iff for any d1 ∈ D1 , any f1 ∈ Rd1 , and any initial segment δ1 there are functions δ2 and f2 satisfying: 1. Θ(δ1 d∞ 1 , f1 ) = (δ2 , f2 ), 2. δ2 converges to some description d2 ∈ D2 such that f2 ∈ Rd2 , 3. if σ is r -admissible for d2 and f2 , then Ξ(d2 σ) is r -admissible for d1 and f1 . D1 is rUEx -reducible to D2 , iff D1 is rUEx -reducible to D2 via some Θ and Ξ . Completeness is defined as usual. As in the UEx -case, rUEx -reducibility is transitive; so the rUEx -completeness of one set may help to verify the rUEx completeness of others. Lemma 10 If D1 , D2 , D3 are description sets such that D1 is rUEx -reducible to D2 and D2 is rUEx -reducible to D3 , then D1 is rUEx -reducible to D3 . Lemma 11 Let D1 , D2 ∈ rUEx . If D1 is rUEx -complete and rUEx -reducible to D2 , then D2 is rUEx -complete. Recall that, intuitively, sets describing just one singleton recursive core may be rUEx -complete. This is affirmed by Example 12, the proof of which is omitted. Example 12 Let s, g ∈ R such that ϕi = s and ϕx =↑∞ , if i, x ∈ N, x = i. Then {g(i) | i ∈ N} is rUEx -complete, but not UEx -complete. g(i)
g(i)
Example 12 helps to characterize rUEx -completeness. In particular, it shows that the demand ‘Pψ is dense’ has to be dropped.
50
S. Zilles
Theorem 13 Let D ∈ rUEx . D is rUEx -complete, iff there are a recursive numbering ψ and a limiting r. e. family (di )i∈N of descriptions in D such that: 1. ψi belongs to Rdi for all i ∈ N; 2. for each i, n ∈ N there are infinitely many j ∈ N satisfying ψi =n ψj and (di , ψi ) = (dj , ψj ). Proof. Fix a description set D in rUEx . Necessity. Assume D is rUEx -complete. Lemma 11 implies that the description set {g(i) | i ∈ N} from Example 12 is rUEx -reducible to D, say via Θ and Ξ. Fix a one-one r. e. family (αi )i∈N of all finite tuples over N. For i ∈ N, i coding the pair (x, y), define (δi , ψi ) := Θ(αy g(x)∞ , s). By definition, ψ is a recursive numbering and, for all i ∈ N, the sequence δi converges to some di ∈ D such that ψi ∈ Rdi . Hence (di )i∈N is a limiting r. e. family of descriptions in D. It remains to verify Property 2. For that purpose fix i, n ∈ N. By definition, if i encodes (x, y), we have Θ(αy g(x)∞ , s) = (δi , ψi ). The properties of Θ yield some m ∈ N such that Θ(αy g(x)m , s) = (δi , α ) for some δi and α with δi ⊆ δi and ψi [n] ⊆ α ⊆ ψi . Now choose any x ∈ N such that x = x. Moreover, there is some y ∈ N such that αy = αy g(x)m . If j encodes (x , y ), this yields Θ(αy g(x)m g(x )∞ , s) = (δj , ψj ), where α ⊆ ψj . In particular ψj =n ψi . Assume (di , ψi ) = (dj , ψj ). Suppose σ is any rUEx -admissible sequence for di and ψi . Then Ξ(di σ) is rUEx -admissible for both g(x) and s and g(x ) and s. As x is the only ϕg(x) -number for s and x is the only ϕg(x ) -number for s, the latter is impossible. So (di , ψi ) = (dj , ψj ). Repeating this argument for any x with x = x yields the desired property. Sufficiency. First note: if (di )i∈N is a limiting r. e. family and ψ any recursive numbering, such that {(di , ψi ) | i ∈ N} is an infinite set, then there are a limiting r. e. family (di )i∈N and a recursive numbering ψ , such that {(di , ψi ) | i ∈ N} ⊆ {(di , ψi ) | i ∈ N} and i = j implies (di , ψi ) = (dj , ψj ). Details are omitted. So let D, ψ, (di )i∈N fulfil the demands of Theorem 7 and assume wlog that i = j implies (di , ψi ) = (dj , ψj ). Let d be the numbering associated to the limiting r. e. family (di )i∈N . We show that the set {g(i) | i ∈ N} from Example 12 is rUEx -reducible to D; so Lemma 11 implies that D is rUEx -complete. For that purpose fix a one-one numbering η ∈ R such that Pη equals the set Cconst := {αi∞ | α is a finite tuple over N and i ∈ N} of all recursive finite variants of constant functions. Using a construction from [8] we define an operator Θ mapping Pη into Pψ . In parallel, a function θ is constructed to mark used indices. Let Θ (η0 ) := ψ0 and θ(0) = 0. If i > 0, let Θ (ηi ) be defined as follows. For x < i, let mx be maximal with ηi (mx ) = ηx (mx ). Let m := maxx
Intrinsic Complexity of Uniform Learning
51
Choose h = min(H); return Θ (ηi ) := ψh , moreover let θ(i) := h. (* Because of Property 2, the index h exists. As ψ is recursive, h is found effectively. *) Note that Θ is a recursive operator mapping Pη into Pψ . θ is a recursive function, that maps each number i to the index h used in the construction of Θ (ηi ) = ψh . θ is one-one, yet it may happen, that Θ (ηi ) = Θ (ηj ), but θ(i) = θ(j) for some i, j ∈ N. It remains to define a recursive meta-operator Θ and a recursive operator Ξ such that {g(i) | i ∈ N} is rUEx -reducible to D via Θ and Ξ. If δ is an infinite sequence, define Θ(δ, s) as follows. For each x ∈ N, let jx ∈ N be minimal such that ηjx =x δ. Let ix := θ(jx ). Return Θ(δ, s) := ((di0 (0), di1 (1), . . . ), Θ (δ)). Clearly, the output of Θ depends only on δ. If δ converges, then Θ(δ, s) = (δ , f ), where f ∈ Pψ and δ converges to some description di such that i = θ(j) for the minimal number j satisfying ηj = δ. To define an operator Ξ, compute Ξ(dσ) for d ∈ N, σ ∈ R as follows. For x ∈ N let X := {y ≤ x | dy (x) = d and, for all z ≤ x, if ϕdσ(x) (z) is defined in x steps of computation, then ϕdσ(x) (z) = ψy (z)}. If X is empty, let ix = 0, otherwise let ix be the minimum of X. (* In the limit, the only i satisfying di = d and ϕdlim(σ) = ψi is found – provided that i exists. *) For each x ∈ N, compute jx ∈ N with θ(jx ) = ix . (* In the limit, a number j with θ(j) = i, di = d, and Θ (ηj ) = ψi is found – provided that j exists. *) Return Ξ(dσ) := (g −1 (ηj0 (0)), g −1 (ηj1 (1)), g −1 (ηj2 (2)), . . . ). (* g −1 denotes the function inverse to g. Ξ(dσ) converges to g −1 (l), where l is the limit of ηj with Θ (ηj ) = ψi and di equals d – provided that i and j exist. *) To show that {g(i) | i ∈ N} is rUEx -reducible to D via Θ and Ξ, fix some δ1 ∈ R converging to some description d ∈ {g(i) | i ∈ N}. First, by the remarks below the definition of Θ, we obtain Θ(δ1 , s) = (δ2 , f2 ), where f2 ∈ Pψ and δ2 converges to some description di such that i = θ(j) for the minimal j satisfying ηj = δ1 . This implies f2 = ψi . In particular, f2 ∈ Rdi . Second, if σ ∈ R is r -admissible for di and ψi , then Ξ(di σ) converges to g −1 (d) (by the note in the definition of Ξ and d = lim(ηj )). Recall that g −1 (d) is the only ϕd -program of s, whenever d ∈ {g(i) | i ∈ N}. Hence Ξ(di σ) is r -admissible for d and s. So {g(i) | i ∈ N} is rUEx -reducible to D and finally D is rUEx -complete. As an immediate consequence of Theorems 7 and 13 we have: Corollary 14 Let D ∈ rUEx . If D is UEx -complete, then D is rUEx -complete. 3.3
Algorithmic Structure of Complete Classes in Uniform Learning
Theorems 7 and 13 suggest a weakness of the notion of intrinsic complexity, similar to the non-uniform case: though UEx -/rUEx -complete sets involve a
52
S. Zilles
topologically complex structure, expressed by Property 2, this goes along with the demand for a limiting r. e. subset combined with an r. e. subset Pψ of the union of all represented recursive cores. The latter again can be seen as a noncomplex algorithmic structure. Now Theorem 15 shows that there are non-complete description sets, for which the properties of Theorems 7 and 13 can be fulfilled, but only if the demand for limiting r. e. sets is dropped. These sets are algorithmically more complex than our examples of UEx -complete sets, but they belong to a lower degree of intrinsic complexity. Theorem 15 Let C ⊆ R. Then there is a set D ∈ rUEx such that 1. C equals the union of all recursive cores described by D, 2. D is not rUEx -complete (and hence not UEx -complete). Proof. Fix a list A0 , A1 , . . . of all infinite limiting r. e. sets such that ϕd0 ∈ C and d ∞ ϕx+1 =↑ for all i, x ∈ N and d ∈ Ai . Let A := i∈N Ai and C = {f0 , f1 , . . . }. Define a set D0 as follows. Fix the least elements d0 , d0 of A0 , d0 < d0 . Let I0 := {d0 }, I0 := {d0 }. Let e0 ∈ A \ (I0 ∪ I0 ) be minimal such that f0 ∈ Re0 . (* e0 exists, because A contains infinitely many descriptions d with ϕd0 = f0 . *) Let D0 := I0 ∪ {e0 }. (* The disjoint sets D0 and I0 both intersect with A0 ; some recursive core described by D0 equals {f0 }. *) Moreover, for any k ∈ N, define a set Dk+1 as follows. Fix the least elements dk+1 , dk+1 of Ak+1 \ (Dk ∪ Ik ), dk+1 < dk+1 . (* These have not been touched in the definition of D0 , . . . , Dk yet. *) Let Ik+1 := Dk ∪{dk+1 }, Ik+1 := Ik ∪{dk+1 }. Let ek+1 ∈ A\(Ik+1 ∪Ik+1 ) be minimal such that fk+1 ∈ Rek+1 . (* ek+1 exists, because A contains infinitely many descriptions d with ϕd0 = fk+1 . *) Let Dk+1 := Ik+1 ∪{ek+1 }. (* The disjoint sets Dk+1 and Ik+1 both intersect with Ak+1 ; some recursive core described by Dk+1 equals {fk+1 }. *) Choose D := k∈N Dk ⊂ A, so D does not contain any infinite limiting r. e. set. As ϕdx+1 =↑∞ for all d ∈ D, x ∈ N, we have D ∈ rUEx . Moreover, C is the union of all cores described by D. It remains to prove that D is not rUEx -complete. Assume D is rUEx -complete. Then some limiting r. e. set {di | i ∈ N} ⊆ D and some ψ ∈ R fulfil the conditions of Theorem 13. In particular, {(di , ψi ) | i ∈ N} is infinite. As D does not contain any infinite limiting r. e. set, the set {di | i ∈ N} is finite. card Rdi = 1 for i ∈ N implies that {ψi | i ∈ N} is finite, too; thus {(di , ψi ) | i ∈ N} is finite – a contradiction. So D is not rUEx -complete. The reason why each UEx -/rUEx -complete set D contains a limiting r. e. subset representing a decomposition of an r. e. class is that certain properties of UEx -complete sets are ‘transferred’ by meta-operators Θ. This corroborates the possible interpretation that our approach of intrinsic complexity just makes a
Intrinsic Complexity of Uniform Learning
53
class complete, if it is a suitable ‘target’ for recursive meta-operators – similar to the non-uniform case. By the way, Theorem 15 shows, that every Ex -complete class C has a decomposition represented by a description set which is not UEx -complete – answering a question in Section 3.1. Acknowledgements. My thanks are due to the referees and to Steffen Lange for their comments correcting and improving a former version of this paper, moreover to Frank Stephan for a very helpful discussion on the technical details.
References 1. Baliga, G.; Case, J.; Jain, S. (1999); The synthesis of language learners, Information and Computation 152:16–43. 2. Blum, L.; Blum, M. (1975); Toward a mathematical theory of inductive inference, Information and Control 28:125–155. 3. Case, J.; Smith, C. (1983); Comparison of identification criteria for machine inductive inference, Theoretical Computer Science 25:193–220. 4. Freivalds, R.; Kinber, E.; Smith, C. (1995); On the intrinsic complexity of learning, Information and Computation 123:64–71. 5. Garey, M; Johnson, D. (1979); Computers and Intractability – A Guide to the Theory of NP-Completeness, Freeman and Company. 6. Gold, E. M. (1965); Limiting recursion, Journal of Symbolic Logic 30:28–48. 7. Gold, E. M. (1967); Language identification in the limit, Information and Control 10:447–474. 8. Jain, S.; Kinber, E.; Papazian, C.; Smith, C.; Wiehagen, R. (2003); On the intrinsic complexity of learning recursive functions, Information and Computation 184:45– 70. 9. Jain, S.; Kinber, E.; Wiehagen, R. (2000); Language learning from texts: Degrees of intrinsic complexity and their characterizations, Proc. 13th Annual Conference on Computational Learning Theory, Morgan Kaufmann, 47–58. 10. Jain, S.; Sharma, A. (1996); The intrinsic complexity of language identification, Journal of Computer and System Sciences 52:393–402. 11. Jain, S.; Sharma, A. (1997); The structure of intrinsic complexity of learning, Journal of Symbolic Logic 62:1187–1201. 12. Jantke, K. P. (1979); Natural properties of strategies identifying recursive functions, Elektronische Informationsverarbeitung und Kybernetik 15:487–496. 13. Kapur, S.; Bilardi, G. (1992); On uniform learnability of language families, Information Processing Letters 44:35–38. 14. Osherson, D.; Stob, M.; Weinstein, S. (1988); Synthesizing inductive expertise, Information and Computation 77:138–161. 15. Rogers, H. (1987); Theory of Recursive Functions and Effective Computability, MIT Press. 16. Zilles, S. (2001); On the synthesis of strategies identifying recursive functions, Proc. 14th Annual Conference on Computational Learning Theory, LNAI 2111, pp. 160–176, Springer-Verlag. 17. Zilles, S. (2001); On the comparison of inductive inference criteria for uniform learning of finite classes, Proc. 12th Int. Conference on Algorithmic Learning Theory, LNAI 2225, pp. 251–266, Springer-Verlag.
On Ordinal VC-Dimension and Some Notions of Complexity Eric Martin1 , Arun Sharma2 , and Frank Stephan2 1
School of Computer Science and Engineering, UNSW Sydney, NSW 2052, Australia [email protected] 2 National ICT Australia, UNSW Sydney, NSW 2052, Australia {Arun.Sharma,Frank.Stephan}@nicta.com.au
Abstract. We generalize the classical notion of VC-dimension to ordinal VC-dimension, in the context of logical learning paradigms. Logical learning paradigms encompass the numerical learning paradigms commonly studied in Inductive inference. A logical learning paradigm is defined as a set W of structures over some vocabulary, and a set D of first-order formulas that represent data. The sets of models of ϕ in W, where ϕ varies over D, generate a natural topology W over W. We show that if D is closed under boolean operators, then the notion of ordinal VC-dimension offers a perfect characterization for the problem of predicting the truth of the members of D in a member of W, with an ordinal bound on the number of mistakes. This shows that the notion of VC-dimension has a natural interpretation in Inductive Inference, when cast into a logical setting. We also study the relationships between predictive complexity, selective complexity—a variation on predictive complexity—and mind change complexity. The assumptions that D is closed under boolean operators and that W is compact often play a crucial role to establish connections between these concepts. Keywords: Inductive inference, logical paradigms, VC-dimension, predictive complexity, mind change bounds.
1
Introduction
The notion of VC-dimension is a key concept in PAC-learning [6,12,13]. The notion of finite telltale is a key concept in Inductive inference [3,8]. It can be claimed that VC-dimension is to PAC-learning what finite telltales are to Inductive inference. Both provide a characterization of learnability, for fundamental classes of learning paradigms, in the respective settings. Both take the form of a condition where finiteness is a key requirement, in frameworks that deal with infinite objects. In logical learning paradigms of identification in the limit, it has been shown that the finite telltale condition can be seen as a generalization of the compactness property, the latter being the hallmark of, equivalently, finite learning or deductive inference [10]. The finite telltale condition can even be generalized and be interpreted as a property of β-weak compactness, that characterizes classification with less than β mind changes [10]. There are extensions of R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, pp. 54–68, 2003. c Springer-Verlag Berlin Heidelberg 2003
On Ordinal VC-Dimension and Some Notions of Complexity
55
VC dimension to infinite domains [4]. But there are few essential connection between VC-dimension and some fundamental concepts from Inductive inference: the relevance of the concept of VC-dimension seems to be closely related to the existence of probability distributions over the sample space. Though connections exists between PAC learning and Inductive inference (e.g., [5]), it does not seem that VC-dimension has any chance to play a key role in learning paradigms of Inductive inference that do not introduce probability distributions over the sample space. We will show that VC-dimension can still provide a perfect characterization for the problem of predicting whether a possible datum is true or false in the underlying world, in the realm of Inductive inference. But for such a characterization to be possible, the condition that the set of possible data is closed under boolean operators has to be imposed. The fact that we work in a logical setting is of course essential to express this condition in a simple, meaningful and natural way. The notions and main results are stated with no computability condition on the procedures that analyze the data and output hypotheses. This is necessary to obtain perfect equivalences, and provides strong evidence that the concepts involved are naturally connected. When computability is a requirement, the relationships become more complex. Our aim is to encourage the study of the connections between ordinal VC-dimension and predictive complexity in paradigms of Inductive inference. Obtaining perfect connections for ideal, unconstrained paradigms, suggests that further work in the same direction should be carried out, in the context of more realistic or constrained paradigms. Moreover, the results will be illustrated with examples that always involve effective procedures. We proceed as follows. We introduce some background notions and notation in Section 2. We define the various complexity measures in Section 3. We study the relationships between these complexity measures in Section 4. We conclude in Section 5.
2
Background
The class of ordinals is denoted by Ord. Let a set X be given. The set of finite sequences of members of X, including the empty sequence (), is represented by X . Given a σ ∈ X , the set of members of X that occur in σ is denoted by rng(σ). Given a finite or an infinite sequence σ of members of X and a natural number i that, is case σ is finite, is at most equal to the length of σ, we represent by σ|i the initial segment of σ of length i. Concatenation between sequences is represented by , and sequences consisting of a unique element are often identified with that element. We write ⊂ (respect., ⊆) for strict (respect., nonstrict) inclusion between sets, as well as for the notion of a finite sequence being a strict (respect., nonstrict) initial segment of another sequence. We also use the notation ⊃. Let two sets X, Y and a partial function f from X into Y be given. Given x ∈ X, we write f (x) =↓ when f (x) is defined, and f (x) =↑ otherwise. Given two members x, x of X, we write f (x) = f (x ) when both f (x) and f (x ) are defined and equal; we write f (x) = f (x ) otherwise. Let R be a binary relation over a set X. Recall that R is a well-founded iff every
56
E. Martin, A. Sharma, and F. Stephan
nonempty subset Y of X contains an element x such that no member y of Y satisfies R(y, x). Suppose that R is well founded. We then denote by ρR the unique function from X into Ord such that for all x ∈ X: ρR (x) = sup{ρR (y) + 1 : y ∈ X, R(y, x)}. The length of R, written as |R|, is the least ordinal not in the range of ρR . Note that |R| = 0 iff X = ∅. For example, Figure 1 depicts a finite binary relation R of length 5. In this diagram, an arrow joins a point x to a point y iff R(x, y) holds. For all points x in the field of R, the value of ρR (x) is indicated.
4 3
3
3
2
2 0
1 0
1
0
0
Fig. 1. A finite binary relation of length 5
Let us introduce the logical learning paradigms and their constituents. We denote by V a countable vocabulary, i.e., a countable set of function symbols (possibly including constants) and predicate symbols. Let us adopt a convention. If we say that V contains 0 and s, then 0 denotes a constant and s a unary function symbol. Moreover, given a nonnull n ∈ N, n is used as an abbreviation for the term obtained from 0 by n applications of s to 0. We denote by D a nonempty set of first-order sentences (closed formulas) over V that represent data. Three important cases are sets of closed atoms (to model learning from texts), sets of closed literals (i.e., closed atoms and their negations, to model learning from informants), and sets of sentences closed under boolean operators, a natural example being the set of quantifier free sentences. Note that quantifier free sentences convey no more information than closed literals. Still, the assumption that D is closed under boolean operators will play a key role in this paper. We denote by some symbol to be used when no datum is presented. Given a member σ of (D ∪ {}) , we set cnt(σ) = rng(σ) ∩ D. We denote by W a nonempty set of structures over V. An important case is given by the set of all Herbrand structures, i.e., structures over V each of whose individuals interprets a unique closed term. (When we consider Herbrand structures, V contains at least one constant). Given a member M of W and a set E
On Ordinal VC-Dimension and Some Notions of Complexity
57
of formulas, the E-diagram of M, denoted DiagE (M), is the set of all members of E that are true in M. We say that a set T of first-order formulas is consistent in W iff T has a model in W. Given a set T of first-order formulas over V, we denote by Mod(T ) the set of models of T , and by ModW (T ) the set of models of T in W (i.e., ModW (T ) = Mod(T ) ∩ W). We denote by P the triple (V, D, W). We call P a logical paradigm. This is a simplification of the notion of logical paradigm investigated in [9,10]. Learning paradigms in the numerical setting are naturally cast into the logical setting as follows. Set V = {0, s, P } where P is a unary predicate symbol. Let E be the set {P (n) : n ∈ N}. If C is the set of languages to be learnt, we define W as the set of Herbrand structures whose E-diagrams are {P (n) : n ∈ L} where L varies over C. The choice of D depends on the type of data: D = E when data are positive, D = E ∪ {¬ψ : ψ ∈ E} when data are positive or negative.
3
The Notions of Complexity
We now define the various concepts of complexity we need in this paper, starting with VC-dimension. The notion of a set of hypothesis shattering a set of data takes the following form when hypotheses are represented as structures and data as formulas. Definition 1. Let a set E of formulas be given. We say that W shatters E iff E is finite and for all subsets D of E, DiagE (M) = D for some M ∈ W. Traditionally, the VC-dimension is defined as the greatest number n such that some set consisting of n distinct elements is shattered [13]. When such an n does not exist, the VC-dimension is considered to be either undefined or infinite. We extend the notion of VC-dimension from natural numbers to ordinals as follows. Definition 2. Let X be the set of nonempty subsets of D that W shatters. Let R be the restriction of ⊃ to X. The VC-dimension of P is equal to the length of R if R is well-founded, and undefined otherwise. As an intuitive interpretation, the VC-dimension of P is determined by the following game, where we assume for simplicity that D is infinite. Consider two players Anke and Boris. Anke has to output an increasing sequence of nonempty finite subsets of D and Boris a decreasing sequence of ordinals. – In round 1, Anke outputs a nonempty finite subset of D and Boris outputs α. – If in the n-th round Anke has output a finite set D ⊆ D and Boris has output a nonnull ordinal β, then the following is done in round n + 1. • Anke outputs a finite set E with D ⊂ E ⊆ D. • Boris outputs an ordinal strictly below β. The game terminates after Boris has output 0. If Anke’s last set is shattered, Anke wins the game. Otherwise Boris wins the game. The VC-dimension of P
58
E. Martin, A. Sharma, and F. Stephan
is the smallest nonnull ordinal α for which Boris has a winning-strategy. If this ordinal does not exist, the VC-dimension is undefined. Given a structure M, we call environment (in P) an infinite sequence e of members of D ∪ {} such that for all ϕ ∈ D, ϕ occurs in e iff ϕ ∈ DiagD (M). So environments correspond to texts when D is the set of closed atoms, and to informants when D is the set of closed literals. Identification in the limit and the corresponding notion of complexity are defined next.
Definition 3. An identifier (for P) is a partial function from (D ∪ {}) into {DiagD (M) : M ∈ W}. Let an identifier f be given.
– We say that a member σ of (D ∪ {}) is consistent in W just in case there exists M ∈ W such that for all ϕ ∈ D that occur in σ, M |= ϕ. – We say that f is is successful (in P) iff for every M ∈ W and for every environment e for M, f (e|k ) = DiagD (M) for almost every k ∈ N. – Let X be the set of all σ ∈ (D ∪ {}) such that σ is consistent in W and f (τ ) =↓ for some initial segment τ of σ. We denote by Rf the binary relation over X such that for all σ, τ ∈ X, Rf (σ, τ ) holds iff τ ⊂ σ and f (σ) = f (τ ). The identification complexity of P is the least ordinal of the form |Rf |, where f is an identifier that is successful in P and Rf is well-founded; if such an f does not exist, the identification complexity of P is undefined. If the identification complexity of P is equal to nonnull ordinal β, then the D-diagrams of the members of W are identifiable in the limit with less than β mind changes. (Note that the usual notion of mind change complexity considers the least ordinal β such that at most, rather than less than, β mind changes are sometimes necessary for the procedure to converge [1,2,7]. There are good theoretical reasons for preferring the ‘less than’ formulation.) Note that if some identifier is successful in P, then there are countably many D-diagrams of members of W only. The following is a characterization of identification complexity based on a generalization Angluin’s finite telltale characterization of learnability in the limit [3]. Proposition 4. The identification complexity of P is defined and equal to nonnull ordinal β iff there exists a sequence (βM )M∈W of ordinals smaller than β and for all M ∈ W, there exists a finite AM ⊆ DiagD (M) such that for all N ∈ W: (∗)
if AM ⊆ DiagD (N) and βM ≤ βN then DiagD (N) = DiagD (M).
The next notion of complexity we define is based on selectors. Intuitively, a selector for P is a procedure that, for any given M ∈ W, selects in the limit all formulas from DiagD (M) or their negations, and is correct cofinitely many times. In order to be able to achieve this, the selector needs to receive for every selected formula ϕ, a feedback whether ϕ is or is not true in M. Thus one can represent the selector as a partial function where at stage n the input is from {0, 1}n and represents the feed-back on the previous selections.
On Ordinal VC-Dimension and Some Notions of Complexity
59
Definition 5. A selector (for P) is a partial function f from {0, 1} into D. Let a selector f be given.
– We say that a member σ of {0, 1} is consistent with W and f just in case there exists M ∈ W such that for all τ ∈ {0, 1} and i ∈ {0, 1} with τ i ⊆ σ, f (τ ) =↓, and M |= f (τ ) iff i = 1. – We say that f is is successful (in P) iff for every M ∈ W, there is a string t of finitely many 0’s and infinitely many 1’s such that every finite initial segment of t is consistent with W and f and for all ϕ ∈ DiagD (M), either f (σ) = ϕ or f (σ) = ¬ϕ for some finite initial segment σ of t. – Let X be the set of all σ ∈ {0, 1} that are consistent with W and f , and that end with a 0. We denote by Rf the binary relation over X such that for all σ, τ ∈ X, Rf (σ, τ ) holds iff τ ⊂ σ. The selective complexity of P is the least ordinal of the form |Rf |, where f is a selector that is successful in P and Rf is well-founded; if such a f does not exist, the selective complexity of P is undefined. The last notion of complexity we define is based on predictors. Whereas selectors have control over the formulas the truth of which they want to predict, a predictor has to take the members of D as they come.
Definition 6. A predictor (for P) is a partial function f from (D × {0, 1}) × D into {0, 1}. Let a predictor f be given.
– Let σ = ((ϕ0 , 0 ), . . . , (ϕp , p )) ∈ (D × {0, 1}) be given. We say that σ is consistent with W and f iff there exists M ∈ W such that for all i ≤ p: • f (σ|i , ϕi ) =↓, and • M |= ϕi iff i = 1 If σ is consistent with W and f , we call the number of i ≤ p such that f (σ|i , ϕi ) = i , the number of mispredictions that f makes on σ (in P). – We say that f is successful (in P) iff for every member M of W and every t ∈ (D×{0, 1})N , the following holds. Assume that every finite initial segment of t is consistent with W and f . Then there exists n ∈ N such that for all finite initial segments σ of t, f makes at most n mispredictions on σ. – Let X be the set of all σ ∈ (D × {0, 1}) that are consistent with W and f , and on which f makes at least one misprediction. We denote by Rf the binary relation over X such that for all σ, τ ∈ X, Rf (σ, τ ) holds iff τ ⊂ σ and f makes more mispredictions on σ than on τ . The predictive complexity of P is the least ordinal of the form |Rf |, where f is a predictor that is successful in P and Rf is well-founded; if such an f does not exist, the predictive complexity of P is undefined. Clearly, if the predictive complexity of P is defined, then the selective complexity of P is defined, and at most equal to the former. The three notions of complexity we have introduced can, similarly to VC-dimension, be interpreted in terms of the outcome of a game between Anke and Boris. For instance, for predictive complexity, Anke selects formulas from D and Boris has to make a
60
E. Martin, A. Sharma, and F. Stephan
prediction whether the formula holds (in the unknown world M) or not. Anke tells Boris whether the prediction is correct. If not, Boris has to count down an ordinal counter. The predictive complexity is then the least ordinal to start with for which Boris has a winning strategy.
4
Relationships between the Complexity Measures
Let us first introduce an alternative definition of the selective complexity, followed by a definition and a lemma, that are just technical tools for proving some of the main propositions. Notation 7. Let α ∈ Ord be given, and suppose that Γβ has been defined for all β < α. Let Γα be the set of all subsets U of W such that for all ϕ ∈ D, U ∩ Mod(ϕ) or U ∩ Mod(¬ϕ) belongs to β<α Γβ ∪ {∅}. Property 8. The predictive complexity of P is defined iff W ∈ this case it is equal to the least ordinal α with W ∈ Γα .
α∈Ord
Γα ; in
Definition 9. A shatterer (for P) is a sequence (Yα )α∈Ord of sets of subsets of D with the following properties. – W shatters all members of α∈Ord Yα ; – for all α, β ∈ Ord with β ≤ α and for all D, E ⊆ D with D ⊆ E, if E ∈ Yα then D ∈ Yβ ; – for all α, β ∈ Ord with β < α and for all E ∈ Yα , Yβ contains a proper superset of E. Lemma 10. Let a sequence (Mα )α∈Ord be inductively defined as follows. For all α ∈ Ord, Mα is the set of all finite subsets D of D such that W shatters D and for all β < α, Mβ contains a proper superset of D. – (Mα )α∈Ord is a shatterer. – For all shatterers (Yα )α∈Ord and α ∈ Ord, Yα ⊆ Mα . – There exists a least ordinal α with Mα = Mα+1 ∪ {∅}; the VC-dimension of P is defined and equal to α if Mα = {∅}, and undefined otherwise. We are now in a position to state and prove one of the main results of the paper, which relates VC-dimension to predictive complexity. Proposition 11. Suppose that D is closed under boolean operators. Then the VC-dimension of P is defined iff the predictive complexity of P is defined; moreover, if they are defined then they are equal.
On Ordinal VC-Dimension and Some Notions of Complexity
61
Proof. Given α ∈ Ord, let Γα denote the set of subsets of W as defined in Notation 7. For all α ∈ Ord, set Zα = {U ⊆ W : U ∈ / β<α Γβ ∪ {∅}}. For all ordinals α, let Yα be the set of all finite subsets E of D such that for all D ⊆ E, {M ∈ W : DiagE (M) = D} ∈ Zα . We show that (Yα )α∈Ord is a shatterer. Using the facts that ∅ ∈ / Zα and Γβ ⊆ Γα for all α, β ∈ Ord with β ≤ α, it is immediately verified that the first two conditions in Definition 9 are satisfied. For the third condition, let α, β ∈ Ord with β < α and E ∈ Yα be given. Given D ⊆ E, set ϕD = D ∧ ¬ (E \ D). By the definition of Yα , ModW (ϕD ) ∈ Zα . Moreover, ModW (ϕD ) ∈ / Γβ . Hence there exists ψD∈ D such that ModW (ϕD ∧ ψD ) ∈ Zβ and ModW (ϕD ∧ ¬ψD ) ∈ Zβ . Set ϕ = {ϕD ∧ ψD : D ⊆ E}. It is immediately verified that E ∪ {ϕ} ∈ Yβ . Assume that the predictive complexity of P is undefined. Let an ordinal α be given. Then W ∈ / Γα , hence W ∈ Zα+1 , hence ∅ ∈ Yα+1 . This implies that Yα contains a non-empty set, and by Lemma 10, the VC-dimension of P is not equal to α. It follows that the VC-dimension of P is undefined. Assume that the predictive complexity of P is defined and equal to ordinal γ. So W ∈ Γγ ∩ Zγ . Let ϕ ∈ D be given, and suppose that {ϕ} ∈ Yγ . Then both ModW (ϕ) and ModW (¬ϕ) are members of Zγ , which is in contradiction with W ∈ Γγ . Hence Yγ = {∅}, and by Lemma 10 again, the VC-dimension of P is at least equal to γ. It is easily verified that the VC-dimension of P is defined and at most equal to γ, which completes the proof of the proposition. The assumption that D is closed under negation is essential in Proposition 11, as shown in the next result. Proposition 12. For every nonnull n ∈ N, there exists a finite vocabulary V and a finite set B of formulas over V with the following property. Suppose that V = V , W is the set of Herbrand models of B, and D is the set of positive quantifier-free sentences. Then the VC-dimension of P is equal to 1 and the predictive complexity of P is equal to ω × n. Proof. Let a nonnull n ∈ N be given. Let V consist of a unary function symbol s, a binary predicate symbol <, an n-ary predicate symbol P and equality. Let B consist of the following formulas, which express that P is a predicate on Nn that is downward closed for the lexicographic ordering. – ∀x(x < s(x)) ∧ ∀xyz((x < y ∧ y < z) → (x < z)) ∧ ∀xy(x < y → ¬(y < x)); – ∀x1 . . . xn y1 . . . yn (( i
62
E. Martin, A. Sharma, and F. Stephan
Proposition 13. Assume that D is closed under negation and that W is compact. If the identification complexity of P is defined and equal to ordinal α, then the predictive complexity of P is defined and at most equal to ω × α. Proof. Suppose that the identification complexity of P is defined and equal to ordinal α. Choose a canonical identifier f for P. Let a predictor h be defined as follows. Given a member σ = ((ϕ1 , 1 ), . . . , (ϕn , n )) of (D × {0, 1}) , put σ = (ψ1 , . . . , ψn ) where for all nonnull i ≤ n, ψi = ϕi if i = 1 and ψi = ¬ϕi oth σ )) ⊆ Mod(ϕ) erwise. Let σ ∈ (D × {0, 1}) and ϕ ∈ D be given. If ModW (rng( then h(σ, ϕ) = 1. If ModW (rng( σ )) ⊆ Mod(¬ϕ) then h(σ, ϕ) = 0. If neither ModW (rng( σ )) ⊆ Mod(ϕ) nor ModW (rng( σ )) ⊆ Mod(¬ϕ), and if f ( σ ) is defined and is a model of ϕ then h(σ, ϕ) = 1. Otherwise h(σ, ϕ) = 0. It is immediately verified that h is successful on P. The proof of the proposition is completed if we show that the length of Rh is smaller than ω ×α. Let a member of the field of Rh of the form σ (ϕ, 0) be given. Let X be the set of all τ ∈ (D ∪ {}) such that rng( σ ) ⊆ rng(τ ), {ϕ, ¬ϕ}∩rng(τ ) = ∅, τ is consistent in W, and f (τ ) =↓. By the choice of f , and since the restriction of W to ModW σ )) is compact, there (rng( exists a finite F ⊆ X such that ModW (rng( σ )) = {ModW (rng(τ )) : τ ∈ F }. Let n be the sum of the lengths of the members of F . It is easy to verify that ρRh (σ (ϕ, 0)) ≤ ω × supτ ∈F ρRf (τ ) + n. This implies immediately that Rh is at most equal to ω × α, as wanted. When D is closed under negation, identification and selective complexities are very close concepts, as shown next. Proposition 14. Suppose that D is closed under negation. If the selective complexity of P is defined and equal to ordinal α, then the identification complexity of P is defined and at most equal to α + 1. Proof. Let a selector g be successful in P and such that the length of Rg is defined and equal to ordinal α. Without loss of generality, we can assume that for all σ, τ ∈ {0, 1} , if σ ⊂ τ then g(σ) = g(τ ). Let an identifier f be defined as follows. Let σ = (ϕ0 , . . . , ϕn ) ∈ (D ∪ {}) be such that cnt(σ) is consistent in W. We define a p ∈ N, a member σ of {0, 1} and a finite sequence (ψi )i
On Ordinal VC-Dimension and Some Notions of Complexity
63
Proposition 15. Suppose that D is closed under boolean operators, W is countable and W is compact. Then there exists ordinals α, β such that: – – – – –
ω × α ≤ β ≤ ω × (α + 1); the VC-dimension of P is equal to β; the predictive complexity of P is equal to β; the identification complexity of P is equal to α + 1; the selective complexity of P is equal to α or α + 1.
Proof. We first show that the VC-dimension of P is defined, which together with Propositions 11 and 14, implies immediately that the predictive complexity of P, the identification complexity of P, and the selective complexity of P are also defined. Suppose for a contradiction that there exists an infinite subset X of D such that for all finite E ⊆ X and D ⊆ E, D = DiagE (M) for some M ∈ W. Since W is compact, it follows that for all Y ⊆ X, Y = DiagX (M) for some M ∈ W, which contradicts the assumption that W is countable. Thus every infinite subset of D has a finite subset that W does not shatter, and the VCdimension of P is defined, as wanted. We now show that the identification complexity of P is a successor ordinal. Choose a canonical identifier f for P. It suffices to show that the length of Rf is not a limit ordinal. Suppose otherwise for a contradiction. Let X be the set of all finite subsets D of D such that f (D) =↓. Since W is compact and f is successful in P, there exists a finite subset F of X such that W = {ModW (D) : D ∈ F }. This implies that |Rf | is equal to sup(ρRf (σ) + 1 : D ∈ F ), hence is smaller than |Rf | since F is finite and |Rf | is a limit ordinal. Contradiction. Denote by α + 1 the identification complexity of P, and by β the predictive complexity of P. By Proposition 11, the VC-dimension of P is equal to β. By Proposition 14, the selective complexity of P is at least equal to α. We show that the selective complexity of P is at most equal to α + 1. Choose a set-driven identifier f such that f is successful in P and the length of Rf is defined and equal to ordinal α + 1. Fix an enumeration (ϕi )i∈N of D. We define by induction a sequence (hi )i∈N of selectors with finite domains such that for all i, j ∈ N, hi and hj have disjoint domains. Let i ∈ N be given, and suppose that hj has been defined for all j < i. If i = 0 put Z = {()}. If i = 0, let Z be the set of all ⊆-maximal members of the domain of hi−1 . Let σ ∈ Z be given. If the σ = () let σ = σ. If σ is nonempty and equal to ( 0 , . . . , k ), denote by σ sequence (ψ0 , . . . , ψk ) of members of D such that for all j ≤ k, ψj = f (σ|j ) if j = 1, and ψj = ¬f (σ|j ) otherwise. If ModW (rng( σ )) ⊆ Mod(ϕi ) then set σ )) ⊆ Mod(¬ϕi ) then set hi (σ) = ¬ϕi . Suppose that hi (σ) = ϕi . If ModW (rng( neither ModW (rng( σ )) ⊆ Mod(ϕi ) nor ModW (rng( σ )) ⊆ Mod(¬ϕi ). Let Xσ be the set of all τ ∈ (D ∪ {}) such that rng( σ ) ⊆ rng(τ ), {ϕi , ¬ϕi } ∩ rng(τ ) = ∅, τ is consistent in W, and f (τ ) =↓. By the choice of f , and since the restriction of W to ModW (rng( σ )) is compact, there exists a finite Fσ ⊆ Xσ such that ModW (rng( σ )) = {ModW (rng(τ )) : τ ∈ Fσ }. Without loss of generality, we can assume that there exists ψ0 , . . . , ψk ∈ D such that for all for all τ ∈ Fσ and for all j ≤ k, either ψj or ¬ψj occurs in τ , and every formula that occurs in τ is of the form ψj or ¬ψj for some j ≤ k. Let n be the cardinality of Fσ ,
64
E. Martin, A. Sharma, and F. Stephan
and fix an enumeration (Dp )p
Let σ ∈ (D ∪ {}) be such that f (σ) =↓, and let γ be the least ordinal with ModW (cnt(σ)) ∈ Γγ . Suppose for a contradiction that γ is neither 0 nor a limit ordinal. It then follows from the definition of Γγ that there exists ϕ ∈ D such that Γγ−1 contains neither ModW (cnt(σ) ∪ {ϕ}) nor ModW (cnt(σ) ∪ {¬ϕ}), which is impossible by the definition of f . Let λ be the number of limit ordinals less than or equal to β. We infer easily from the previous remark that α + 1 ≤ λ + 1, hence ω × α ≤ ω × λ ≤ β, and we are done. We illustrate the previous results, especially the bounds that have been obtained, with a few examples. Example 16. Let V consists of 0, s, and a unary predicate symbol P . Let W be the set of Herbrand models of ∀x(P (x) ∧ P (s(x)) → P (s(s(x)))) ∧ ∃y(P (y) ∧ P (s(y))). Then the identification complexity of P is 1 but the predictive complexity of P is undefined. Example 17. Assume that D is closed under Boolean operators. If the VCdimension of P is a nonnull n ∈ N then both the identification and the selective complexity of P are 1, and the predictive complexity of P is n. Proposition 18. Suppose that V consists of 0, s, and a unary predicate symbol P . Set D = {P (n) : n ∈ N}. Assume that W is the set of Herbrand structures whose D-diagrams are {nk : n ∈ N} for k ∈ N. Then the identification complexity of P is 2, and both the VC-dimension and the predictive complexity of P are ω. Proof. It is easily verified that an identifier can be successful in P by first hypothesizing {P (0)}, and in the case of an environment for the structure whose D-diagram is T = {P (nk) : n ∈ N} for some nonnull k ∈ N, by eventually changing {P (0)} to T . If follows that the identification complexity of P is 2. It is
On Ordinal VC-Dimension and Some Notions of Complexity
65
easily verified that the VC-dimension and the predictive complexity of P are at most equal to ω. To see that they are at least equal to ω, let a nonnull n ∈ N be given, and let p0 , p1 , . . . , p2n −1 be an enumeration of the first 2n prime numbers. For all k < n, let qk be the product of all pi , 0 ≤ i < 2n , such that the (k + 1)st bit in the binary representation of i is equal to 1. It is immediately verified that W shatters {P (q0 ), P (q1 ), . . . , P (qn−1 )}, implying that the VC-dimension of P is at least equal to n, hence the predictive complexity of P is also at least equal to n. Example 19. Suppose that V consists of 0, s, and a unary predicate symbol P . Assume that W is the set of Herbrand structures M such that for all n ∈ N, if M |= P (n) then there exists at most n natural numbers m such that M |= P (m). Suppose that D is the set of quantifier free sentences. Then the identification complexity of P is ω+1, the selective complexity of P is ω, and the VC-dimension of P is ω 2 . Note that if in the previous example, we remove from W the structure M such that M |= ∀x¬P (x)—resulting in a noncompact topological space W—, then the identification complexity becomes ω; but in both cases, the traditional notion of mind change complexity is ω. Proposition 20. Let a countable ordinal α be given. There exists a finite vocabulary V and two sets K and E of sentences over V such that if V = V , D = E, and W is the set of Herbrand models of K, then the identification complexity of P is α + 1, the selective complexity of P is α, and both the VC-dimension and the predictive complexity of P are ω × α. Proof. Suppose that V consists of a constant, a unary function symbol, a unary predicate symbol P , and a binary predicate symbol ≤. Identify the set X of closed terms with the set of ordinals smaller than α, writing X = {cβ : β < α}. Let D be the closure of {P (cβ ) : β < α} under boolean operators. Suppose that W is the set of Herbrand models of {(cβ , cγ ) : β, γ < α, β ≤ γ} ∪ {∀x∀y((P (x) ∧ x ≤ y) → P (y))}.
We define by induction, for all members σ of (D ∪ {}) , three ordinals: counterσ , aboveσ , and belowσ . Set counter() = α, above() = ω α and below() = 0. Let σ ∈ (D ∪ {}) and x ∈ D ∪ {} be given, and suppose that counterσ , aboveσ and belowσ have been defined. Let aboveσx be the minimal ordinal β such that cβ occurs in σ x if such a β exists, and let aboveσx be ω α otherwise. Let belowσx be the maximal ordinal β such that cβ occurs in σ x if such a β exists, and let belowσx be 0 otherwise. If there exist (unique) ordinals β, γ such that β < counterσ , aboveσ = ω β × γ and belowσ + ω β ≥ aboveσ , then set counterσx = β; otherwise set counterσx = counterσ . Let f be the identifier defined as follows. Let σ ∈ (D ∪ {}) be given. If counterσ = ω α then f (σ) is the D-diagram of the member M of W such that M |= ∀x¬P (x); otherwise, f (σ) is the D-diagram of the member M of W such that M |= ∀x(P (x) ↔ x ≥ caboveσ ).
66
E. Martin, A. Sharma, and F. Stephan
It is immediately verified that f is successful in P, and that the length of Rf is defined and equal to α + 1. It is also easily verified that the identification complexity of P cannot be smaller than α + 1, and that the selective complexity of P is α. Since W is a compact topological space, it follows from Propositions 11 and 15 that the VC-dimension and the predictive complexity of P are equal, and at most equal to ω × α. So to complete the proof of the proposition, it suffices to show that the VC-dimension of P is at least equal to ω × α. Suppose for a contraction that the VC-dimension of P is equal to ordinal α with α < α. The argument uses the presentation of the VC-dimension as a game between Anke and Boris. At round n, Anke defines a set of ordinals On and a formula ψn , then outputs a set En , before Boris outputs an ordinal βn . Set O0 = {0}, ψ0 = P (cα ), E0 = {ψ0 }, and β0 = α . Let n ∈ N be given and suppose that On , ψn , En and βn have been defined. Let ordinal γ and m ∈ N be such that βn = ω × γ + m. Set On+1 = On ∪ {δ + ω γ + 2m : δ ∈ On }. Let ψn+1 be a formula expressing that the cardinality of the members x of On+1 that have property P is odd. Let En+1 = {ψ0 . . . , ψn+1 }. Note that On+1 expands On with exactly one ordinal between two ordinals in On with no ordinal in On in between, plus an ordinal greater than all ordinals in On . It is easily verified that for all n ∈ N, W shatters En . Hence Anke wins the game, contradiction. If the previous example were modified by taking {P (cβ ) : β < α} as set of possible data, resulting in a new paradigm P , then the identification complexity of P would clearly be equal to ω α + 1. More generally, there exists a relationship between the identification complexities of two paradigms that differ in their sets of possible data—one set of data being closed under negation in one paradigm, and not in the other. This relationship is considered in the next proposition, and the previous example shows that the “exponential” difference between the identification complexities of both paradigms is almost as large as it can be. First note that Angluin’s finite telltale condition takes the following form in the logical framework. Lemma 21. Some identifier is successful in P iff for all M ∈ W, there exists a finite A ⊆ DiagD (M) such that no N ∈ W satisfies A ⊆ DiagD (N) ⊂ DiagD (M). Proposition 22. Suppose that D is the closure under negation of some set of sentences D , and set P = (V, D , W). Assume that W is compact, the identification complexity of P is defined and equal to nonnull ordinal α, and some identifier is successful in P . Then the identification complexity of P is defined and smaller than ω α . Proof. The proof is trivial if {DiagD (M) : M ∈ W} is finite, so suppose otherwise. Let (ψi )i∈N be a repetition free enumeration of D . Let X be the set of finite sequences of the form (ϕ0 , . . . , ϕn−1 ), n ∈ N, where for all i < n, ϕi = ψi or ϕi = ¬ψi , such that {ϕ0 , . . . , ϕn−1 } is consistent in W. Let f be a canonical identifier for P. Let a finite subset E of D be consistent in W. Denote by UE the set of all ⊆-minimal members σ of X such that f (σ) is defined and contains E.
On Ordinal VC-Dimension and Some Notions of Complexity
67
Suppose for a contradiction that UE is infinite. By K¨ onig’s lemma, there exists a sequence (ξ)i∈N of formulas such that for all n ∈ N, (ξ0 , . . . , ξn−1 ) = σ|n for some σ ∈ UE . Note that E ⊆ rng(σ) for cofinitely many members σ of UE , hence E ⊆ {ξi : i ∈ N}. Moreover, for all distinct σ, τ ∈ UE , neither σ ⊆ τ nor τ ⊆ σ, hence σ ⊂ {ξi : i ∈ N} for cofinitely many members σ of UE . Hence f ((ξ0 , . . . , ξn )) is either undefined or does not contain E for infinitely many n ∈ N. By compactness of W, {ξi }i∈N is the D-diagram of a member of W. But this is in contradiction with the fact that f is successful in P. So we have verified that UE is finite. Note that since f is successful in P, ModW (E) is included in {ModW (cnt(σ)) : σ ∈ UE }. Let (σ1 , . . . , σp ) be a repetition free enumeration of UE with ρRf (σ1 ) ≥ . . . ≥ ρRf (σp ). Set βE = ω ρRf (σ1 ) + . . . + ω ρRf (σp ) . Let finite E, F ⊆ D be such that E ⊆ F , UF \ UE = ∅, and UF is nonempty. Let τ ∈ UF \ UE be given. Then there exists i ∈ {1, . . . , p} such that σi ⊂ τ , / UF . Let D be the set of all members of UF that strictly f (σi ) = f (τ ), and σi ∈ extend σi . Clearly, ρRf (σi ) > ρRf (γ) for all γ ∈ D. Moreover, the factor ω ρRf (σi ) in the sum βE is replaced in the sum βF by the factors ω ρRf (γ) , with γ varying over D. Due to the powers of ω, it is easily verified that the extra factors ω ρRf (γ) , γ ∈ D, ‘weight less’ than ω ρRf (σi ) . Since every σi ∈ UE is either also in UF or replaced in EF by a D with the properties just described, we conclude that βE > βF . Fix a repetition free enumeration (Ti )i∈N of {DiagD (M) : M ∈ W}. By Lemma 21, choose for all i ∈ N a finite subset Ai of Ti such that no member M of W satisfies Ai ⊆ DiagD (M) ⊂ Ti . Let Y be the set of all σ ∈ (D ∪ {}) such that Ai ⊆ cnt(τ ) ⊆ Ti for some i ∈ N and some initial segment τ of σ. For all σ ∈ Y , denote by σ the ⊆-maximal initial segment of σ such that Ai ⊆ cnt( σ ) ⊆ Ti for some i ∈ N. Now define an identifier f for P as follows. / Y then f (σ) =↑; otherwise, f (σ) = Ti Let σ ∈ (D ∪ {}) be given. If σ ∈ where i ∈ N is least with Ai ⊆ cnt( σ ) ⊆ Ti . Let σ, τ ∈ (D ∪ {}) be such that σ ⊂ τ , and f (σ) and f (τ ) are defined but distinct. Then both σ and τ belong to Y . Clearly, there exists i ∈ N such that cnt(σ) ⊆ Ti but cnt(τ ) ⊆ Ti , which implies that Ti ∈ {ModW (cnt(γ)) : γ ∈ Ucnt(τ ) } \ {ModW (cnt(γ)) : γ ∈ Ucnt(σ) }. We infer that Ucnt(τ ) is nonempty and distinct from Ucnt(σ) , which we know implies that βcnt(σ) > βcnt(τ ) . It then follows that the height of Rf is defined and at most equal to ω α . Moreover, since W is compact, the reasoning in Proposition 15 shows that the identification complexity of P is not a limit ordinal, hence is smaller than ω α , as wanted. It was shown in [11] that every class of languages that is finitely identifiable from informants is also identifiable in the limit from texts. But such a relationship does not generalize to identifiability from informants with one mind change at most. Indeed, the class C consisting of N and all finite initial segments of N is not learnable in the limit from texts, as proved in [8], whereas C is clearly learnable from informants with one mind change at most. Cast into the logical framework,
68
E. Martin, A. Sharma, and F. Stephan
this provides an example of V, W, D, and D such that D is the closure of D under negation, W is compact, the identification complexity of P is equal to 1, but no identifier is successful in P = (V, D , W). This shows that Proposition 22 does not hold if the assumption that some identifier is successful in P is dropped.
5
Conclusion
In ideals paradigms of Inductive inference, finite tell tails conditions offer characterizations of identification in the limit or of classification, with or without a mind change bound. Assuming that the set of data is closed under boolean operators, the VC-dimension offers a characterization of prediction. An extra topological assumption of compactness enables to provide a complete picture of the relationship between VC-dimension and other notions of complexity, including mind change bound complexity.
References 1. Ambainis, A., Jain, S., Sharma, A.: Ordinal mind change complexity of language identification. Theoretical Computer Science. 220(2) pp. 323–343 (1999) 2. Ambainis, A., Freivalds, R., Smith, C.: Inductive Inference with Procrastination: Back to Definitions. Fundamenta Informaticae. 40 pp. 1–16 (1999) 3. Angluin, D. Inductive Inference of Formal Languages from Positive Data. Information and Control 45 p. 117–135 (1980) 4. Ben-David, S., Gurvits, L.: A note on VC-Dimension and Measure of Sets of Reals. Combinatorics Probability and Computing 9, 391–405 (2000) 5. Ben-David, S., Jacovi, M.: On Learning in the Limit and Non-Uniform (, δ)Learning. In Proceedings of the Sixth Conference on Computational Learning Theory. ACM Press pp. 209–217 (1993) 6. Blumer, A., Ehrenfeucht, A., Haussler, D., Warmuth, M.: Learnability and the Vapnik-Chervonenkis Dimension. J. ACM 36(4) pp. 929–965 (1989) 7. Freivalds, R., Smith, C.: On the role of procrastination for machine learning. Inform. Comput. 107(2) pp. 237–271 (1993) 8. Gold, E.: Language Identification in the Limit. Information and Control. 10 (1967) 9. Martin, E., Sharma, A., Stephan, F.: A General Theory of Deduction, Induction, and Learning. In Jantke, K., Shinohara, A.: Proceedings of the Fourth International Conference on Discovery Science. Springer-Verlag, LNAI 2226 pp. 228–242 (2001) 10. Martin, E., Sharma, A., Stephan, F.: Logic, Learning, and Topology in a Common Framework. In Cesa-Bianchi, N., Numao, M., Reischuk, R.: Proc. of the 13th Intern. Conf. on Alg. Learning Theory. Springer-Verlag, LNAI 2533 pp. 248–262 (2002) 11. Sharma, A.: A note on batch and incremental learnability. Journal of Computer and System Sciences 56 pp. 272–276 (1998) 12. Valiant, L.: A Theory of the Learnable. Commun. ACM 27(11) pp. 1134–1142 (1984) 13. Vapnik, V., Chervonenkis, A.: On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probabilities and its Applications 16(2) pp. 264–280 (1971)
Learning of Erasing Primitive Formal Systems from Positive Examples Jin Uemura and Masako Sato Department of Mathematics and Information Sciences Osaka Prefecture University, Sakai, Osaka 599-8531, Japan {jin, sato}@mi.cias.osakafu-u.ac.jp
Abstract. An elementary formal system, EFS for short, introduced by Smullyan is a kind of logic program over strings, and is regarded as a grammar to generate a language. Arikawa and his colleagues introduced some subclasses of EFSs which correspond to Chomsky hierarchy, and showed that they constitute a useful framework for language learning. This paper considers a subclass of EFSs, called primitive EFSs, in view of inductive inference in Gold framework from positive examples. Shinohara showed that the class of languages generated by primitive EFSs is inferable from positive examples, where ε substitutions, i.e., substitutions that may substitute the empty string for variables is not allowed. In the present paper, we consider allowing ε substitutions, and call such EFSs erasing EFSs. It is unknown whether or not the class of erasing pattern languages is learnable from positive examples. An erasing pattern language is a language generated by an erasing EFS with just one axiom. We first show that the class PFSL of languages generated by erasing primitive EFSs does not have finite elasticity, but has M-finite thickness. The notions of finite elasticity and M-finite thickness were introduced by Wright, and Moriyama and Sato, respectively, to present sufficient conditions for learnability from positive examples. Moriyama and Sato showed that a language class with M-finite thickness is learnable from positive examples if and only if for each language in the class, there is a finite tell-tale set of the language. Then we show the class PFSL is learnable from positive examples by presenting a finite tell-tale set for each language in the class.
1
Introduction
An elementary formal system, EFS for short, is a kind of logic program over strings consisting of finitely many axioms. A pattern is a finite string of constant symbols and variables. A pattern is regular, if each variable appears in the pattern at most once. In EFSs, patterns are used for terms in logic programming. For example, Γ = {p(ab) ←, p(axb) ← p(x)} is an EFS with two axioms, where p is a unary predicate symbol, a and b are constant symbols and x is a variable, where patterns ab and axb are used as terms. R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, pp. 69–83, 2003. c Springer-Verlag Berlin Heidelberg 2003
70
J. Uemura and M. Sato
An EFS generates a language of constant strings obtained by applying substitutions for variables and Modus Ponens to axioms in the EFS. In the above example, the language generated by Γ is L(Γ ) = {an bn | n ≥ 1}. The framework of EFSs was introduced by Smullyan [14] to develop his recursive function theory. Arikawa and his colleagues [2] introduced some subclasses of EFSs whose language classes correspond to Chomsky hierarchy. Among them, the class of length-bounded EFSs that generates the class of context-sensitive languages was especially investigated from a viewpoint of learning of languages from positive examples in Gold framework [4](Shinohara [13], Moriyama&Sato [5], Mukouchi&Arikawa [7], Sato [10]). Wright [16] introduced a notion of finite elasticity as a sufficient condition for learnability from positive examples. Shinohara [13] showed that the class of languages generated by length-bounded EFSs with at most n axioms has finite elasticity, where ε substitutions, i.e., substitutions that may substitute the empty string for variables are not allowed. In the present paper, we consider allowing ε substitutions, and call such EFSs erasing EFSs. As easily seen, the language generated by an erasing EFS is different from that generated by the nonerasing EFS with the same axioms. The language of an EFS Γ = {p(π) ←} is called an erasing or extended pattern language, if Γ is erasing, and nonerasing pattern language otherwise. It is well known that the class of nonerasing pattern languages is learnable from positive examples ([1]), but unknown for the class of erasing pattern languages. Note that it was shown that the latter class is not learnable from positive examples when the number of constant symbols is two ([9]). The present authors [15] proved compactness theorem of bounded unions of erasing regular pattern languages which plays an essential role in designing an efficient learning algorithm for unions. An EFS is regular, if each axiom in the EFS is of the form p(π) ← q1 (x1 ), · · · , qn (xn ), where π is a regular pattern, p, qi s are unary predicate symbols and x1 , · · · , xn are all of the mutually distinct variables appearing in π. The class of languages generated by erasing regular formal system, RFS for short, corresponds to that of context-free languages([2]). Recently Mukouchi [8] has showed that the class of languages generated by erasing RFSs with at most n axioms has finite elasticity, and so is learnable from positive examples, provided every regular pattern π in axioms is of restricted form called canonical, i.e., contains no successive variables. In this paper, we deal with a subclass of RFSs called primitive formal systems, PFSs for short, consisting of exactly two axioms of the forms p(π) ← and p(τ ) ← p(x1 ), · · · , p(xn ), where τ is possible to be noncanonical form, but imposed some condition when π = ε. The class of nonerasing PFSs was firstly introduced by Shinohara [12], and is a proper subclass of RFSs. Note that Mukouchi [8] has shown that the class of languages generated by erasing EFSs with two axioms where patterns in the axioms are not always regular is not learnable from positive examples under Σ ≥ 2.
Learning of Erasing Primitive Formal Systems from Positive Examples
71
Unfortunately, the class PFSL of languages generated by erasing PFSs is shown not to have finite elasticity. Moriyama and Sato [5] has introduced a notion of M-finite thickness, and showed that a class with M-finite thickness is learnable from positive examples if and only if for each language in the class, there is a finite tell-tale set of the language. As well as finite elasticity, M-finite thickness is known to have some good properties such as closure properties for various class operations, but M-finite thickness merely is not a sufficient condition for learnability ([10]). We first investigate the inclusion problem L(Γ ) ⊆ L(Γ ) for given PFSs Γ and Γ . Then, we show that the class of languages generated by PFSs has Mfinite thickness as well as the class of nonerasing length-bounded EFSs. Finally we show that the class PFSL is learnable from positive examples by showing that for each PFS Γ , there is a finite tell-tale set of the language generated by Γ.
2
Erasing PFS Languages
Let Σ be a finite alphabet, X be a countable set of variables, and Π be a set of predicate symbols. We assume these sets Σ, X, Π are mutually disjoint. Each predicate symbol is associated with a positive integer called arity. A pattern is a string (possibly empty string ε) on Σ ∪ X. An atom is an expression of the form p(π1 , · · · , πn ), where p is a predicate symbol with arity n and π1 , · · · , πn are patterns. A definite clause is a clause of the form A ← B1 , · · · , Bm (m ≥ 0), where A, B1 , · · · , Bm are atoms. The atom A is called the head and the part B1 , · · · , Bm the body of the definite clause. Definition 1. An elementary formal system, EFS for short, is a finite set of definite clauses. For an EFS Γ , each definite clause in Γ is called an axiom of Γ. A substitution is a homomorphism from patterns to patterns that maps each symbol a ∈ Σ to itself. We permit an ε substitution that can map some variables to the empty string. By πθ, we denote the image of a pattern π by a substitution θ. For an atom A = p(π1 , · · · , πn ) and a clause C = A ← B1 , · · · , Bm , we define Aθ = p(π1 θ, · · · , πn θ) and Cθ = Aθ ← B1 θ, · · · , Bm θ. A definite clause C is provable from an EFS Γ , denoted by Γ C, if C is obtained by finitely many (possibly 0) applications of substitutions and Modus Ponens. We define the language L(Γ, p) = {w ∈ Σ ∗ | Γ p(w)}, where p is a unary predicate symbol. Definition 2. An EFS Γ is a simple formal system, an SFS for short, if each clause in Γ is of the form p(π) ← q1 (x1 ), · · · , qn (xn ), where p, qi s are unary predicate symbols and x1 , · · · , xn are mutually distinct variables appearing in π. A pattern π is regular if each variable appears in π at most once. An SFS Γ is a regular formal system, an RFS for short, if all patterns in heads of clauses in
72
J. Uemura and M. Sato
Γ are regular. An RFS Γ is a primitive formal system, a PFS for short, if it contains exactly two clauses of the forms p(π) ←
and
p(τ ) ← p(x1 ), p(x2 ), · · · , p(xn ),
where x1 , x2 , · · · , xn are all of the variables appearing in τ . The former is called a base step and the latter an induction step of Γ . A language L is an erasing EFS (resp., SFS, RFS or PFS) language if L = L(Γ, p) for some EFS (resp., SFS, RFS or PFS) Γ and some unary predicate symbol. A language L is an erasing regular pattern language if L = L(Γ, p) for some RFS Γ = {p(π) ←}. It can be shown that the class of erasing RFS languages corresponds to that of context-free languages (Arikawa et al.[2]). Thus a PFS language is contextfree, but not always regular. In fact, the following PFS language is not regular. Example 1. Let us consider the PFS Γ = {p(ε) ←, p(axb) ← p(x)}, where Σ = {a, b}. Then as easily seen, L(Γ, p) = {an bn | n ≥ 0}. In this paper, we deal with the class of PFS languages. In what follows, we fix a unary predicate symbol, say p, and denote L(Γ, p) by L(Γ ) simply. Moreover, we denote a PFS Γ = {p(π) ←, p(τ ) ← p(x1 ), · · · , p(xn )} and L(Γ ), by Γ = (π, τ ) and L(π, τ ), respectively. Similarly by L(π) we denote L({p(π) ←}). For patterns π and τ , we introduce binary relations and ≡ as follows: π τ if π = τ θ for some substitution θ, and π ≡ τ if π τ and τ π. A renaming of variables of a substitution θ such that xθ ∈ X, and x = y implies xθ = yθ for any x, y ∈ X. In this paper, we identify patterns equivalent from each other by renaming. Thus ax1 bx2 = ax2 bx1 and ax1 bx2 ≡ ax1 x2 bx3 but ax1 bx2 = ax1 x2 bx3 . A pattern π is of canonical form if π ≡ τ implies |π| ≤ |τ | for any pattern τ , and π contains exactly n variables x1 , · · · , xn for some integer n and the leftmost occurrence of xi is to the left of the leftmost occurrence of xi+1 for each i, where |π| is the length of the pattern π. Lemma 1 (Shinohara [11]). Suppose that Σ ≥ 3. Let π and τ be regular patterns. Then (i) π τ ⇐⇒ L(π) ⊆ L(τ ),
(ii) π ≡ τ ⇐⇒ L(π) = L(τ ).
By the definition of L(π, τ ), it follows that L(π, τ ) = L(π) ∪ L , where L is the set of strings obtained by at least one applying the induction step of Γ . Concerning an erasing regular pattern language L(π), Shinohara [11] showed that there is a unique canonical pattern π equivalent to any regular pattern π. Clearly the canonical pattern π has the form of w0 x1 w1 x2 · · · wn−1 xn wn (w0 , wn ∈ Σ ∗ , wi ∈ Σ + (i = 1, · · · , n − 1), and L(π) = L(π ), provided that Σ ≥ 3. Hereafter, we assume that Σ ≥ 3 and the pattern π in a PFS Γ = (π, τ ) is assumed to be of canonical form. On the other hand,
Learning of Erasing Primitive Formal Systems from Positive Examples
73
concerning with τ in the induction step, we can not assume τ to be of canonical form. Indeed, let Γ = (aa, bx1 x2 b). As easily seen, b(aa)(aa)b ∈ L(Γ ). On the other hand, for Γ = (aa, bx1 b), we have b(aa)(aa)b ∈ L(Γ ). Thus L(Γ ) = L(Γ ). For Γ = (π, τ ), we define the following particular pattern τπ = τ {x := π | x appears in τ }, where variables substituted to the variables in τ are taken to be distinct, and so τπ is always regular. Then we have for every w ∈ L(Γ ), |w| ≥ min{|c(π)|, |c(τπ )|}, where c(π) is the string obtained from π by substituting the empty string to all variables. For Γ = (π, τ ), τ = x means L(Γ ) = L(π). That is, the induction step p(x) ← p(x) is redundant, and thus we assume τ = x. A string w ∈ Σ + is a multiple string of a string u, called a component, if w = ul for some l ≥ 2, and is a multiple string if there is a component of w. A component u for w is maximal if there is no component u for w satisfying |u | > |u|. We denote by PFS the sets of all PFSs except for PFSs Γ = (ε, τ ) such that τε is a multiple string. Definition 3. A PFS Γ is reduced if L(Γ ) L(Γ ) for any Γ Γ .
3
Inductive Inference
We first give the notion of identification in the limit from positive examples ([4]). A language class L = L0 , L1 , · · · over Σ is an indexed family of recursive languages if there is a computable function f : N × Σ ∗ → {0, 1} such that f (i, w) = 1 if w ∈ Li , otherwise 0, where N = {i | i ≥ 0}. The function f is called a membership function. Hereafter we confine ourselves to indexed families of recursive languages. An infinite sequence of strings w1 , w2 , · · · over Σ is a positive presentation of a language L, if L = {wn | n ≥ 1} holds. An inference machine is an effective procedure M that runs in stages 1, 2, · · · , and requests an example and produces a hypothesis in N based on the examples so far received. Let M be an inference machine and σ = w1 , w2 , · · · be an infinite sequence of strings. We denote by hn the hypothesis produced by M at stage n after the examples w1 , · · · , wn are fed to M . M on input σ converges to h if there is an integer n0 ∈ N such that hn = h for every n ≥ n0 . M identifies in the limit or infers a language L from positive examples, if for any positive presentation σ of L, M on input σ converges to h with L = Lh . A class of languages L is inferable from positive examples if there is an inference machine that infers any language in L from positive examples. Angluin [1] gave a characterizing theorem for a language class L to be inferable from positive examples if and only if there exists an effective procedure
74
J. Uemura and M. Sato
that enumerates for every language in L a set SL such that SL ⊆ L, SL is finite and SL ⊆ L for all L ∈ L with L L. The set SL is called a finite tell-tale set of L. Mukouchi [8] has showed that the class of SFS languages defined by at most two clauses is not inferable from positive examples by considering an infinite sequence of SFSs as follows: Γn = {p(x1 x2 · · · xn−1 xn xn xn−1 · · · x2 x1 ) ←},
(n ≥ 1).
We should note that the pattern x1 x2 · · · xn−1 xn xn xn−1 · · · x2 x1 is not regular, and thus the above ∞EFSs are not RFSs, but SFSs. They showed L(Γ1 ) L(Γ2 ) · · · L(Γ ) and n=1 L(Γn ) = L(Γ ) under Σ ≥ 2, where Γ = {p(x1 x1 ) ←,
p(x1 x2 x1 ) ← p(x2 )}.
It means that the language L(Γ ) does not have any finite tell-tale set within the class. Angluin [1] gave a very useful sufficient condition for inferability called finite thickness. The class of erasing regular pattern languages discussed in this paper was shown to have finite thickness by Shinohara [11] as well as nonerasing pattern languages (Angluin[1]). Wright [16] introduced another sufficient condition for inferability called finite elasticity more general than finite thickness ([6]). A class L has finite elasticity, if there is no infinite sequence of strings w0 , w1 , · · · and no infinite sequence of languages L1 , L2 , · · · in L satisfying {w0 , w1 , · · · , wn−1 } ⊆ Ln but wn ∈ Ln for every n ≥ 1. Finite elasticity has a good property in a sense that it is closed under various class operations such as union, intersection and so on (Wright[16], Moriyama & Sato [5], Sato [10]). Shinohara [13] proved that the class of nonerasing length-bounded EFS languages generated by at most k clauses has finite elasticity, and so is inferable from positive examples. Mukouchi [8] has showed that the class of erasing RFS languages generated by at most k clauses has finite elasticity similarly, provided that all regular patterns in heads of induction steps are of canonical forms. Without the condition of canonical form, the inferability, however, is not valid as shown below. Theorem 1. The class PFSL does not have finite elasticity. Proof. Define PFSs Γn = (ε, τn ) (n ≥ 1) as follows: τn = a(x1 x2 · · · xn )b for n ≥ 1, where a, b ∈ Σ, a = b. Then we can show that {ε, ab, a(ab)2 b, · · · , a(ab)k b} ⊆ L(Γk ) but a(ab)k+1 b ∈ L(Γk ) for k ≥ 1. Thus the infinite sequence (wn )n≥0 of strings and infinite sequence (Γn )n≥1 of PFSs satisfies the above conditions, where w0 = ε, wn = a(ab)n b (n ≥ 1). Hence the class PFSL has infinite elasticity. Moriyama&Sato [5] introduced a notion of M-finite thickness by generalizing finite thickness. For nonempty finite set S ⊆ Σ ∗ , we define MIN(S, L) = {L ∈ L | L is a minimal language of S within L},
Learning of Erasing Primitive Formal Systems from Positive Examples
75
where L is a minimal language of S within L if no language L ∈ L satisfy S ⊆ L L. Definition 4. A class L has M-finite thickness, if for any nonempty set S ⊆ Σ ∗ , (i) MIN(S, L) < ∞, (ii) S ⊆ L ∈ L implies that there is a language L ∈ MIN(S, L) such that L ⊆ L. M-finite thickness merely is not a sufficient condition for inferability from positive examples, but is closed under various class operations such as union, concatenation and so on as well as finite elasticity ([5,10]). Theorem 2 (Moriyama&Sato [5]). If a class L has M-finite thickness, then the class L is inferable from positive examples if and only if for each language L ∈ L, there is a finite tell-tale set of L.
4
Reduced PFSs
This section gives a characterization of a reduced PFS. Hereafter, we assume that Σ ≥ 3. Then by Lemma 1, for any regular patterns π and τ , π τ ⇐⇒ L(π) ⊆ L(τ ). Let Γ = (π, τ ) be a PFS. If τ ∈ Σ ∗ , clearly L(Γ ) = L(π) ∪ {τ }. Thus the PFS Γ is reduced if and only if τ π. Hereafter, we consider a PFS Γ = (π, τ ) with var(τ ) = φ, where var(τ ∞ ) = {x1 , x2 , · · · , xn } is the set of variables contained in τ . We define Γτ = t=1 Γt , where Γt is recursively defined as follows: Γ1 = {τπ } and for each t ≥ 2, Γt = Γt−1 ∪ {τ {x1 := ξ1 , x2 := ξ2 , · · · , xn := ξn } | ξi ∈ Γt−1 ∪ {π}, i = 1, 2, · · · , n}. Clearly every ξ in Γτ always contains π as a substring whenever var(τ ) = φ. By the definitions of L(Γ ) and the above Γτ , it follows that: Lemma 2. Let Γ = (π, τ ) be a PFS. Then L(Γ ) = L(π) ∪ L(Γτ ) holds, where L(Γτ ) = ξ∈Γτ L(ξ). By the definition of a reduced PFS, Γ is reduced if and only if L(π) L(Γ ), i.e., L(Γτ ) ⊆ L(π). Clearly if π ∈ Σ ∗ , then Γ is always reduced. For a pattern π with var(π) = φ, let us denote by Aπ the longest constant prefix and by Bπ the longest constant suffix of π. For instance, Aπ = ab and Bπ = ε if π = abx1 aax2 . Lemma 3. Let Γ = (π, τ ) be a PFS with var(π) = φ. For any ξ ∈ Γτ , there is a pair (i0 , j0 ) of positive integers such that Aξ = Aiτ0 Aπ ,
Bξ = Bπ Bτj0 .
For two strings v, w ∈ Σ ∗ , v p w denotes that v is a prefix of w, and v s w means that v is a suffix of w. Moreover, w p v ∗ means that w p v i for some i ≥ 0. Similarly we define w s v ∗ . By Pref, we denote the set of pairs (v, w) of strings satisfying v p w or w p v. Similarly we define the set Suff.
76
J. Uemura and M. Sato
Lemma 4. Let π and τ be regular patterns containing variables. Then L(π) ∩ L(τ ) = φ ⇐⇒ (Aπ , Aτ ) ∈ Pref,
(Bπ , Bτ ) ∈ Suff.
Lemma 5. Let π = Aπ (xπ x )Bπ and τ = Aτ (xτ x )Bτ . If π is a substring of τ , then L(τ ) ⊆ L(π) ⇐⇒ Aπ p Aτ , Bπ s Bτ . By the above lemmas, the next result immediately follows: Theorem 3. Let Γ = (π, τ ) be a PFS. If Aπ = Bπ = ε or Aτ = Bτ = ε, then Γ is not reduced. Remember that the pattern π for a PFS Γ = (π, τ ) is assumed to be of canonical form. Thus by the above theorem, it follows that: Corollary 1. Let Γ = (π, τ ) be a reduced PFS with var(τ ) = φ. Any pattern ξ ∈ Γτ does not contain successive variables. In terms of the above lemma, every pattern in Γτ can be assumed to be of canonical form for a reduced PFS Γ = (π, τ ). Lemma 6. For w ∈ Σ ∗ and v ∈ Σ + , (i) ∃ i0 ≥ 1 s.t. w p v i0 w ⇐⇒ w p v ∗ ⇐⇒ ∀i ≥ 0, w p v i w, (ii) ∃ j0 ≥ 1 s.t. w s wv j0 ⇐⇒ w s v ∗ ⇐⇒ ∀j ≥ 0, w s wv j . Theorem 4. Let Γ = (π, τ ) be a PFS. Then the following statements are equivalent: (i) Γ is reduced. (ii) L(π) ∩ L(Γτ ) = φ. (iii) [Aπ p A∗τ and Aτ = ε] or [Bπ s Bτ∗ and Bτ = ε)], if π ∈ Σ ∗ . Proof. If π ∈ Σ ∗ , clearly (i) and (ii) are equivalent. Hereafter we assume π ∈ Σ ∗ , i.e., π contains some variables. (i) ⇒ (iii). Assume that Γ is reduced, and [Aπ p A∗τ or Aτ = ε] and [Bπ s Bτ∗ or Bτ = ε]. If Aτ = Bτ = ε, then by Theorem 3, Γ is not reduced. A case of Aπ p A∗τ and Bπ s Bτ∗ . Then by Lemma 6, we have Aπ p (Aτ )i Aπ ,
Bπ s Bπ (Bτ )j ,
i, j = 0, 1, 2, · · · .
Let ξ ∈ Γτ be an arbitrary pattern. Then by Lemma 3, there are integers i0 , j0 ≥ 1 such that Aξ = (Aτ )i0 Aπ and Bξ = Bπ (Bτ )j0 . It implies that Aπ p Aξ and Bπ s Bξ hold. Since π is a substring of ξ, by Lemma 5, we get L(ξ) ⊆ L(π). It means that L(Γτ ) ⊆ L(π) holds, and a contradiction. Similarly we can prove for [Aπ p A∗τ and Bτ = ε] or [Aτ = ε and Bπ s Bτ∗ ]. (iii) ⇒ (ii). Assume that L(Γτ ) ∩ L(π) = φ. Then there is a pattern ξ ∈ Γτ satisfying L(ξ) ∩ L(π) = φ. By Lemma 4, (Aπ , Aξ ) ∈ Pref,
(Bπ , Bξ ) ∈ Suff.
Learning of Erasing Primitive Formal Systems from Positive Examples
77
By ξ ∈ Γτ , similarly to the above, there are i0 , j0 ≥ 1 satisfying Aξ = (Aτ )i0 Aπ and Bξ = Bπ (Bτ )j0 . It means that Aπ p (Aτ )i0 Aπ . By Lemma 6(i), Aπ p A∗τ holds. Similarly we have Bπ s Bτ∗ . It contradicts the assumption of (iii). (ii) ⇒ (i) is clear. By the above theorem and Theorem 3, it follows immediately that: Corollary 2. Given a PFS Γ = (π, τ ), the decision problem of whether Γ is reduced is computable in time order O(|π| + |τ |).
5
Inclusion Problem for Erasing PFS Languages
In this section, we deal with the inclusion problem for erasing PFS languages. Lemma 7. Let Γ = (π, τ ) be a reduced PFS and let γ be a pattern such that L(γ) ⊆ L(Γ ). Then γ π ⇐⇒ L(γ) ⊆ L(Γτ ). Proof. We only prove a case of π ∈ Σ ∗ since it can be easily shown for π ∈ Σ ∗ . Since Γ is reduced, by Theorem 4, L(π) ∩ L(Γτ ) = φ holds, and so (⇐) is valid. Moreover [Aπ p A∗τ and Aτ = ε] or [Bπ s Bτ∗ and Bτ = ε] hold. (⇒) We consider only a case of Aπ p (Aτ )∗ and Aτ = ε. We can prove similarly for the other case. In this case, by Lemma 6, (∗)
Aπ p (Aτ )i Aπ ,
i = 1, 2, · · · .
Suppose that γ π and L(γ) ⊆ L(Γτ ). By L(γ) ⊆ L(Γ ), it implies that L(γ) ∩ L(π) = φ and L(γ) ∩ L(ξ) = φ for some ξ ∈ Γτ . Then by Lemma 4, (∗∗)
(Aγ , Aπ ) ∈ Pref,
(Aγ , Aξ ) ∈ Pref.
Since ξ ∈ Γτ , by Lemma 3 there is an integer i0 ≥ 1 such that (∗ ∗ ∗) Aξ = (Aτ )i0 Aπ . By (∗), Aπ p Aξ . Claim A. If |Aπ | ≤ |Aγ |, then Aπ p Aξ . The proof of Claim A. If |Aπ | ≤ |Aγ |, by (∗∗), it follows that Aπ p Aγ . If |Aξ | ≤ |Aγ |, by (∗∗), Aξ p Aγ holds, and so Aπ p Aξ holds. Otherwise, i.e., |Aξ | > |Aγ |, by (∗∗), Aγ p Aξ holds, and so, Aπ p Aξ holds. As mentioned above, since Aπ p Aξ , |Aπ | > |Aγ | must hold. Claim B. If |Aπ | > |Aγ |, then L(γ) ⊆ L(Γ ). The proof of Claim B. By the assumption of our claim, both of the lengths of Aπ and Aξ are larger than |Aγ |. Let a and b be the |Aγ | + 1-th constant symbols from the left sides, respectively. Since Σ ≥ 3, there is a symbol c ∈ Σ such that c = a, b. Let x be the variable of the most left in γ and let γc = γ{x := cx}.
78
J. Uemura and M. Sato
Then L(γc ) ⊆ L(Γ ) and Aγc = (Aγ )c hold. Clearly (Aγc , Ajτ Aπ ) ∈ Pref (j = 0, 1, · · · ) holds, and so by Lemma 4, we have L(γc ) ∩ L(π) = φ. Furthermore, by Lemma 3, for any ξ ∈ Γτ , it implies that L(γc ) ∩ L(ξ ) = φ. Thus L(γc ) ⊆ L(Γ ). That is, L(γ) ⊆ L(Γ ). The claim B leads a contradiction because of L(γ) ⊆ L(Γ ). Lemma 8. Let Γ = (π, τ ) be a reduced PFS and let γ be a pattern with L(γ) ⊆ L(Γτ ). Then L(γ) ∩ L(τπ ) = φ ⇐⇒ γ τπ . The proof can be done similarly to the proof of Lemma 7. Lemma 9. Let Γ = (α, β) be a reduced PFS and let π be a pattern. Then L(Γβ ) ⊆ L(π) ⇐⇒ βα π,
ββα π.
Lemma 10. Let Γ = (α, β) be a reduced PFS and let π be a pattern. Then L(Γ ) ⊆ L(π) ⇐⇒ α π, βα π. By Lemma 9 and Lemma 10, we have the following: Theorem 5. Let Γ = (π, τ ) and Γ = (α, β) be reduced PFSs. If L(Γ ) ⊆ L(Γ ), the following equivalences are valid. (i) (ii) (iii)
6
L(Γ ) ⊆ L(π) ⇐⇒ α π, βα π, L(α) ⊆ L(π), L(Γβ ) ⊆ L(Γτ ) ⇐⇒ α π, L(α) ⊆ L(Γτ ), L(Γβ ) ⊆ L(π) ⇐⇒ α π,
βα π, βα π,
ββα π.
Learnability of the Class PF SL
In the present section, we show that the class PFSL is inferable from positive examples. In order to do it, the class will be shown that the class has M -finite thickness and each language has a finite tell-tale set. 6.1
Finite Tell-Tale Sets for PFS Languages
For a pattern π, S(π) denotes the set of strings obtained from π by substituting each variable to the empty string or a constant symbol in Σ. Lemma 11 (Shinohara [11]). Suppose that Σ ≥ 3. For any regular pattern π, there is no regular pattern τ satisfying S(π) ⊆ L(τ ) L(π).
Learning of Erasing Primitive Formal Systems from Positive Examples
79
For a reduced PFS Γ = (π, τ ), we introduce the following finite subset of L(Γ ): T (Γ ) = L(Γ ) ∩ Σ ≤|ττπ | , where Σ ≤l denotes the set of strings with at most length l for each l ≥ 0. We first consider a PFS Γ = (π, τ ) with var(π) = φ. Clearly S(π) ∪ S(τπ ) ∪ S(ττπ ) ⊆ T (Γ ), where τπ = ττπ = τ if τ ∈ Σ ∗ . Note that the string c(π) is the unique shortest string of L(Γ ) and L(π) if var(τ ) = φ, and that c(τπ ) is that of L(Γτ ). Theorem 6. Let Γ = (π, τ ) be a reduced PFS with var(π) = φ. Then there does not exist a PFS Γ such that T (Γ ) ⊆ L(Γ ) L(Γ ). Proof. We can easily prove for a case of τ ∈ Σ ∗ , and thus consider the case of var(τ ) = φ. We assume T (Γ ) ⊆ L(Γ ) L(Γ ) for some PFS Γ = (α, β). We can assume that Γ is reduced. Clearly c(π) and c(α) are the shortest strings of L(Γ ) and L(Γ ), respectively, and by the above assumption, c(π) = c(α)(= w). Claim A. α = π and S(τπ ) ∪ S(ττπ ) ⊆ L(Γβ ) L(Γτ ). The proof of Claim A. If α π, by Lemma 7, we have L(α) ⊆ L(Γτ ). It means that w ∈ L(Γτ ), and a contradiction. Thus α π, and so L(α) ⊆ L(π). If α ≺ π, then α π{x := ε} for some x ∈ var(π) because of c(π) = c(α). If the variable x is not the first or the last symbol of π, then π = π1 (axb)π2 for some a, b ∈ Σ and for some patterns π1 , π2 . Let c ∈ Σ such that c = a, b, and let w = π{x := c, x := ε x = x}. Clearly w ∈ S(π) and |w | = |w| + 1. As easily seen, w ∈ L(α) holds. By T (Γ ) ⊆ L(Γ ), w ∈ L(Γβ ) must hold. The length of the shortest string c(βα ) in L(Γβ ) is larger than |w|, and is equal to |w| + 1 if and only if β = dx or xd for some d ∈ Σ. In any case, w = dw, wd, and so w ∈ L(Γ ). It is a contradiction. Similarly we can prove for the variable x to be the first symbol or the last symbol of π. Hence α ≡ π must hold. Since both π and α are of canonical form, it implies that π = α. Therefore we have S(τπ ) ∪ S(ττπ ) ⊆ L(Γβ ) L(Γτ ). Claim B. βπ = τπ . The proof of Claim B. By the inclusions of the claim A, c(βπ ) = c(τπ ) holds. If βπ τπ , by Lemma 8, we have L(βπ ) ∩ L(τπ ) = φ. It contradicts c(βπ ) ∈ L(βπ ) ∩ L(τπ ). Hence βπ τπ holds. Similarly, by the claim A, S(τπ ) ⊆ L(Γβ ) holds. If τπ βπ , by Lemma 8 L(τπ ) ∩ L(βπ ) = φ, and a contradiction. Therefore τπ βπ holds. Consequently, we obtain βπ ≡ τπ . In terms of Corollary 1, both βπ and τπ are patterns of canonical form, and thus βπ = τπ . Since π contains variables, it is easily shown that β = τ . It means that L(Γ ) = L(Γ ) holds, and a contradiction.
80
J. Uemura and M. Sato
The following result plays an essential role in our problem on finite tell-tale set. Lemma 12. Let w be a nonempty string in Σ + . (i) If w is a nonmultiple string, then there do not exist strings u, v = ε satisfying w = uv = vu or wu = vw. (ii) If u is a maximal component for w, then wv1 = v2 w implies vi ∈ u∗ for i = 1, 2. Now we consider a PFS Γ = (w, τ ) for w ∈ Σ + . We introduce the following strings: ri (τ ) = τ {xi := τw , xj := w | j = i}, i = 1, · · · , n, where var(τ ) = {x1 , · · · , xn } for n ≥ 1. Then we have {w, s} ∪ {ri (τ ) | i = 1, · · · , n} ⊆ T (Γ ), where s = τw . Note that the string w is the unique shortest string of L(w, τ ) and S(Γ ), and s is the unique second shortest string of them. By Lemma 12, it follows that: Lemma 13. Let Γ = (w, τ ) be a reduced PFS with w ∈ Σ + and var(τ ) = {x1 , · · · , xn } (n ≥ 1). If s is a nonmultiple string, then ri (τ ) = rj (τ ) for i = j. Lemma 14. Let Γ = (w, τ ) be a reduced PFS with w ∈ Σ + and var(τ ) = {x1 , · · · , xn } (n ≥ 1), and let u be a maximal component for s. If w ∈ u+ , then ri (τ ) = rj (τ ) for all i and j. Otherwise ri (τ ) = rj (τ ) for i = j. Theorem 7. Let Γ = (w, τ ) be a reduced PFS with w ∈ Σ + . Then there does not exist a PFS Γ such that T (Γ ) ⊆ L(Γ ) L(Γ ). Proof. We assume that there is a PFS Γ = (α, β) satisfying the inclusion relations in our theorem. Then clearly w and s(= τw ) are the unique shortest string and the unique second shortest string of L(Γ ). It implies that α = w and βα = s. Let τ = v0 x1 v1 x2 · · · xn vn and β = v0 y1 · · · yn vn for some vi , vj ∈ Σ ∗ (i = 0, · · · , n, j = 0, · · · , n ). (i) A case that s is a nonmultiple string. In this case, by Lemma 13, ri (τ ) = rj (τ ) for i = j. Clearly the strings ri (τ ) are the third shortest strings of L(Γ ) and S(Γ ). Similarly the strings rj (β)s are those of L(Γ ). By our assumption, (∗)
{ri (τ ) | i = 1, · · · , n} = {rj (β) | j = 1, · · · , n }.
Since there strings are distinct as mentioned above, we have n = n .
Learning of Erasing Primitive Formal Systems from Positive Examples
81
We show that vi = vi for i = 0, · · · , n. Assume that vi = vi for i = 0, · · · , i0 −2 and vi0 −1 = vi0 −1 for some i0 ≥ 1. Then ri (τ ) = ri (β) for i = 1, · · · , i0 −1. By the above, there is an integer j ≥ i0 satisfying ri0 (τ ) = rj (β). By 2|s| = |ri0 (τ )|+|w|, the substring s appears in both ri0 (τ ) and rj (τ ) at different but overlapped places. It means that sw = w s for some nonempty strings w and w . Applying Lemma 12, it contradicts to the assumption on s. Hence vi = vi holds for every i. Therefore we have Γ = Γ and a contradiction. (ii) A case that a string u is a maximal component for s. By Lemma 14, if w ∈ u+ , then ri (τ ) = rj (τ ) for any i, j, and moreover vi , vi ∈ u∗ . It means that L(Γ ) = L(Γ ) ⊆ u+ , and a contradiction. By Lemma 14, if w ∈ u+ , then ri (τ ) = rj (τ ) for i = j. Similarly to the case of (i), we obtain n = n and (∗). Furthermore, we can show that ww = w w for some nonempty strings w and w , and a contradiction. Finally, we consider a PFS Γ = (ε, τ ), provided τε is nonmultiple string. We define the following strings: for τ = u0 X1 u1 · · · uk−1 Xk uk for u0 , uk ∈ Σ ∗ , ui ∈ Σ + (i = 1, · · · , k − 1), Xi ∈ X + for i = 1, · · · , k, rˆi (τ ) = τ {Xi := s, Xj := ε, j = i},
ti (τ ) = τ {Xi := s|Xi | , Xj := ε, j = i},
for i = 1, · · · , k, where s = τε , Xi := s means a substitution of x := s, x := ε(x = x) for some x in Xi and for any x = x in Xi and Xi := s|Xi | denotes x := s for every x in Xi . Then clearly {ε, s} ∪ {ˆ ri (τ ), ti (τ ) | i = 1, · · · , k} ⊆ T (Γ ).
Theorem 8. Let Γ = (ε, τ ) be a reduced PFS, where τε is a nonmultiple string. Then there does not exist a PFS Γ = (α, β) such that T (Γ ) ⊆ L(Γ ) L(Γ ). Proof. We assume that there is a PFS Γ = (α, β) satisfying the inclusion relations in our theorem. Then clearly α = ε and u0 u1 · · · uk = u0 u1 · · · uk (= s), where τ = u0 X1 u1 X2 · · · Xk uk and β = u0 Y1 · · · Yk uk for some u0 , u0 , uk , uk ∈ Σ ∗ , ui , uj ∈ Σ + (i = 1, · · · , k − 1, j = 1, · · · , k − 1) and Xi , Yj ∈ X + for each i, j. Claim A. k = k and ui = ui for i = 1, · · · , k. Since s is nonmultiple string, the proof of Claim A can be done by showing rˆi (τ ) = rˆj (τ ) for i = j similarly to the proof of Theorem 7. Similarly we can proof Xi = Yi for each i. Theorem 9. Let Γ be a reduced PFS in PFS. Then the set T (Γ ) is a finite tell-tall set of the language L(Γ ).
82
6.2
J. Uemura and M. Sato
M-Finite Thickness
Theorem 10. The class PFSL has M -finite thickness. Proof. Let S ⊆ Σ ∗ be nonempty finite set. Let lmin and lmax be the shortest length and the longest length of strings in S, respectively. Claim A. MIN(S, PFSL) < ∞ holds. The proof of Claim A. Let L(Γ ) ∈ MIN(S, PFSL) for Γ = (π, τ ). Without loss of generality, we can assume that Γ is reduced. As mentioned in §2, every w ∈ L(Γ ) satisfies |w| ≥ min{|c(π)|, |c(τπ )|}. A case of π = ε. In this case, |c(π)| and |c(τπ )| is less than or equal to lmax . Since π is of canonical form and the number of variables in τ is bounded by lmax , there are at most finitely many such PFSs. A case of π = ε. In this case, c(τ ) = τπ holds. Similarly to the above, we have |c(τ )| ≤ lmax , and thus there are at most finitely many such constant strings. Let us put |c(τ )| = l. Then as easily seen, lengths of strings in L(ε, τ ) are multiples of l. Thus lmax = kl for some k ≥ 0. Let us put τ = w0 X1 w1 · · · wn−1 Xn wn . A variable x ∈ var(τ ) is nonerasable w.r.t. S if S ⊆ L(ε, τ {x := ε}). We can assume that every variable in τ is nonerasable w.r.t. S, and such a τ is called a nonerasable pattern w.r.t. S. Clearly |Xi | ≤ k holds for every i. Hence there are at most finitely many PFSs in MIN(S, PFSL). Claim B. For any PFS Γ , if S ⊆ L(Γ ), then L(Γ ) ⊆ L(Γ ) for some Γ ∈ MIN(S, PFSL). The proof of Claim B. Let S ⊆ L(Γ ) for Γ = (π, τ ). We can assume that Γ is reduced, and L(Γ ) ∈ MIN(S, PFSL). Then we have S ⊆ L(Γ ) L(Γ ) for some reduced PFS Γ = (α, β). Similarly to the proof of Claim A, it can be shown that there are at most finitely many such L(Γ ) containing the set S. By the above and Theorem 10, we obtain the following main theorem. Theorem 11. The class PFSL is inferable from positive examples.
References 1. D. Angluin: Inductive inference of formal languages from positive data, Information and Control, 45, 117–135, (1980). 2. S. Arikawa, T. Shinohara and A. Yamamoto: Learning elementary formal system, Theoretical Computer Science, 95, 97–113, (1992). 3. H. Arimura, T. Shinohara and S. Otsuki: Finding minimal generalizations for unions of pattern languages and its application to inductive inference from positive data, Lecture Notes in Computer Science, 775, 646–660, (1994). 4. E.M. Gold: Language identification in the limit, Information and Computation, 10, 447–474, (1967). 5. T. Moriyama and M. Sato: Properties of language classes with finite elasticity, IEICE Transactions on Information and Systems, E78-D(5), 532–538, (1995).
Learning of Erasing Primitive Formal Systems from Positive Examples
83
6. T. Motoki, T. Shinohara and K. Wright: The correct definition of finite elasticity: Corrigendum to identification of unions, Proceedings of the 4th Annual Workshop on Computational Learning Theory, 375–375, (1991). 7. Y. Mukouchi and S. Arikawa: Towards a mathematical theory of machine discovery from facts, Theoretical Computer Science, 137, 53–84, (1995). 8. Y. Mukouchi: Note on Learnability of Subclasses of Erasing Elementary Formal Systems from Positive Examples, in preparation. 9. D. Reidenbach: Result on Inductive Inference of Extended Pattern Languages, Lecture Notes in Artificial Intelligence, 2533, 308–320, (2002). 10. M. Sato: Inductive Inference of Formal Languages, Bulletin of Informatics and Cybernetics, 27(1), 85–106, (1995). 11. T. Shinohara: Polynomial time inference of extended regular pattern languages, Lecture Notes in Computer Science, 147, 115–127, (1982). 12. T. Shinohara: Inductive inference of formal systems from positive data, Bulletin of Information and Cybernetics, 22, 9–18, (1986). 13. T. Shinohara: Rich classes inferable from positive data, Information and Control, 108, 175–186, (1994). 14. R.M. Smullyan: “Theory of Formal Systems,” Princeton University Press, 1961. 15. J. Uemura and M. Sato: Compactness and Learning of Unions of Erasing Regular Pattern Languages, Lecture Notes in Artificial Intelligence, 2533, 293–307, (2002). 16. K. Wright: Identification of unions of languages drawn from positive data, in Proceedings of the 2nd Annual Workshop on Computational Learning Theory, 328– 333, (1989).
Changing the Inference Type – Keeping the Hypothesis Space Frank Balbach Institut f¨ ur Theoretische Informatik, Universit¨ at zu L¨ ubeck Wallstraße 40, 23560 L¨ ubeck, Germany [email protected]
Abstract. In inductive inference all learning takes place in hypothesis spaces. We investigate for which classes of recursive functions learnability according to an inference type I implies learnability according to a different inference type J within the same hypothesis space. Several classical inference types are considered. Among FIN, CONS-CP, and CP the above implication is true, for all relevant classes, independently from the hypothesis space. On the other hand, it is proved that for many other pairs (I, J ) hypothesis spaces exist that allow full I learning power, but limit that of J to finite classes. Only in a few cases (e. g. LIM vs. CONS) the result does depend on the actual class to be learned.
1
Introduction
In inductive inference a scenario is investigated where a learner receives more and more data about a target object and outputs a sequence of hypotheses. The learner is successful if its sequence of hypotheses eventually converges to a single description for the target object. Usually a learner is required to learn all objects from a (possibly infinite) class. This model of learning in the limit [11], applied to classes of recursive functions, has been studied thoroughly. Thereby, many variations of the basic model have been developed [4,19,1,8,12,10,9,13]. All these models are referred to as “inference types.” They often differ in the constraints placed on the intermediate hypotheses or in the way the sequence of hypotheses has to converge. Common to all inference types, however, is the need to interpret the hypotheses a learner (“strategy”) outputs. This is usually done by means of a hypothesis space. Its design is thus of key importance to the learning success. In the inductive inference of recursive functions, hypothesis spaces are numberings of partial recursive functions. Hypotheses are represented by indices in such numberings. It is well known that G¨ odel numberings [15] (acceptable numberings [14]) serve, in many inference types, as universal hypothesis spaces. That is, any solvable learning problem can be solved within such a numbering. Of course, knowing that a solution exists is not sufficient in practical applications. One rather needs to know how to construct an appropriate learner. R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, pp. 84–98, 2003. c Springer-Verlag Berlin Heidelberg 2003
Changing the Inference Type – Keeping the Hypothesis Space
85
Constructing learning strategies is usually easier in hypothesis spaces specifically designed for a certain learning task than it is in G¨ odel numberings. A first approach to such hypothesis spaces is given by numbering theoretical characterizations [19,9]. As an example, let us consider: A class U of recursive functions is learnable in the limit iff there is a numbering ψ such that (a) ψ contains all functions of U and (b) for different indices i, j one can uniformly compute an upper bound d(i, j) of an argument on which the i-th and j-th function in ψ differ. This characterization provides a sufficient criterion for hypothesis spaces ψ to be suitable for the learning of U . Moreover, learners for classes U can be built uniformly from the hypothesis space ψ and the function d. Specialized hypothesis spaces are also necessary in combination with general types of strategies, like identification by enumeration [11]. In its basic form this strategy outputs the least index whose associated function is consistent with the data known to the strategy so far. Therefore it often needs to decide the consistency of an index with some known data, which usually cannot be done in G¨ odel numberings [20]. Despite their advantages regarding the construction of learning strategies, specialized non-G¨ odel numberings suffer from several drawbacks. One of them is a loss of flexibility. Changing the class to be learned or switching to a different type of learning, thereby leaving the hypothesis space unchanged, might lead to an unsolvable new learning problem. In this paper we will concentrate on the consequences of switching to another inference type after fixing the hypothesis space. The inductive inference paradigm can also be viewed from a more practical point of view. Here, the hypothesis spaces correspond to certain output formats, or description languages, for hypotheses. The inference types find their counterpart in various requirements put on the behavior of a learning algorithm and/or the quality of its hypotheses. Consider a typical practical learning algorithm that uses a certain description language for hypothesis and learns according to some requirements. In practice, requirements are often changing. The output format might be fixed, however. The first natural question then is whether the algorithm can still handle the new requirements or whether there is at all an algorithm that can cope with the new situation. However, the new requirements could be too demanding for such an algorithm to exist. But what happens if a learning algorithm for the new problem is known that (unfortunately) uses a different output format? Is this additional fact enough to conclude that there must be such an algorithm for the previously fixed format, too? In general the possible answers are “yes, this conclusion can be drawn”, “no, it cannot”, or “it depends...” In case of a positive answer the immediate next question is how to find such an algorithm. Can it even be constructed effectively from the previous one?
86
F. Balbach
The following pages give answers to these questions in the general framework of inductive inference of recursive functions, thereby revealing that all three kinds of answers do indeed occur, depending on the inference types under consideration. Section 3 addresses the “no” answers, Sect. 4 presents some “yes” answers, and Sect. 5 some intermediate results. Finally, Sect. 6 gives an overview of all results obtained.
2
Preliminaries
Notations not explained herein follow standard conventions [15]. By IN we denote the set {0, 1, 2, . . .} of natural numbers; inclusion and proper inclusion are denoted by ⊆ and ⊂, respectively; card A is the cardinality of the set A. We denote the set difference of A and B by A \ B. For n ∈ IN the set of all n-ary partial recursive functions over IN will be written P n , the set of all recursive functions Rn . As abbreviation for P 1 and R1 we use P and R, respectively. For f ∈ P, x ∈ IN we write f (x) ↓, if f is defined on input x and f (x) ↑ otherwise. Functions f, g ∈ P fulfill f =n g iff {(x, f (x)) | x ≤ n and f (x) ↓} = {(x, g(x)) | x ≤ n and g(x) ↓}. For i ∈ IN and n ≥ 1, in denotes the n-tuple (i, . . . , i). Functions can be identified with the sequence of their values, e. g. f = 1n 0∞ means f (x) = 1 for 0 ≤ x < n and f (x) = 0 for x ≥ n. If for f ∈ P and n ∈ IN the values f (0), . . . , f (n) are defined, we will write f n for the initial segment (f (0), . . . , f (n)) and implicitly identify every f n with a natural number via a computable bijective coding function. For a tuple α = (α0 , . . . , αn ) over IN and a class U ⊆ R, we write α U iff there is an f ∈ U such that f n = α. In order to abbreviate certain statements, the symbols ∧ (“and”), =⇒ (“implies”) and ⇐⇒ (“iff”) as well as the quantifiers ∀ (“for all”), ∀∞ (“for all but finitely many”), and ∃ (“exists”) will be used. Let ψ ∈ P 2 . Then ψ is called numbering and Pψ := {ψi | i ∈ IN} denotes the set of the functions enumerated by ψ; for i ∈ IN the function ψi is defined odel numbering by ψi (x) := ψ(i, x) for all x. A numbering ϕ ∈ P 2 is called G¨ (acceptable numbering) iff (1) Pϕ = P and (2) ∀ψ ∈ P 2 ∃c ∈ R ∀i [ψi = ϕc(i) ]. We use ϕ to denote a fixed G¨ odel numbering. For a function ϕi we will write Si if the function plays the role of a learning strategy (see below). Let Φ be a Blum complexity measure [6] for ϕ. For i ∈ IN we will write ϕi (x) ↓n instead of Φi (x) ↓≤ n. In the basic learning model, a strategy S ∈ P learns a class U ⊆ R with respect to a hypothesis space ψ ∈ P 2 . The strategy receives one after another initial segments f n of a function f ∈ U as input and generates a sequence of hypotheses S(f n ) as output. Each hypothesis is interpreted as the function ψS(f n ) . The basic inference type, learning in the limit [11], gives the learning strategy the freedom to output whatever it wants, as long as it reaches a point beyond that the output remains constant as well as correct.
Changing the Inference Type – Keeping the Hypothesis Space
87
Definition 1. A class U ⊆ R is said to be learned in the limit with respect to (or in) an hypothesis space ψ ∈ P 2 by a strategy S ∈ P iff (1) ∀f ∈ U ∀n [S(f n ) ↓], (2) ∀f ∈ U ∃i [ψi = f ∧ ∀∞ n [S(f n ) = i]]. This fact will be written U ∈ LIMψ (S). Furthermore LIMψ := {U | U ⊆ R ∧ ∃S ∈ P [U ∈ LIMψ (S)]} denotes the set of LIM learnable classes with respect to ψ and LIM := ψ∈P 2 LIMψ the entire set of LIM learnable classes. Various inference types are built from LIM by adding conditions to the intermediate hypotheses. Probably the most natural one is the consistency condition demanding that the hypothesized function always agrees with the data already known to the strategy [2,7]. It has been intensively studied [3,18,20,16]. In any case, the hypothesized functions have to be total in order to be correct, hence it is natural to demand that all hypotheses refer to total functions. This is called total learning [19]. It is restricted even further within the so called class preserving learning [5] where the hypothesized functions are required to be members of the class to be learned. Both, total and class preserving learning, can be combined with the consistency condition [12]. Definition 2. Let U ⊆ R be a class, ψ ∈ P 2 a hypothesis space, and S ∈ P a strategy such that U ∈ LIMψ (S). (1) U ∈ CONSψ (S) iff ∀f ∈ U ∀n [f =n ψS(f n ) ] . (2) U ∈ TOTALψ (S) iff ∀f ∈ U ∀n [ψS(f n ) ∈ R] . (3) U ∈ CPψ (S) iff ∀f ∈ U ∀n [ψS(f n ) ∈ U ] . (4) U ∈ CONS-TOTALψ (S) iff ∀f ∈ U ∀n [f =n ψS(f n ) ∈ R] . (5) U ∈ CONS-CPψ (S) iff ∀f ∈ U ∀n [f =n ψS(f n ) ∈ U ] . CPψ , TOTALψ , CONSψ , CONS-TOTALψ , and CONS-CPψ as well as CP, TOTAL, CONS, CONS-TOTAL, and CONS-CP are defined in analogy to LIMψ and LIM. Instead of convergence to a single correct hypothesis, the behaviorally correct learning in the limit only requires that a learning strategy outputs almost always arbitrary, but correct hypotheses [3]. Definition 3. A class U is said to be learned behaviorally correct in the limit by a strategy S with respect to a hypothesis space ψ (written: U ∈ BCψ (S)) iff (1) ∀f ∈ U ∀n [S(f n ) ↓], (2) ∀f ∈ U ∀∞ n [ψS(f n ) = f ]. BCψ and BC are defined analogously to the previous inference types. Convergence of a different kind takes place in the finite learning model, also called one-shot learning. Here the strategy may, on each function, output the special hypothesis “?” finitely many times until it outputs a correct one [11]. Definition 4. A class U is said to be learned finitely by a strategy S with respect to a hypothesis space ψ (written: U ∈ FINψ (S)) iff (1) ∀f ∈ U ∀n [S(f n ) ↓],
88
F. Balbach
(2) ∀f ∈ U ∃n (a) ∀x < n [S(f x ) = ?], (b) ∃i [ψi = f ∧ ∀x ≥ n [S(f x ) = i]]. FINψ and FIN are defined in the obvious way. For strategies S (except for BC strategies), the convergence point on a learned function f is denoted by Conv(S, f ) := min{n | ∀x ≥ n [S(f x ) = S(f n )]}. and the final hypothesis of S on f by lim S(f ) := limn→∞ S(f n ). The convergence point for BC strategies depends on the hypothesis space ψ and is defined by Convψ (S, f ) := min{n | ∀x ≥ n [ψS(f x ) = f ]}. For every inference type introduced here, G¨ odel numberings present a universal hypothesis space insofar as every learnable class can be learned in any such numbering [13]. Lemma 5. Let U ⊆ R be a class of recursive functions, I ∈ {LIM, CONS, TOTAL, CP, CONS-TOTAL, CONS-CP, BC, FIN} an inference type, and ϕ an arbitrary G¨ odel numbering. Then Iϕ = I. Some inference types have another property, namely that the learning goal can always be achieved by a strategy defined on every input. Lemma 6. Let I ∈ {LIM, TOTAL, CP, BC, FIN} be an inference type and S ∈ P, U ⊆ R, ψ ∈ P 2 satisfying U ∈ Iψ (S). Then there is a strategy T ∈ R such that U ∈ Iψ (T ). Both lemmata will be used implicitly in the next sections. The set of classes contained in total numberings is denoted by NUM := {U | ∃ψ ∈ R2 [U ⊆ Pψ ]}. It is known [11] that every class U ∈ NUM enumerated by ψ ∈ R2 , i. e. U ⊆ Pψ , can be learned with respect to ψ by the strategy Enumψ (f n ) := min{i | ψi =n f }. The relations between the inference types in terms of learnable classes are described in the following theorem [3,18,12]. Theorem 7. (1) CONS-CP ⊂ CP ⊂ TOTAL = CONS-TOTAL ⊂ CONS ⊂ LIM ⊂ BC, (2) FIN ⊂ CP, and (3) NUM ⊂ TOTAL. (4) Inference types whose relations are not explicitly stated are incomparable. The question formulated in the introduction can now be expressed more formally by: Does U ∈ J satisfy ∀ψ ∈ P 2 [U ∈ Iψ =⇒ U ∈ Jψ ]? We give a name to the condition contained therein. Definition 8. Let I, J ∈ {BC, LIM, FIN, CONS, TOTAL, CP, CONS-CP, CONS-TOTAL} be inference types. A class U ⊆ R has the property (satisfies the condition) (I → J ) iff ∀ψ ∈ P 2 [U ∈ Iψ =⇒ U ∈ Jψ ]. Note that all finite classes U satisfy (I → J ) for all introduced inference types I and J . The set {U | U ∈ I ∩ J and U satisfies (I → J )} will be called scope of (I → J ). Classes U ∈ / I ∩ J are not considered since it is obvious whether they satisfy (I → J ) or not. A scope containing exactly the finite classes will be called minimal, a scope equal to I ∩ J will be called maximal. Often (I → J ) is fulfilled for all I ∩ J simply because ∀ψ ∈ P 2 [Iψ ⊆ Jψ ]. The next lemma states when this happens.
Changing the Inference Type – Keeping the Hypothesis Space
89
Lemma 9. (1) ∀ψ ∈ P 2 [FINψ ⊆ CPψ ⊆ TOTALψ ⊆ CONSψ ⊆ LIMψ ⊆ BCψ ], (2) ∀ψ ∈ P 2 [CONS-CPψ ⊆ CONS-TOTALψ ⊆ TOTALψ ], (3) ∀ψ ∈ P 2 [CONS-CPψ ⊆ CPψ ]. Note, however, that I ⊆ J is not sufficient for ∀ψ ∈ P 2 [Iψ ⊆ Jψ ]. Lemma 10. (1) ∃ψ ∈ P 2 [TOTALψ ⊆ CONS-TOTALψ ], (2) ∃ψ ∈ P 2 [CPψ ⊆ CONS-TOTALψ ].
3
Negative Results and Biased Hypothesis Spaces
We will first consider the transition from TOTAL learning to CONS learning. Both requirements are rather natural and lie close to each other in the hierarchy (see Theorem 7 and Lemma 9). Hence, one would not expect any problems if one wants to learn a TOTAL class in a TOTAL way with respect to a hypothesis space suitable for learning this class consistently. However, this expectation is wrong. As soon as the class to be learned is infinite, the hypothesis space could be a “bad” one preventing the TOTAL learning of the class. This is the subject of the following Theorem 11. Theorem 11. Let U ∈ TOTAL be an infinite class. Then a hypothesis space ψ ∈ P 2 exists such that U ∈ CONSψ \ TOTALψ . Proof. Let U ∈ TOTAL. The hypothesis space ψ will be defined via diagonalization against all (TOTAL) learning strategies. The functions in ψ will be grouped into consecutive blocks Zj of increasing size. Within the j-th block, which contains j + 2 functions, diagonalization against the j + 1 strategies S0 , . . . , Sj happens. The functions in the block will be defined, argument by argument, to equal ϕj . Meanwhile the output of the strategies S0 , . . . , Sj on initial segments of ϕj is watched. As soon as an Si is found to output a hypothesis z from within the j-th block, the definition of ψz is stopped, resulting in ψz ∈ P \ R. From now on neither Si nor ψz are taken into account during the ongoing definition of the j-th block. The formal algorithm for the j-th block is given below. (j)
L0 := {0, . . . , j} (j) G0 := Zj x := 0 While ϕxj ↓ do: (j) (1) For all z ∈ Gx : ψz (x) := ϕj (x) (j) (2) For all z∈ Zj \ Gx : ψz (x) := ↑
(j) (j) := (, y) | ∈ Lx ∧ y ≤ x ∧ S (ϕyj ) ↓x ∈ Gx (j) (j) (j) (j) (4) Gx+1 := Gx \ S (ϕyj ) | (, y) ∈ Px ∧ y = min{z | (, z) ∈ Px } (j)
(3) Px
90
F. Balbach
(j) (j) (j) (5) Lx+1 := Lx \ | ∃y [(, y) ∈ Px ] (6) x := x + 1 Note that, since the indices outnumber the strategies, there must remain at least one z ∈ Zj such that ψz = ϕj . To prove U ∈ / TOTALψ , we assume an Si such that U ∈ TOTALψ (Si ). Since U is infinite, there must be an f ∈ U such that Si converges on f to an index k ∈ Zj for a j ≥ i. All total functions in the j-th block equal ϕj , hence f = ψk = ϕj . Therefore, Si outputs on ϕj almost always indices in Zj . Thus, either Si or k (or both) will be “eliminated.” In either case, Si outputs a non-total hypothesis on f ∈ U , contradicting the assumption. In order to show U ∈ CONSψ , let R ∈ P be a strategy such that U ∈ CONS-TOTALϕ (R). A strategy T learning U consistently in ψ works as follows: T (f n ) := min Gn(R(f
n
))
.
n
(R(f ))
For f ∈ U , ϕR(f n ) is total, hence Gn exists and can be computed using the algorithm above. Let f ∈ U and j = lim R(f n ) be the final hypothesis of R on f . Then T converges against the least k ∈ Zj not “eliminated.” It follows that ψk = ϕj , hence T converges correctly. That the intermediate hypotheses of T are consistent is a consequence of R being a consistent strategy for U in ψ. If consistent learnability of a class is not sufficient for its total learnability, then neither LIM nor BC learnability are. Moreover, consistent learnability cannot be sufficient for learning in a CONS-TOTAL, CP, CONS-CP, or FIN way. Corollary 12. Let I ∈ {CONS, LIM, BC} and J ∈ {FIN, CONS-CP, TOTAL, CONS-TOTAL,CP} be inference types and let U ∈ J be an infinite class. Then a hypothesis space ψ ∈ P 2 exists such that U ∈ Iψ \ Jψ . A closer look at the proof of Theorem 11 reveals that the constructed hypothesis space ψ does not depend on the class U . In ψ no infinite class U ∈ TOTAL can be learned totally. But any such U can be learned consistently in ψ. Even more is true: The hypothesis space ψ does in fact allow for the full learning power of consistent learning as well as of limit learning in general. Behaviorally correct learning, on the other hand, does not increase the learning power further. Corollary 13. There is a hypothesis space ψ satisfying (1) TOTALψ = {U ⊆ R | card U < ∞}, (2) CONSψ = CONS, (3) LIMψ = LIM, (4) BCψ = LIM. Proof. Let ψ be the hypothesis space constructed in the proof of Theorem 11. (1) is obvious, as well as the ⊆-part of (2). In order to prove CONSψ ⊇ CONS let U ∈ CONS be a class and R ∈ P a strategy such that U ∈ CONSϕ (R). We
Changing the Inference Type – Keeping the Hypothesis Space
91
define T as in the proof of Theorem 11 with the only difference that R is a CONS strategy for U . The proof of (3) proceeds similar to that of (2). In order to show U ∈ LIMψ for a U ∈ LIM we use a strategy R ∈ R with U ∈ LIMϕ (R) and define T as follows: (R(f m ))
T (f n ) := “For m = 0, . . . , n compute the sets Gm for at most n steps. Let m be the greatest index such that the computation is finished.
(R(f m ))
Output min Gm
.”
This modification is necessary since ϕR(f n ) need not be defined up to n, (R(f n ))
hence Gn
need not exist.
(4) LIM = LIMψ ⊆ BCψ follows from (3) and the definitions of BC and LIM. In order to show BCψ ⊆ LIM we assume a U ∈ BCψ with U ∈ / LIM. Let S ∈ R be such that U ∈ BCψ (S). Then there must be a function g ∈ U such that card {S(g n ) | n ∈ IN} = ∞ (otherwise one could amalgamate the finitely many indices of each function of U and learn U in the limit, in contradiction to our assumption). Recall the grouping of the indices of ψ in blocks Zj . The hypotheses of S on g reach infinitely many such blocks. We assume without loss of generality that n S / outputs on g at most one index from each such block, that is ∀n [S(g ) ∈ m ) ]. (One can construct an S with this property from an S without Z S(g m
k will be selected for “elimination” in step (4) since (i, m) ∈ Px and m = (j) min{z | (i, z) ∈ Px } (remember that, by assumption an S, g m is the only segment where S outputs a hypothesis from Zj ). This contradicts the conclusion above that k “survives.” Note that part (1) of Corollary 13 remains true if TOTAL is substituted by CONS-TOTAL, CP, CONS-CP, or FIN. Obviously, ψ is biased towards CONS and LIM and against most other inference types considered. The diagonalization technique from the proof of Theorem 11 can, suitably modified, be used for other inference types as well. In order to prove the next theorem we need a well known lemma that provides a stronger version of Lemma 6 for LIM. Lemma 14. There is a function ρ ∈ R such that ∀i ∈ IN (1) ϕρ(i) ∈ R, (2) ∀ψ ∈ P 2 ∀U ⊆ R [U ∈ LIMψ (ϕi ) =⇒ U ∈ LIMψ (ϕρ(i) )].
92
F. Balbach
That is, for every LIM strategy ϕi , ρ effectively constructs an everywhere defined LIM strategy ϕρ(i) with at least the same learning power. We abbreviate the strategies from the last lemma by Si := ϕρ(i) . Theorem 15. Let U ∈ LIM be an infinite class. Then there is a hypothesis space ψ ∈ P 2 such that U ∈ BCψ \ LIMψ . Proof. This proof uses the same blockwise grouping of indices as the proof of Theorem 11. However, diagonalization happens against all strategies Si instead of Si . For every j ∈ IN the functions with indices z ∈ Zj are defined as follows: ϕj (x), if ∀i ≤ j ∃n ≥ x [ϕnj ↓ ∧Si (ϕnj ) = z], ψz (x) := ↑, otherwise. Provided ϕj ∈ R, the function ψz equals ϕj iff every strategy S0 , . . . , Sj outputs on ϕj infinitely often a hypothesis different from z. If, on the other hand, one of those strategies converges on ϕj to z, then ψz ∈ / R. Furthermore there is a z ∈ Zj with ψz = ϕj . Assume U ∈ LIMψ (Si ) for an i ∈ IN. Then an f ∈ U and a j ≥ i exist such that k := lim Si (f ) ∈ Zj . Hence ψk = f = ϕj holds. But then, according to the definition of ψk , Si does not converge to k, a contradiction. In order to prove U ∈ BCψ , let R ∈ R be a LIM strategy for U in ϕ. A BC strategy for U in ψ can be defined in the following way. T (f n ) := “(1) For all z ∈ ZR(f n ) and x ≤ n compute each ψz (x) for at most n steps. (2) Output a z such that ψz has a longest initial segment computed in (1).” Let f ∈ U and j := lim R(f ). Then T outputs on f almost always indices of Zj . Let m be the length of the longest initial segment of all non-total functions from Zj . Let n ≥ Conv(R, f ). Eventually n will be great enough for step (1) to compute an initial segment longer than m, since the total function ϕj = f appears in the j-th block. Then T will output an index z of a total function ψz = ϕj = f and therefore f behaviorally correct. The proof does not only prove the scope of (BC → LIM) to be minimal. Again, it yields a biased hypothesis space, this time favouring BC and disliking LIM. Corollary 16. There is a hypothesis space ψ satisfying (1) LIMψ = {U ⊆ R | card U < ∞}, (2) BCψ =BC. We will now turn to the property (TOTAL → FIN). Once again we apply the same proof technique. But this time we only get a proof for the minimality of the scope. We are not provided with a biased hypothesis space. Theorem 17. Let U ∈ FIN be an infinite class. Then a hypothesis space ψ ∈ P 2 exists such that U ∈ TOTALψ \ FINψ .
Changing the Inference Type – Keeping the Hypothesis Space
93
Proof. Let R ∈ R such that U ∈ FINϕ (R). We need to modify the construction of the proof of Theorem 11 in such a way that the class U will be taken into account. This is inevitable, as will be shown in Theorem 18. The construction of the j-th block starts by watching R on the function ϕj . Only when (and if) R converges finitely to the index j, a construction very similar to that of the proof of Theorem 11 takes place. For the sake of completeness the algorithm constructing the block with indices from Zj is given below. x := 0 While both (A) ϕxj ↓ and (B) R(ϕxj ) = ? do: For all z ∈ Zj : ψz (x) := ϕj (x) x := x + 1 Case 1: (A) and (B) hold for all x. Then ∀z ∈ Zj [ψz = ϕj ] and R does not learn ϕj , hence ϕj ∈ / U. Case 2: There is x0 such that (A) does not hold. Then ∀z ∈ Zj ∀x ≥ x0 [ψz (x) ↑] and therefore ∀z ∈ Zj [ψz ∈ / U ]. Case 3: There is x0 such that (A) holds, but (B) does not, i. e. ϕxj 0 ↓ and R(ϕxj 0 ) ∈ IN. Then ∀z ∈ Zj [ψz =x0 −1 ϕj ] follows and two cases are distinguished: Case 3.1: R(ϕxj 0 ) = j. Then define for all z ∈ Zj and x ≥ x0 : ψz (x) := ↑. Case 3.2: R(ϕxj 0 ) = j. Then the construction goes on in the following way. (j)
Lx0 := {0, . . . , j} (j) Gx0 := Zj x := x0 While ϕxj ↓ do: Perform steps (1) to (6) as in the proof of Theorem 11. The proofs of U ∈ TOTALψ and U ∈ / FINψ are not very different from the corresponding ones in Theorem 11, but somewhat more technical, and will be omitted due to space constraints. The question, whether there is a biased hypothesis space, i. e. a ψ such that TOTALψ = TOTAL and FINψ = {U | card U < ∞}, has been answered by an anonymous reviewer of this paper’s submitted version. The proof is also due to this reviewer. Theorem 18. If ψ is a hypothesis space with TOTALψ = TOTAL, then there is an infinite class in FINψ . Proof. Let U1 = {0n 1∞ | n ∈ IN}. U1 is in TOTAL and thus in TOTALψ . Let S be a TOTALψ strategy for U1 . Now U2 = {ψS(0n 1m ) | n ∈ IN and m is the first number such that m > 0 and ψS(0n 1m ) extends 0n 1 as a function}
94
F. Balbach
is a class of total functions since S only outputs indices of total functions. Furthermore, the test whether ψS(0n 1m ) extends 0n 1 can be carried out effectively for m = 1, 2, . . . since the indices to be simulated are total. Thus U2 is well-defined and recursively enumerable. For every n there is exactly one function in U2 which extends 0n 1; thus U2 is infinite. Furthermore, U2 is in FINψ : on input which does not start with 0n 1, one outputs “?”, on input that starts with 0n 1 one outputs the ψ-index for the unique function in U2 which extends 0n 1. Among all properties (I → J ) so far proven to have minimal scope, the property (TOTAL → FIN) is the only one where no biased hypothesis space exists.
4
Positive Results
The previous section might look somewhat discouraging. However, not every property (I → J ) can be proved to have minimal scope. Contrary, there are inference types I and J such that the scope of (I → J ) is maximal. Theorem 19. If U ∈ FIN and ψ ∈ P 2 such that U ∈ CPψ , then U ∈ FINψ . Proof. Let R, S ∈ R be such that U ∈ FINϕ (R) and U ∈ CPψ (S). Define T (f n ) := “If R(f n ) = ? then output ‘?’. n n If R(f n ) ∈ IN ∧ ∀m ≤ n [ψS(f m ) ↓= f ] then output ‘?’. n n If R(f n ) ∈ IN ∧ ∃m ≤ n [ψS(f m ) ↓= f ] then: n n m := min{m ≤ n | ψS(f m ) ↓= f }. Output S(f m ).” On f ∈ U the strategy T outputs “?” until an n with n ≥ Conv(R, f ) and ψS(f n ) =n f is reached. Since R is a FIN strategy that has already converged on f , there can be no function g = f in U such that g =n f (otherwise R would converge on such a g to the same final hypothesis as it does on f , a contradiction). But if there is only one function in U starting with f n , this function must be identical to ψS(f n ) , for S is a CP strategy. It follows that S(f n ) is a correct hypothesis for f with respect to ψ. Theorem 19 addresses the transition from class preserving learning to finite learning. It does not only say that this translation is possible, it also provides an algorithm to construct a finite learning strategy from a class preserving one. Thus it presents a positive answer to the introductory question on effective constructability of such learning algorithms. The next theorem gives a very similar result concerning (CP → CONS-CP). Theorem 20. If U ∈ CONS-CP and ψ ∈ P 2 such that U ∈ CPψ , then U ∈ CONS-CPψ .
Changing the Inference Type – Keeping the Hypothesis Space
95
Proof. The proof is similar to that of Theorem 19. Let R ∈ P be such that U ∈ CONS-CPϕ (R) and S ∈ R such that U ∈ CPψ (S). Define T (f n ) := “If ψS(f n ) =n f then output S(f n ). If ψS(f n ) =n f then: g := ϕR(f n ) . Find m > n with ψS(gm ) =n f and output S(g m ).” Let f ∈ U and n ∈ IN. Then ψS(f n ) ∈ U follows, since S is a class preserving strategy. Furthermore “ψS(f n ) =n f ?” is decideable. If this condition is satisfied, T (f n ) = S(f n ) is a class preserving consistent hypothesis; else for g := ϕR(f n ) we get g ∈ U and g =n f because of the properties of S. Since S learns all functions of U in the limit, an m > n must exist such that ψS(gm ) = g =n f . This condition can easily be checked because S outputs only total hypotheses with respect to ϕ on functions g ∈ U . Thus, such an m can be found effectively and S(g m ) is a consistent and class preserving hypothesis. It remains to show that T learns every f ∈ U in the limit. For n ≥ Conv(S, f ) the first condition in the definition of T is satisfied and T (f n ) = S(f n ). Hence, T converges on f to the same final hypothesis as S. Corollary 21. If U ∈ CONS-CP and ψ ∈ P 2 such that U ∈ FINψ , then U ∈ CONS-CPψ . Corollary 22. If U ∈ FIN and ψ ∈ P 2 such that U ∈ CONS-CPψ , then U ∈ FINψ . Corollary 22 is dual to Corollary 21. Thus, for all classes U ∈ CONS-CP∩FIN and all hypothesis spaces ψ the equivalence U ∈ FINψ ⇐⇒ U ∈ CONS-CPψ holds. Moreover, CP can be added to this equivalence. Corollary 23. For all U ∈ CONS-CP ∩ FIN and for all ψ ∈ P 2 , U ∈ FINψ ⇐⇒ U ∈ CONS-CPψ ⇐⇒ U ∈ CPψ . The equivalence described in the last corollary is remarkable in two ways. First, it concernes an inference type and its consistent variant and, second, it concernes two inference types that are incomparable regarding their learning power. It shall be noted, however, that this equivalence is only valid within a relatively small area, FIN ∩ CONS-CP. But nevertheless, none of the other introduced inference types could be added to the statement of Corollary 23. Hence, there is indeed a close relationship between FIN, CONS-CP, and CP.
5
An Intermediate Result — CONS vs. LIM
So far, every examined property (I → J ) had either minimal or maximal scope. This need not be so for every pair of inference types. In order to show this, we will turn our attention to the condition (LIM → CONS). All classes U ∈ CP satisfy (LIM → CONS), as has already been proved [16].
96
F. Balbach
Theorem 24. If U ∈ CP and ψ ∈ P 2 such that U ∈ LIMψ , then U ∈ CONSψ . Naturally the question arises, whether (LIM → CONS) holds not only for the classes U ∈ CP, but for all U ∈ CONS. The next theorem gives a negative answer to this question. It shows that even in NUM there are classes which do not satisfy (LIM → CONS). This is remarkable since NUM classes tend to be easily learnable. After all, there are always total numberings that can be used as hypothesis spaces for them. However, the properties (I → J ) take into account all numberings, whether total or not. This is not unrealistic, even in the case of NUM classes, because deciding whether a class is embedded in a total numbering can be much harder than to find a non-total hypothesis space suitable for learning the class [17]. Theorem 25. There is a class U ∈ NUM and a hypothesis space ψ ∈ P 2 such that U ∈ LIMψ \ CONSψ . Proof. For the construction of ψ let η ∈ R2 be a numbering of V := {f ∈ R | ∀∞ n [f (n) = 0]} \ {0∞ } with the property ∀i ∀j [i = j =⇒ ηi = ηj ]. For all i ∈ IN set ψ2i+1 := ηi and ψ2i (0) := i. For all i ∈ IN and x ≥ 1 define (=: (A)) i, if Si (ix+1 ) ∈ 2IN \ {2i}, i, if Si (ix+1 ) ∈ 2IN + 1, (=: (B)) ψ2i (x) := x+1 (i ) = 2i, (=: (C)) ↑, if S i ↑, if Si (ix+1 ) ↑ . (=: (D)) Define U := V ∪ {ψ2i | ψ2i = i∞ }. Clearly, U ∈ NUM. Furthermore, U is learnable in the limit with respect to ψ by the strategy T defined as follows: 2 · f (0), if f (0) = . . . = f (n), n T (f ) := 2 · Enumη (f n ) + 1, otherwise. To prove U ∈ / CONSψ , assume a strategy Si such that U ∈ CONSψ (Si ). Considering the behavior of Si on initial segments of i∞ , and thereby the definition of ψ2i , we distinguish four cases: Case 1: There is x ≥ 1 such that (A) happens. Then Si (ix+1 ) = 2k for a k = i. Hence, ψSi (ix+1 ) (0) = ψ2k (0) = k = i and Si outputs an inconsistent hypothesis on ix+1 U , a contradiction. Case 2: There is x ≥ 1 such that (C) happens. Then Si (ix+1 ) = 2i, but ψ2i (x) ↑. Thus, Si is inconsistent on ix+1 U , a contradiction. Case 3: There is x ≥ 1 such that (D) happens. Then Si is undefined on ix+1 U , a contradiction. Case 4: For all x ≥ 1 (B) happens. Then ψ2i = i∞ ∈ U and Si outputs almost always odd hypothesis on i∞ . Since the odd indices in ψ belong to nonconstant functions, Si does not converge to a correct hypothesis on i∞ ∈ U ,a contradiction. The hypothesis space ψ constructed in the last proof satisfies the conditions (a) and (b) of the characterization theorem given in the introduction. Hence, ψ is a somewhat more “natural” hypothesis space (cf. proof of Theorem 11) biased towards an inference type, namely LIM, although only for a certain class U .
Changing the Inference Type – Keeping the Hypothesis Space
97
Table 1. Overview of the scope of (I → J ). A + means maximal scope, a − minimal. For a set M of classes, + M means the scope is a superset of M; − M means the intersection of scope and M contains only the finite classes. Finally, − U means, a counter-example for the maximality of the scope exists. I
\
J
FIN
CONS-CP CP
CONS-TOTAL TOTAL CONS LIM
FIN
+
+
+
−U + CONS-CP
+
+
+
CONS-CP
+
+
+
+
+
+
+
CP
+
+
+
−U + CONS-CP
+
+
+
− NUM + − FIN∩CONS-CP
+
+
+
CONS-TOTAL − NUM − NUM − CONS-CP − FIN TOTAL
−
− NUM − FIN
− NUM − FIN
−U + CONS-CP
+
+
+
CONS
−
−
−
−
−
+
+
LIM
−
−
−
−
−
−U + + CP
BC
−
−
−
−
−
−
6
−
Overview of Results Concerning (I → J )
Table 1 tries to present the numerous results of this paper in a clear manner. Results stated in that table, but not proved within this paper, can be obtained via techniques similar to those presented in the previous sections. Note that the scope of (I → J ) has not been fully characterized for every such property. Acknowledgments. This paper is based on my diploma thesis at the University of Kaiserslautern. It is a pleasure for me to thank Sandra Zilles and Rolf Wiehagen for their continuous support and helpful advice. Many thanks also to Thomas Zeugmann for many valuable hints and insights. Finally, I wish to thank the members of the Program Committee of the ALT 2003 for carefully reading the paper. In particular I am indebted to the anonymous reviewer who provided the proof of Theorem 18.
References 1. D. Angluin, C. Smith. Inductive inference: theory and methods. ComputingSurveys 15, 237–269, 1983. 2. J. Barzdin. Inductive inference of automata, functions and programs, Proceedings International Congress of Math., 455–460, Vancouver, 1974. 3. J.M. Barsdin. Dve Teoremui o predjelnom sintjese funkzii. Teorija algorithmov i programm I 82–88, Latviiskii. Gosudarstvenyi univ., Riga 1974.
98
F. Balbach
4. J.M. Barsdin, R.W. Freiwald: Prognosirovanje i predjelnyi sinfjes effektivno peretschislimyich klassov funkzii. Teorija algorithmov i programm I 101–111, Latviiskii Gosudarstvenyi universitjet, Riga 1974. 5. H.-R. Beick. Einige qualitative Aspekte bei der Erkennung von Klassen allgemein rekursiver Funktionen. Diplomarbeit, Humboldt-Universit¨ at, Berlin, 1979. 6. M. Blum. A machine independent theory of the complexity of recursive functions, Journal of Association for Computing Machinery, Vol. 11, 322–336, April 1967. 7. L. Blum, M. Blum. Toward a Mathematical Theory of Inductive Inference, Information and Control 28, 125–155, 1975. 8. J. Case, C. Smith. Comparison of Identification Criteria for Machine Inductive Inference, Theoretical Computer Science 25, 193–220, 1983. 9. R. Freivalds, E. B. Kinber, R. Wiehagen. How Inductive Inference Strategies Discover Their Errors, Information and Computation 118, 208–226, 1995. 10. R. Freivalds. Inductive inference of recursive functions: Qualitative theory, (J. B¯ arzdi¸ nˇs and D. Bjorner, Eds.) Baltic Computer Science, LNCS 502, 77–110, Springer-Verlag, 1991. 11. E. M. Gold. Language identification in the limit, Information and Control 10, 447–474, 1967. 12. K. P. Jantke, H.-R. Beick. Combining Postulates of Naturalness in Inductive Inference, Elektronische Informationsverarbeitung und Kybernetik 17, 465–484, 1981. 13. S. Jain, D. Osherson, J. S. Royer, A. Sharma. Systems that Learn: An Introduction to Learning Theory, second edition, MIT Press, Cambridge, Massachusetts, 1999. 14. M. Machtey, P. Young. An Introduction to the General Theory of Algorithms, North-Holland, New York, 1978. 15. H. Rogers. Theory of Recursive Functions and Effective Computability. McGraw– Hill, New York, 1967. 16. W. Stein. Konsistentes und inkonsistentes Lernen im Limes. Dissertation, Universit¨ at Kaiserslautern, 1998. 17. F. Stephan, T. Zeugmann. Learning Classes of Approximations to Non-Recursive Functions, Theoretical Computer Science Vol. 288, Issue 2, 309–341, 2002. (Special Issue ALT ’99). 18. R. Wiehagen. Limes-Erkennung rekursiver Funktionen durch spezielle Strategien, Elektronische Informationsverarbeitung und Kybernetik 12 1/2, 93–99, 1976. 19. R. Wiehagen. Zur Theorie der algorithmischen Erkennung, Dissertation B, Sektion Mathematik, Humboldt-Universit¨ at, Berlin, 1978. 20. R. Wiehagen, T. Zeugmann. Learning and Consistency, (K. P. Jantke, S. Lange, Eds.) Alg. Learning for Knowledge-Based Systems, LNAI 961, 1–24, Springer, 1995.
Robust Inference of Relevant Attributes Jan Arpe and R¨ udiger Reischuk Institut f¨ ur Theoretische Informatik, Universit¨ at zu L¨ ubeck Wallstr. 40, 23560 L¨ ubeck, Germany {arpe/reischuk}@tcs.uni-luebeck.de
Abstract. Given n Boolean input variables representing a set of attritubes, we consider Boolean functions f (i.e., binary classifications of tuples) that actually depend only on a small but unknown subset of these variables/attributes, in the following called relevant. The goal is to determine the relevant attributes given a sequence of examples - input vectors X and corresponding classifications f (X). We analyze two simple greedy strategies and prove that they are able to achieve this goal for various kinds of Boolean functions and various input distributions according to which the examples are drawn at random. This generalizes results obtained by Akutsu, Miyano, and Kuhara for the uniform distribution. The analysis also provides explicit upper bounds on the number of necessary examples. They depend on the distribution and combinatorial properties of the function to be inferred. Our second contribution is an extension of these results to the situation where attribute noise is present, i.e., a certain number of input bits xi may be wrong. This is a typical situation, e.g., in medical research or computational biology, where not all attributes can be measured reliably. We show that even in such an error-prone situation, reliable inference of the relevant attributes can be performed, because our greedy strategies are robust even against a linear number of errors.
1
Introduction
In many data mining applications, one is faced with the situation that a binary classification of elements with a large number of attributes only depends on a small subset of these attributes. A central task is then to infer these relevant attributes from a given input sample consisting of a series of examples X(k) = (x1 (k), . . . , xn (k)) with classifications y(k) for k = 1, 2, . . . , m, i.e., one wants to find a set of variables xi1 , . . . , xid such that the sample can be explained by a function f : {0, 1}n → {0, 1} that depends only on these d variables. A function f is said to explain the sample, if f (x1 (k), . . . , xn (k)) = y(k) for all k. Moreover, since real data usually contain noise, it is of particular interest to design algorithms that in some sense behave ‘robustly’ with respect to input disturbances. When inferring relevant attributes, two natural questions that can be asked:
Supported by DFG research grant Re 672/3.
R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, pp. 99–113, 2003. c Springer-Verlag Berlin Heidelberg 2003
100
J. Arpe and R. Reischuk
1. Given a fixed sample of an unknown concept, what is the minimum number of variables that explain the sample? 2. How many examples does one need to generate in order to find out the actual relevant attributes? The first question gives rise to an optimization problem introduced in Sect. 3, whereas the second one can be considered as an algorithmic learning problem. In both cases, however, the key task is to infer relevant variables from a sample. Thus our goal is to design efficient algorithms that find a small set of variables explaining the input sample. Akutsu and Bao [2] proposed a greedy algorithm based on a well-known greedy strategy for the Set Cover problem (see [12]). Akutsu, Miyano, and Kuhara [3] describe an efficient implementation of this approach and give an average case analysis of the algorithm for two special types of functions, namely AND and OR of arbitrary literals, under the uniform distribution of input examples. In Sect. 4, we simplify the greedy strategy. We call this strategy Greedy Ranking and show that its performance is similar to the one obtained in [3]. In Sect. 5, the average case analysis of [3] is generalized in two respects: to a broader class of functions and to weaker assumptions on the input distributions. It turns out that a modification of our approach, namely taking the smallest sets of the ranking, may also be useful for some classes of functions and input distributions. We call this strategy Modest Ranking, since its ‘modest’ behavior of first selecting the smallest sets is in contrast to the greedy strategy of taking the largest sets. We apply these very general results to some typical input distributions and some specific functions of major interest (e.g., monomials, clauses, and threshold functions) in Sect. 6. After these investigations we turn to the ‘real case’ of samples that contain partial errors. In Sect. 7, we assume that for each attribute there is a certain (generally unknown) error probability δi that the value for this attribute xi in an input vector is flipped. This noise model called product random attribute noise ([11]) has been applied to the PAC learning model as well. Note that it is quite different from the classification noise model [5]. We show that for some δ > 0 depending only on combinatorial properties of the function f to be inferred and on the probability distribution according to which the samples are generated, one can tolerate any constant fractions δi ≤ δ of such erroneous bits and still infer the relevant attributes successfully with high probability using the ranking strategies. In addition to their general ability of robustly inferring relevant attributes, the number of examples needed to handle disturbed inputs only grows at most by a factor of 4. Finally, in Sect. 8, we consider a different approach for the parity function, since the ranking strategies do not work in this case. Inferring relevant attributes is related to finding association rules (also called functional dependencies/relations) – a well-studied problem (e.g., see [1,15]). In the variant considered in this paper, the target attribute Y is fixed as in [2,3]. The goal of efficiently inferring concepts with many irrelevant attributes (so-called attribute-efficient learning) has attracted much attention in the past (e.g., see [13,8,9,17]). Most authors consider the mistake-bounded model. In this
Robust Inference of Relevant Attributes
101
on-line setting, one tries to minimize the number of examples for which the current hypothesis turns out to be wrong. There are several ways known how to convert on-line algorithms with low mistake bounds into efficient PAC learning algorithms (see [4,14,13]). In this paper, we consider the finite exact learning model: From a randomly selected sample of small size, we have to compute a single hypothesis that with high probability has to be correct (with accuracy 1). Recently, Mossel, O’Donnell, and Servedio [16] have introduced an algorithm that exactly learns the class of concepts f with n input variables and d relevant attributes (also called d-juntas) under uniform distribution with confidence 1−δ ω in time (nd ) ω+1 · poly(n, 2d , log(1/δ)), where ω < 2.376 is the matrix multiplication exponent. The Target Ranking algorithm we introduce runs in time O(m2 n) on samples of size m. In order to achieve confidence 1 − δ, we roughly need c · log(1/δ) · log n examples, where c depends on the base function f˜ (i.e., the restriction of f to its relevant variables), the number of relevant attributes d, and the probability distribution according to which the examples are drawn. In particular, restricting to the uniform distribution, for arbitrary f satisfying a certain statistical property, c can be bounded by poly(2d ). In this case we are able to exactly infer the relevant attributes with confidence 1 − δ in time n · poly(log n, 2d , log(1/δ)). Due to space limitations, most proofs have to be omitted. Details are presented in [7].
2
Preliminaries
A concept is a Boolean function f : {0, 1}n → {0, 1}, a concept class is a set of concepts. A concept f : {0, 1}n → {0, 1} depends on variable xi , if the two (n−1)ary subfunctions fxi =0 and fxi =1 with variable xi fixed to 0 and 1 respectively are not identical. If f depends on xi , then attribute xi is called relevant for f , otherwise irrelevant. We denote the set of relevant (resp. irrelevant) attributes by V + (f ) (resp. V − (f )). If f is clear from the context, we just write V + and V − . We denote by f˜ the restriction of f to its relevant variables and call it the base function of f . An example is a vector (x1 , . . . , xn ; y) ∈ {0, 1}n+1 . It is an example for f , if y = f (x1 , . . . , xn ). The values of x1 , . . . , xn are called variable or attribute assignments, whereas the value for y is called a label. A sequence (x1 (k), . . . , xn (k); y(k)) (k = 1, . . . , m) of examples for f is called a sample for f of size m, and f is said to explain the sample. A sample T is a sequence of examples such that there exists some f that explains the sample. If f depends only on variables from the set {xi1 , . . . , xid }, then we also say that these variables explain T . A sample is stored in a matrix each line of which represents x1 (1) . . . xn (1) | y(1)) .. .. ∈ {0, 1}m×(n+1) , where .. one example: T = (X; y) = ... . . . x1 (m) . . . xn (m) | y(m) X is the submatrix consisting of the variable assignments in the examples, and y is the column vector containing the labels of the examples. A sample T may contain a certain combination of attributes several times. Then, of course, it is necessary that for k = l the following implication holds:
102
J. Arpe and R. Reischuk
X(k) = X(l)
=⇒
y(k) = y(l) .
(1)
Indeed, if (1) does not hold for some k = l, then by definition, T is not a sample. In the noisy case, however, it may well be that different combinations of attributes yield different labels, but due to false measurements of the attributes, the values for x1 , . . . , xn all look the same. We assume that the examples of a sample T are drawn according to a fixed probability distribution p : {0, 1}n → [0, 1], and we say that T is generated according to p. Definition 1. Let (X; y) ∈ {0, 1}m×(n+1) be a sample. The corresponding functional relations graph is a bipartite labeled graph defined as follows. The vertices are {1, . . . , m}, the edges are S = {{k, l} | y(k) = y(l)}. Each edge {k, l} is labeled by the set of variables xi such that xi (k) = xi (l). The set of edges with a label containing variable xi is denoted by Si = {{k, l} ∈ S | xi (k) = xi (l)}. Proposition 1. Let T = (X; y) ∈ {0, 1}m×(n+1) be a sample and {i1 , . . . , id } ⊆ {1, . . . , n}. Then the following statements are equivalent: (a) xi1 , . . . , xid explain T . (b) For each pair k, l ∈ {1, . . . , m} such that y(k) = y(l) there exists r ∈ {1, . . . , d} such that xir (k) = xir (l). (c) S = Si1 ∪ . . . ∪ Sid .
3
Approximability
Consider the following optimization problem: Inference of Relevant Attributes (INFRA) Instance: sample T = (X; y) = (x1 (k), . . . , xn (k); y(k))k=1,... ,m ∈ {0, 1}m×(n+1) Solution: a function f : {0, 1}n → {0, 1} such that T is a sample for f (i.e., y(k) = f (x1 (k), . . . , xn (k)) for all k ∈ {1, . . . , m}) Measure: |V + (f )| Goal: minimize |V + (f )|
Note that in order to find a small set of explaining attributes for an INFRA instance, we do not have to explicitly define a corresponding concept f , but it is enough to find a set of attributes xi1 , . . . , xid such that for k, l ∈ {1, . . . , m} with y(k) = y(l) there exists r ∈ {1, . . . , d} with xir (k) = xir (l) by Proposition 1. In order to obtain results on the approximability of INFRA, we consider the well-studied Set Cover problem. Note that Proposition 1 yields a reduction from INFRA to Set Cover. Based on this fact, Akutsu and Bao [2] have proved the following theorem: Theorem 1 ([2]). INFRA can be approximated in polynomial time within a factor of 2 ln m + 1.
Robust Inference of Relevant Attributes
103
The next claim is a slightly stronger version of Theorem 8 in [2], since we consider the special case of INFRA for Boolean functions. Proposition 2. Set Cover is reducible to INFRA via a polynomial time computable approximation factor preserving reduction. Applying a result from [10], we obtain the following lower bound: Theorem 2. For any ε > 0, INFRA cannot be approximated within a factor of (1 − ε) ln m unless NP ⊆ DTIME(nO(log log n) ). Therefore, when faced with the INFRA problem, the best one can hope for are efficient approximation algorithms with a nonconstant approximation ratio or fast algorithms providing correct results for ‘most’ inputs. In the rest of this paper, we investigate the latter challenge.
4
From Greedy to Ranking
Let us start with the algorithm discussed in [3] which is presented in Fig. 1. It makes use of the reduction from INFRA to Set Cover given by Proposition 1 and applies a well-known greedy approach to the Set Cover instance obtained. Johnson [12] first analyzed this approach for Set Cover.
input (x1 (k), . . . , xn (k); y(k))k=1,... ,m V := {x1 , x2 , . . . , xn }; S := {{k, l} | y(k) = y(l)} while S = ∅ do for i = 1 to n do Si := {{k, l} ∈ S | xi (k) = xi (l)} find an xi ∈ V with maximum |Si | output xi S := S \ Si ; V := V \ {xi } Fig. 1. Algorithm Greedy
We apply some modifications of this algorithm and analyze their effects. The strategy is based on a ranking of the sets S1 , . . . , Sn by their cardinalities which is done by the procedure Rank Sets, see Fig. 2.
for i = 1 to n do Si := {{k, l} ∈ S | xi (k) = xi (l)} compute π : {1, . . . , n} → {1, . . . , n} such that |Sπ(1) | ≥ |Sπ(2) | ≥ . . . ≥ |Sπ(n) | Fig. 2. Procedure Rank Sets
The results may be worse in some cases, since the new greedy approach is based on a single static ranking. However, we show that the ranking still yields
104
J. Arpe and R. Reischuk
properties similar to Greedy and in addition performs quite robustly when confronted with attribute noise. Greedy Ranking (see Fig. 3) outputs the variables xi with maximum |Si | until these Si ’s cover the whole edge set S. In contrast to Greedy, Greedy Ranking does not recompute the sets Si in each step. Given a concept f , the Greedy Ranking algorithm works correctly, if with high probability, the sets Si for xi ∈ V + are larger than the sets Si for xi ∈ V − . On the other hand, if the converse is the case, i.e., if the sets Si for relevant variables are likely to be smaller than the sets Si for the irrelevant variables, we should make use of an algorithm that outputs the variables corresponding to the smallest sets Si . Instead of being greedy, this algorithm rather behaves modestly, so we call it Modest Ranking (see also Fig. 3).
input (x1 (k), . . . , xn (k); y(k))k=1,... ,m Rank Sets S := {{k, l} | y(k) = y(l)} i := 1 / i = n while S = ∅ do output xπ(i) S := S \ Sπ(i) i := i + 1 / i := i − 1 Fig. 3. Algorithms Greedy/Modest Ranking
Note that, given a sample, all three algorithms terminate after a finite number of steps since by property (1), each pair {k, l} ∈ S belongs to some Si . Clearly, all algorithms presented here compute a cover of S. Thus by the reduction given in Proposition 1, the algorithms work correctly for the optimization problem, i.e., they output sets of variables that explain the input sample. It is not hard to construct instances showing that in general none of the algorithms is superior to the others in terms of finding small sets of explaining variables. It may be the case that an input sample for some concept f can be explained by a proper subset of the relevant variables for f . In case the number d of relevant variables is a priori known, we can overcome this problem by giving d as additional input to the algorithms and output the d variables with the largest (resp., smallest) sets Si . This is done by the Target Ranking and the Modest Target Ranking algorithms (see Fig. 4).
input (x1 (k), . . . , xn (k); y(k))k=1,... ,m , d Rank Sets S := {{k, l} | y(k) = y(l)} for i = 1 to d do output xπ(i) / output xπ(n−i+1) Fig. 4. Algorithms (Modest) Target Ranking
Robust Inference of Relevant Attributes
105
Moreover, if we only have an a priori upper bound d on the number of relevant variables, then the target ranking algorithms output a set of d variables such that the d relevant ones are most likely among them. Definition 2. Given a concept f with relevant variables xi1 , . . . , xid and a sample T for f , we say that an algorithm succeeds in a step, if the output generated in that step is a relevant variable of f . The algorithm is said to be correct, if it is successful in all steps it makes. It is complete, if it finds all relevant variables. Finally, an algorithm is said to be successful, if it is both correct and complete. The following properties are easy to show: Lemma 1. Let f depend on d variables, and let T be a sample for f . (a) If Target Ranking (resp., Modest Target Ranking) is correct on input (T, d), then it is successful, too. (b) If Target Ranking (resp., Modest Target Ranking) is complete on input (T, d), then it is also successful. (c) If Target Ranking (resp., Modest Target Ranking) is successful on input (T, d), then Greedy Ranking (resp., Modest Ranking) is correct on input T . In order to uniquely recognize the relevance of some variable xir , there has to be an edge in the functional relations graph whose only relevant label is xir . Thus, independently of the used learning algorithm, a necessary (but not sufficient) condition to infer the relevance of xir is the occurrence of two examples k, l in the input sample with xir (k) = 0 and xir (l) = 1, but with identical values for all other relevant attributes. By the birthday paradox, already for the √ uniform distribution roughly 2d−1 examples are necessary to guarantee such an occurrence. This shows that in order for any algorithm to be complete, Ω(2d/2 ) examples have to be provided due to information theoretic reasons.
5
Probabilistic Analysis of the Ranking Strategies
Let f : {0, 1}n → {0, 1} be a concept with relevant variables xi1 , . . . , xid , and p be a probability distribution. For x, y ∈ {0, 1}, we denote by ‘xi = x’ the set of examples with xi = x, and by ‘f = y’ the set of examples with f (x1 , . . . , xn ) = y. (x,y) For i ∈ {1, . . . , n} and x, y ∈ {0, 1} define the probability αi that a randomly drawn example (x1 , . . . , xn ; f (xi1 , . . . , xid )) has xi = x and f (xi1 , . . . , xid ) = y, (x,y) = Pr(xi = x ∧ f = y). i.e., αi Let T = (X; y) be a sample of size m for f generated according to p. Define (x,y) K = {1, . . . , m} and Ki = {k ∈ K | xi (k) = x and y(k) = y}. The situation (x,y) | can is depicted in Fig. 5. Since all examples are identically distributed, |Ki (x,y) be considered as a binomially distributed random variable with parameters αi (x,y) (x,y) and m. Analogously to the αi ’s, we denote by βi the corresponding relative (x,y) (x,y) = |Ki | / |K|, and define αi = frequencies in the input sample, i.e., βi (0,0) (1,1) (1,0) (0,1) (0,0) (1,1) (1,0) (0,1) αi αi + αi αi and βi = βi βi + βi βi . It holds |Si | = βi m2 , and for large m, we get the approximation |Si | ≈ αi m2 .
106
J. Arpe and R. Reischuk y(k) = 0
y(k) = 1
(0,0)
(0,1)
Ki
Ki
xi (k) = 0 xi (k) = 1 (1,0)
(1,1)
Ki
Ki edges in Si edges not in Si
Fig. 5. Partition of K with respect to variable xi 1 Lemma 2. For fixed i ∈ {1, . . . , n} and arbitrary 0 ≤ δ ≤ 10 , it holds that 1 2 Pr(|Si | − αi m2 ≥ δm2 ) ≤ 8e− 3 δ m .
Proof. The proof requires lengthy calculations and case distinctions. It is based on standard Chernoff bound techniques and can be found in [7].
The following theorem provides very general conditions that guarantee the success of the ranking algorithms with respect to a concept f : Theorem 3. Let f : {0, 1}n → {0, 1} depend on xi1 , . . . , xid , let T be a sample for f generated according to a probability distribution p : {0, 1}n → [0, 1], and let c > 0. (a) If min{αi | xi ∈ V + } > max{αj | xj ∈ V − }, then with probability 1 − n−c , Target Ranking is successful on input (T, d), provided that m ≥ 12ε−2 ((c + 1) ln n + ln 8), where ε = min { min{αi | xi ∈ V + } − max{αj | xj ∈ V − } , 1/5}. (b) If max{αi | i ∈ V + } < min{αj | j ∈ V − }, then with probability 1 − n−c , Modest Target Ranking is successful on input (T, d), provided that m ≥ 12ε−2 ((c + 1) ln n + ln 8), where ε = min { min{αj | xj ∈ V − } − max{αi | xi ∈ V + } , 1/5}. Proof. We only prove part (a), 2 since (b) can be +done analogously. Let t = 1 + − min α + max α it holds that i j m . Then for xi ∈ V i∈V j∈V 2
ε ε 2 m ≤ Pr |Si | − αi m2 ≥ m2 Pr(|Si | ≤ t) ≤ Pr |Si | ≤ αi − 2 2 2 1 2 − 13 ( 2ε ) m − 12 ε m ≤ 8e = 8e , where the last inequality is due to Lemma 2. Similarly, for xj ∈ V − , 1
2
Pr(|Sj | ≥ t) ≤ 8e− 12 ε
m
.
Robust Inference of Relevant Attributes
107
Target Ranking is successful on input (T, d) iff it is correct in all of its d steps. This is exactly the case, if the largest d sets Si correspond to the relevant variables, i.e., if minxi ∈V + |Si | > maxxj ∈V − |Sj |. We have
Pr min+ |Si | > max− |Sj | ≥ Pr min+ |Si | > t ∧ max− |Sj | < t xi ∈V xj ∈V xi ∈V xj ∈V + = 1 − Pr ∃xi ∈ V |Si | ≤ t ∨ ∃xj ∈ V − |Sj | ≥ t ≥1− Pr (|Si | ≤ t) + Pr (|Sj | ≥ t) xi ∈V +
xj ∈V −
1 2 − 12 ε m
≥ 1 − 8ne If m ≥
12 ε2 ((c + 1) ln n + ln 8),
1 2 − 12 ε m
then 8ne
.
≤ n−c , thus the claim follows.
As Theorem 3 is stated for a general setting, let us now consider some typical input distributions and simplify its conditions in these cases. • Independent Attributes (IA) Suppose that the values for the xi ’s (i = 1, . . . , n) are generated independently of each other, say with Pr(xi = 1) = pi ∈ [0, 1] (thus Pr(xi = 0) = 1 − pi ). Then we say that the sample is IA(p1 , . . . , pn )-generated. Lemma 3. Let T be an IA(p1 , . . . , pn )-generated sample for f . Then, for xj ∈ V − , we have αj = 2pj (1 − pj ) Pr(f = 0) Pr(f = 1). • Independent Equiprobable Attributes (IEA) If T is IA(p1 , . . . , pn )-generated with p1 = . . . = pn = q, then we say that T is IEA(q)-generated. Lemma 4. Let T be an IEA(q)-generated sample for f . (a) For each xj ∈ V − , it holds αj = 2q(1−q) Pr(f = 0) Pr(f = 1). In particular, αj is independent of xj ∈ V − . We denote the common value of these αj ’s by α− in this case. (b) If f is symmetric, then the αi ’s with xi ∈ V + are also independent of i. We denote the common value of these αi ’s by α+ in this case. From the previous lemma and Theorem 3 we immediately obtain the following result on the successfulness of the target ranking algorithms when applied to symmetric Boolean functions. Corollary 1. For f with a symmetric base function f˜, three cases can occur: – α+ > α− : O(log n) input examples suffice such that Target Ranking is successful with high probability. – α+ < α− : O(log n) input examples suffice such that Modest Target Ranking is successful with high probability. – α+ = α− : No success ratios can be guaranteed for the ranking algorithms, regardless of how many input examples are provided. • Uniformly Distributed Attributes (UDA) If the examples are uniformly distributed, i.e., if a sample T is IEA( 12 )-generated, then we say that T is UDA-generated.
108
6
J. Arpe and R. Reischuk
Inferring Specific Concepts
We now consider several basic Boolean functions. Theorem 4 (AND-function). Let {i1 , . . . , id } ⊆ {1, . . . , n}, let f : {0, 1}n → {0, 1} be defined by f (x1 , . . . , xn ) = xi1 ∧ . . . ∧ xid , and let T be an IEA(q)generated sample for f . (a) If q ≤ 12 , then it holds that α+ > α− . Thus the success ratio for Target Ranking may be raised arbitrarily close to 1 by choosing a large enough sample size m ∈ O(log n). (b) If q = 12 , then it holds that α+ −α− = 2−2d−1 > 0. Thus the success ratio for Target Ranking may be raised arbitrarily close to 1 by choosing a large enough sample size m ∈ O(log n) with the constant being of order 24d . (c) If q > 12 , then for sufficiently large d, we have α+ < α− . Thus the success ratio for Modest Target Ranking may be raised arbitrarily close to 1 by choosing a large enough sample size m ∈ O(log n). The same ideas apply to the OR-function with q substituted by 1 − q. Sketch of proof: We have Pr(f = 1) = q d and Pr(f = 0) = 1 − q d , thus α− = (0,0) (1,1) 2q d+1 (1−q)(1−q d ) by Lemma 4 (a). Furthermore, αi = 1−q, αi = q d , and (0,1) αi = 0, yielding α+ = q d (1 − q). Hence, α+ > α− ⇐⇒ q(1 − q d ) < 12 , from which (a) and (c) follow. Similarly, (b) can be shown by plugging in q = 1/2.
Theorem 5 (Monomials). Let {i1 , . . . , id } ⊆ {1, . . . , n}, let lr ∈ {xir , ¬xir } for each r ∈ {1, . . . , d}, and let f : {0, 1}n → {0, 1} be defined by f (x1 , . . . , xn ) = l1 ∧ . . . ∧ ld . Let T be a UDA-generated sample for f . Then Target Ranking is successful with high probability provided that a sample of size m ∈ Ω(24d · log n) is given. Sketch of proof: The analysis is similar to the one in Theorem 4 for q = 12 . In particular, α+ − α− = 2−2d−1 > 0. Now the claim follows from Theorem 3.
Akutsu, Miyano, and Kuhara [3] showed a similar result for the Greedy algorithm. Note that for monomials under uniform input distribution, 2d rows are necessary in order to obtain (in expectation) at least one example with label 1. (If there is no such example, then the sample can be explained by the constant zero function.) It is easy to see that being able to infer the relevant attributes of a function f , the same holds for its negation. In particular, the result for monomials translates to clauses. The case of negating individual attribute values is more complex. At least in case of the uniform distribution the inferability is not effected. Theorem 6 (Threshold functions). Let {i1 , . . . , id } ⊆ {1, . . . , n}, 1 ≤ t ≤ d, d and f : {0, 1}n → {0, 1} be defined by f (x1 , . . . , xn ) = 1 iff r=1 xir ≥ t. Let T be a UDA-generated sample for f . Then Target Ranking is successful with
−4 24d log n . high probability, provided that m ∈ Ω dt
Robust Inference of Relevant Attributes
109
2 −2d−1 Sketch of proof: A straightforward calculation yields α+ − α− = d−1 ·2 . t−1 Now Theorem 3 yields the claim.
If t = d in the previous theorem, then f = AND, and we recover our result from Theorem 4, part (b). Moreover, under uniformly distributed inputs, the gap between α+ and α− for threshold functions is smallest for t ∈{1, d}. The largest d−1 ∈ Θ( √1d 2d ), such gap is reached for t = d2 , the majority function. Since t/2−1 + − −1 we have α − α ∈ Θ(d ). Applying Theorem 3, this proves the following Corollary 2 (Majority function). Let f : {0, 1}n → {0, 1} such that its base function f˜ : {0, 1}d → {0, 1} is the majority function. Then Target Ranking is succeessful with high probability, provided that m ∈ Ω(d2 · log n). For symmetric Boolean functions, one cannot always guarantee α+ = α− , even for UDA-generated samples. A simple counter-example is the parity function f (x1 , . . . , xn ) = (xi1 + . . . + xid ) mod 2 for which αi = 18 for all i ∈ {1, . . . , n}, no matter whether xi ∈ V + or xi ∈ V − . Thus the ranking strategies do not work for the parity function. We provide an alternative solution for such concepts in Sect. 8.
7
Robust Inference
As real data usually contain noise, our ultimate goal is to handle cases in which the attribute values underly certain disturbances. More precisely, we assume that in each input example, attribute xi is flipped with probability δi , i.e., an algorithm obtains xi (k) instead of the correct value xi (k) with probability δi . We call the resulting set of disturbed examples a δ-disturbed sample, where δ = (δ1 , . . . , δn ). Note that this assumption introduces a linear number of faults (with respect to the number of attributes). Fortunately, it can be shown that the ranking algorithms still perform well, if they are given such disturbed samples. The key idea in this case is to examine how much the sets Si computed by the ranking algorithms deviate from the Si ’s intended by the real data. We denote the sets derived from the disturbed data by Sˆi . Furthermore, for i ∈ {1, . . . , n}, let Fi = {k ∈ {1, . . . , m} | the input table contains xi (k) instead of xi (k)} . The following lemma is analogous to Lemma 2: 1 Lemma 5. Let i ∈ {1, . . . , n} with δi ≤ 30 . Then, for ε such that 6δi ≤ ε ≤ 15 , 1 2 2 2 ˆ it holds that Pr |Si | − αi m ≥ εm ≤ 9e− 12 ε m . Sketch of proof: We use the inequality |Sˆi | − αi m2 ≤ |Sˆi | − |Si | + |Si | − αi m2 and compute the probability that each of the summands on the right hand side is bounded by 2ε m2 . Combinatorial investigations yield |Sˆi |−|Si | ≤ m|Fi |+ 12 |Fi |2 . In particular, if |Fi | ≤ 13 εm, then |Sˆi | − |Si | ≤ 12 εm2 . From standard Chernoff 1 ε 1 bounds, it follows that Pr |Fi | ≥ 3ε m ≤ e− 3 · 6 ·m = e− 18 εm , since δi < 6ε (|Fi |
110
J. Arpe and R. Reischuk
can be considered as a binomially distributed random variable with parameters δi and m). Now ε
ε Pr |Sˆi | − αi m2 ≥ εm2 ≤ Pr |Sˆi | − |Si | ≥ m2 ∨ |Si | − αi m2 ≥ m2 2 2
2 ε − 13 ( 2ε ) m + 8e ≤ Pr |Fi | ≥ 3 1 1 2 1 2 ≤ e− 18 εm + 8e− 12 ε m ≤ 9e− 12 ε m , 1 1 2 εm > 12 ε m for ε ≤ 15 .
where we make use of Lemma 2 and the fact that 18 Besides the general information theoretic problem that a sample may already be explained by a proper subset of the relevant variables, just the opposite phenomenon can occur due to disturbances: Sˆ may not be covered by Sˆ1 , . . . , Sˆn , so Greedy, Greedy Ranking, and Modest Ranking – as introduced in Sect. 4 – do not terminate on the corresponding input samples. Therefore, when faced with the disturbed situation, we modify the algorithms as follows: All edges that do not belong to any of the computed Sˆi ’s are ignored, i.e., we compute a new set Sˆnew = Sˆ \ {{k, l} ∈ Sˆ | ∀i ∈ {1, . . . , n} : {k, l} ∈ Sˆi }. The edges removed are exactly those connecting two example nodes with identical attribute values but different labels. All algorithms make use of this set Sˆnew instead of ˆ However, this modification does not effect our analysis, so we continue by S. ˆ In the noisy scenario, Lemma 1 has to be modified as follows: writing S.
Lemma 6. Let f be a concept depending on d variables, and T a δ-disturbed sample for f such that Target Ranking is successful on (T, d). If Greedy Ranking outputs at most d variables on input T , then it is correct. Otherwise, the first d variables output by Greedy Ranking are the relevant ones. We now state our main theorem for the case of disturbed samples: Theorem 7. Let f : {0, 1}n → {0, 1} with relevant variables xi1 , . . . , xid , and let δ = (δ1 , . . . , δn ) ∈ [0, 1]n . Let T be a δ-disturbed sample for f generated according to a probability distribution p : {0, 1}n → [0, 1], and let c > 0. 1 ε for all k ∈ (a) If min{αi | xi ∈ V + } > max{αj | xj ∈ V − } and δk ≤ 12 −c {1, . . . , n}, then with probability 1 − n , Target Ranking is successful on input (T, d), provided that
m ≥ 48 ε−2 ((c + 1) ln n + ln 9) , where ε = min{ min{αi | i ∈ V + } − max{αj | j ∈ V − } , 2/5} . 1 ε for all k ∈ (b) If max{αi | i ∈ V + } < min{αj | j ∈ V − } and δk ≤ 12 {1, . . . , n}, then Modest Target Ranking is successful on input (T, d), provided that m ≥ 48 ε−2 ((c + 1) ln n + ln 9) , where ε = min{ min{αj | j ∈ V − } − max{αi | i ∈ V + } , 2/5}. Proof. Extension of the analysis in the proof of Theorem 3. See [7].
Robust Inference of Relevant Attributes
111
We would like to stress that the algorithms have not been modified in any way in order to overcome the disturbances. In particular, we do not have to assume that the algorithms have any knowledge about the error probabilities δ1 , . . . , δn . Even more, the sample size required for Target Ranking only has to be enlarged by factor 4 in order to obtain the same success probability in case of a (small) constant percentage of errors in the input sample.
8
Inferring Relevant Attributes of the Parity Function
Throughout this section we identify {0, 1} with the two-element field GF(2) n and denote by ⊕ the sum operation in this field. Furthermore, we define |ξ| = i=1 ξi for ξ ∈ {0, 1}n (here the sum is taken in Z). Let f : {0, 1}n → {0, 1} be defined by f (x1 , . . . , xn ) = xi1 ⊕ . . . ⊕ xid for some set of variable indices I = {i1 , . . . , id } ⊆ {1, . . . , n}. Since we have seen at the end of Sect. 5 that ranking the variables according to their occurences in the functional relations graph does not work for the parity function, we present a different algorithm Parity Infer to find the relevant variables. The idea is simply to compute a solution of a system of linear equations associated with the input sample and then to infer from this solution a set of variables that can explain the sample.
input (X; y) ∈ {0, 1}m×(n+1) solve Xξ = y if there is no solution then output ‘sample contains wrong data’ else choose any solution ξ; output all xi ’s such that ξi = 1 Fig. 6. Algorithm Parity Infer
Let us again differenciate between the two aspects, the optimization problem INFRA(⊕) obtained by restricting the instances and the solutions of INFRA to samples for concepts whose base functions are parity functions – such functions can be uniquely described by the set V + of relevant variables – and on the other hand finding exactly the relevant variables of a given but unknown parity function provided that the sample size is large enough. Let T = (X; y) ∈ {0, 1}m×(n+1) . There is a one-to-one correspondence between solutions V + of the INFRA(⊕) instance T and the solutions ξ ∈ {0, 1}n for the system of linear equations Xξ = y given by ξi = 1 iff xi ∈ V + . The task of finding an optimal solution for an INFRA(⊕) instance is equivalent to finding a solution ξ of Xξ = y with minimum |ξ|. Since {xi | i ∈ I} is a solution for T , the system has at least one solution. Moreover, if X has full rank (i.e., rank(X) = n), then there is a unique solution which is of course also an optimal solution in this case. There is a well-known correspondence between the INFRA(⊕) and the Nearest Codeword problem. A Nearest Codeword instance consists of a matrix
112
J. Arpe and R. Reischuk
A ∈ {0, 1}n×r and a vector b ∈ {0, 1}n . A solution is a vector x ∈ {0, 1}r , and the goal is to minimize the Hamming distance of Ax and b (i.e., |Ax ⊕ b|). The obvious reduction is approximation factor preserving. Using a result of [6], this implies Theorem 8. For any ε > 0, INFRA(⊕) cannot be approximated within a factor 1−ε of 2log m unless NP ⊆ DTIME(npolylog(n) ). Despite this negative result, INFRA(⊕) can be solved efficiently on the average. We show that under certain assumptions the variables detected by Parity Infer are exactly the relevant ones with high probability. Theorem 9. Let f : {0, 1}n → {0, 1} such that its base function f˜ is a parity function, and let T = (X; y) ∈ {0, 1}m×n × {0, 1}n be a UDA-generated sample for f . If m ≥ n + k(2 ln k + 1) with k = c log n + 1 for some c > 0, then with probability 1 − n−c , Xξ = y has exactly one solution ξ. Corollary 3. Under the conditions described in Theorem 9, using sample size m = n + O(c log n log log n) Parity Infer is successful with probability 1 − n−c , where c may be chosen arbitrarily large.
9
Conclusions and Further Research
For inferring relevant Boolean valued attributes we have presented ranking algorithms, which are modifications of greedy algorithms proposed earlier. We have extended a negative approximability result to the restriction of only Boolean values and have improved a lower bound by using Feige’s result. General criteria for the successfulness of our algorithms have been established in terms of some statistical values (depending on the concept considered and the probability distribution). These results have been applied to a series of typical input distributions and specific functions. In case of monotone functions, a straightforward modification of our strategy restricts edge set Si to those edges {k, l} with xi (k) = y(k) = 0 and xi (l) = y(l) = 1. This halves the values αj for xj ∈ V − and thus may satisfy (or improve) the conditions of the main theorems for certain monotone functions. Next, we have investigated the case of noisy attribute values. We have shown that our algorithms still succeed with high probability, if their input contains a (small) constant fraction of wrong values. This desirable robustness property is achieved without requiring any specific knowledge about the likelihood of errors. One direction of future research could be to extend these results to more complex Boolean functions such as DNF formulas with a constant number of monomials. Furthermore, the case of robustly inferring relevant attributes of parity functions remains open. Another generalization would be the case that attributes and/or labels may take values from sets with more than two elements. Given an input instance, Greedy Ranking always outputs a proper solution that is capable of explaining the sample. However, if the input has some
Robust Inference of Relevant Attributes
113
disturbances, Greedy Ranking might indeed stop only after having chosen significantly more than the real number of relevant attributes. In such situations, one might be interested in algorithms that – rather than computing an exact solution for the given input data – output a simple solution fitting to an input instance that is in some sense ‘near’ to the input instance. Following Occam’s razor such a simple solution may be much more likely to explain the real phenomenon. Currently, we are working on a general framework for this setting.
References 1. R. Agrawal, T. Imielinski, and A. Swami, Mining Association Rules between Sets of Items in Large Databases. Proc. 1993 ACM SIGMOD Conf., 207–216. 2. T. Akutsu, F. Bao, Approximating Minimum Keys and Optimal Substructure Screens. Proc. 2nd COCOON, Springer LNCS 1090 (1996), 290–299. 3. T. Akutsu, S. Miyano, and S. Kuhara, A Simple Greedy Algorithm for Finding Functional Relations: Efficient Implementation and Average Case Analysis. TCS 292(2) (2003), 481–495. (see also Proc.3rd DS, Springer LNAI 1967 (2000), 86–98.) 4. D. Angluin, Queries and Concept Learning. Machine Learning 2(4) (1988), 319– 342, Kluwer Academic Publishers, Boston. 5. D. Angluin and P. Laird, Learning from noisy examples. Machine Learning 2(4) (1988), 343–370, Kluwer Academic Publishers, Boston. 6. S. Arora, L. Babai, J. Stern, and Z. Sweedyk, The Hardness of Approximate Optima in Lattices, Codes, and Systems of Linear Equations, J. CSS 54 (1997), 317– 331. 7. J. Arpe, R. Reischuk, Robust Inference of Relevant Attributes. Techn. Report, SIIM-TR-A 03-12, Univ. L¨ ubeck, 2003, available at http://www.tcs.mu-luebeck.de/TechReports.html. 8. A. Blum, L. Hellerstein, and N. Littlestone, Learning in the Presence of Finitely or Infinitely Many Irrelevant Attributes. Proc. 4th COLT ’91, 157–166. 9. A. Blum, P. Langley, Selection of Relevant Features and Examples in Machine Learning. Artificial Intelligence 97(1–2), 245–271 (1997). 10. U. Feige, A Threshold of ln n for Approximating Set Cover. J. ACM 45 (1998), 634–652. 11. S. Goldman, H. Sloan, Can PAC Learning Algorithms Tolerate Random Attribute Noise? Algorithmica 14 (1995), 70–84. 12. D. Johnson, Approximation Algorithms for Combinatorial Problems. J. CSS 9 (1974), 256–278. 13. N. Littlestone, Learning Quickly When Irrelevant Attributes Abound: A New Linear-threshold Algorithm. Machine Learning 4(2) (1988), 285–318, Kluwer Academic Publishers, Boston. 14. N. Littlestone, From On-line to Batch Learning. Proc. 2nd COLT 1989, 269–284. 15. H. Mannila, K. R¨ aih¨ a, On the Complexity of Inferring Functional Dependencies. Discrete Applied Mathematics 40 (1992), 237–243. 16. E. Mossel, R. O’Donnell, R. Servedio, Learning Juntas. Proc. STOC ’03, 206–212. 17. L. Valiant, Projection Learning. Machine Learning 37(2) (1999), 115–130, Kluwer Academic Publishers, Boston.
Efficient Learning of Ordered and Unordered Tree Patterns with Contractible Variables Yusuke Suzuki1 , Takayoshi Shoudai1 , Satoshi Matsumoto2 , Tomoyuki Uchida3 , and Tetsuhiro Miyahara3 1
2
Department of Informatics, Kyushu University, Kasuga 816-8580, Japan {y-suzuki,shoudai}@i.kyushu-u.ac.jp Department of Mathematical Sciences, Tokai University, Hiratsuka 259-1292, Japan [email protected] 3 Faculty of Information Sciences, Hiroshima City University, Hiroshima 731-3194, Japan {uchida@cs,miyahara@its}.hiroshima-cu.ac.jp
Abstract. Due to the rapid growth of tree structured data such as Web documents, efficient learning from tree structured data becomes more and more important. In order to represent structural features common to such tree structured data, we propose a term tree, which is a rooted tree pattern consisting of tree structures and labeled variables. A variable is a labeled hyperedge, which can be replaced with any tree. A contractible variable is an erasing variable which is adjacent to a leaf. A contractible variable may be replaced with a singleton vertex. A usual variable, called an uncontractible variable, is replaced with a tree of size at least 2. In this paper, we deal with ordered and unordered term trees with contractible and uncontractible variables such that all variables have mutually distinct variable labels. First we give a polynomial time algorithm for deciding whether or not a given term tree matches a given tree. Let Λ be a set of edge labels. Second, when Λ has more than one edge label, we give a polynomial time algorithm for finding a minimally generalized ordered term tree which explains all given tree data. Lastly, when Λ has infinitely many edge labels, we give a polynomial time algorithm for finding a minimally generalized unordered term tree which explains all given tree data. These results imply that the classes of ordered and unordered term trees are polynomial time inductively inferable from positive data.
1
Introduction
Due to the rapid growth of semistructured data such as Web documents, Information Extraction from semistructured data becomes more and more important. Web documents such as HTML/XML files have no rigid structure and are called semistructured data. According to Object Exchange Model [1], we treat semistructured data as tree structured data. Tree structured data such as HTML/XML files are represented by rooted trees with edge labels. In order to represent a tree structured pattern common to such tree structured data, we R. Gavald` a et al. (Eds.): ALT 2003, LNAI 2842, pp. 114–128, 2003. c Springer-Verlag Berlin Heidelberg 2003
Efficient Learning of Ordered and Unordered Tree Patterns
115
Sec1 Sec2 Comment Sec3 Sec4
Sec1
Sec2
Sec3
Introduction Preliminary
Sec4
Exp1 Exp2 Conclusion
Result1
Exp1 Exp2 Conclusion Comment
Introduction Preliminary
Result2
Exp1 Exp2
Result1 Result2
T1
SubSec3.1 Conclusion SubSec3.2
Introduction Note Preliminary
Sec1 Sec2 Comment Sec3 Sec4
Result1
T2
Result3
Result2
T3 u2 Comment Sec3
y
Sec1 Sec2
Introduction
x
Sec4
Exp1
x
Result1
z
y
Sec1 Sec2
Conclusion
Introduction Preliminary
Sec4
Exp1 Exp2
x
Result2
z
Result1 Result2
SubSec3.1 SubSec3.2
Conclusion
u1
v2 Result3
Note
u3
v1
t1
t2
t3
g1
g2
g3
Fig. 1. Ordered term trees t1 , t2 and t3 and ordered trees T1 , T2 and T3 . An uncontractible (resp. contractible) variable is represented by a single (resp. double) lined box with lines to its elements. The label inside a box is the variable label of the variable.
proposed an ordered term tree and unordered term tree, which are rooted trees with structured variables [12,13]. Many semistructured data have irregularities such as missing or erroneous data. In Object Exchange Model, the data attached to leaves are essential information and such data represented as subtrees. On the other hand, in analyzing tree structured data, sensitive knowledge (or patterns) for slight differences among such data is often meaningless. For example, extracted patterns from HTML/XML files are affected by attributes of tags which can be recognized as noises. Therefore we introduce a new kind of variable, called a contractible variable, that is an erasing variable which is adjacent to a leaf. A contractible variable can be replaced with any tree, including a singleton vertex. A usual variable, called an uncontractible variable, is replaced with a tree which consists of at least 2 vertices. A term tree with only uncontractible variables is very sensitive to noises. By introducing contractible variables, we can find robust term trees for such noises. Shinohara [11] started to study the learnabilities of extended pattern languages of strings with erasing variable. Since this pioneering work, researchers in the field of computational learning theory are interested in classes of string or tree pattern languages with erasing variables which are polynomial time learnable. Recently Uemura et al. [16] showed that classes of unions of erasing regular pattern languages can be polynomial time learnable from positive data. In this paper, we study the learnabilities of classes of tree structured patterns with restricted erasing variables, called contractible variables. A term tree t is said to be regular if all variable labels in t are mutually distinct. The term tree language of an ordered term tree t is the set of all ordered trees which are obtained from t by substituting ordered trees for variables in t.
116
Y. Suzuki et al.
The language shows the representing power of an ordered term tree t. We say that a regular ordered term tree t explains given tree structured data S if the term tree language of t contains all trees in S. A minimally generalized regular ordered term tree t explaining S is a regular ordered term tree t such that t explains S and the language of t is minimal among all term tree languages which contain all trees in S. For example, the term tree t3 in Fig. 1 is a minimally generalized regular ordered term tree explaining T1 , T2 and T3 . And t2 is also minimally generalized regular ordered term trees with no contractible variable explaining T1 , T2 and T3 . On the other hand, t1 is overgeneralized and meaningless, since t1 explains any tree of size at least 2. An ordered term tree using contractible and uncontractible variables rather than an ordered term tree only using uncontractible variables can express the structural feature of ordered trees more correctly. From this reason, we consider that in Fig. 1, t3 is a more precious term tree than t2 . In a similar way, we define the term tree language of an unordered term tree and a minimally generalized regular unordered term tree explaining given tree structured data S. Let Λ be a set of edge labels used in tree structured data. We denote by OTT cΛ (resp. UTT cΛ ) the set of all regular ordered (resp. unordered) term trees with contractible and uncontractible variables. For a set S, the number of elements in S is denoted by |S|. First we give a polynomial time algorithm for deciding whether or not a given regular ordered (resp. unordered) term tree explains an ordered (resp. unordered) tree, where |Λ| ≥ 1. Second when |Λ| ≥ 2, we give a polynomial time algorithm for finding a minimally generalized regular ordered term tree in OTT cΛ which explains all given data. Lastly when |Λ| is infinite, we give a polynomial time algorithm for finding a minimally generalized regular unordered term tree in UTT cΛ which explains all given data. These results imply that both OTT cΛ where |Λ| ≥ 2 and UTT cΛ where |Λ| is infinite are polynomial time inductively inferable from positive data. A term tree is different from other representations of tree structured patterns such as in [2,3,5] in that a term tree has structured variables which can be substituted by arbitrary trees. As related works, we proved the learnability of some classes of term tree languages with no contractible variable. In [13,14], we showed that some fundamental classes of regular ordered term tree languages are polynomial time inductively inferable from positive data. And in [7,9,12], we showed that the class of regular unordered term tree