ECAI 2008: Proceedings, 18th European Conference on Artificial Intelligence, July 21-25, 2008, Patras, Greece : Including Prestigious Applications of Intelligent ... in Artifical Intelligence and Applications)

Def. 5 Consider a FOL language L, a set of integrity constraints C, some update-generating ordering scheme  and a change operator • : L+ × L0 → L+ . A change operation K • U will be called rational with respect to  iff it satisfies the following properties for all ontologies K and updates U :

73

• Limit Cases: If K is not a valid ontology or U is an infeasible update, then: K • U = K. • General Case: If K is a valid ontology and U is a feasible update, then: – Principle of Success: K • U  U – Principle of Validity: K • U is a valid ontology – Principle of Minimal Change: For all valid ontologies K  such that K   U , it holds that Delta(K, K • U ) K Delta(K, K  )

Def. 5 dictates that applying a rational change operator between an update and an ontology should result (in the general case) in a valid ontology (Principle of Validity), which implies the update (Principle of Success). Moreover, for any other valid ontology K  that could be an alternative result, the set of (non-void) side-effects leading to K  (captured by the Delta function) is more “expensive” than the set of (non-void) side-effects leading to the result of the rational change operation. In effect, the rational change operator applies the minimum, with respect to K , set of effects and side-effects.

3

ALGORITHMS

3.1

General-purpose algorithm

We now present our general-purpose algorithm shown in Table 5. For a given language L (predicates and rules) and update-generating ordering , the function takes as inputs the ontology K, an update U , and the set ESE (initially empty) of already considered effects and side-effects. The algorithm identifies all invalidities caused by the predicates of the update, appends the update with each possible alternative side-effect separately and calls itself. Upon returning, it compares the different alternatives with the “min” function (which implements ). Upon termination it returns the effects (U ) and the minimal (per K ) set of side-effects of U upon K. The output of the algorithm, if not infeasible, is ready to be raw applied to K, as stated in Theorem 1. Table 5. Function Update: (U, K, ESE) → Delta(K, K • U ) ST EP 1: If U ∪ ESE is inconsistent, then return INFEASIBLE. ST EP 2: If (K ∪ ESE)  U , then return ∅ ST EP 3: Take an arbitrary ground fact P (x) ∈ U \ ESE such that K  {P (x)} ST EP 4: Find a rule r , such that there is some set V and tuple of constants  y for which ¬P (x) ∈ V and V ∈ Comp(r,  y ) and for all V  ∈ Comp(r, y), V = V  it holds that V   U ∪ ESE . ST EP 5: If there is no such rule, then return {P (x)} ∪ Update(U, K, ESE ∪ {P (x)}). ST EP 6: Otherwise, select (arbitrarily) one such rule, say r , and return min{Update(U ∪ V  , K, ESE)}, where the min is taken over all V  ∈ Comp(r,  y ), V  = V .

Theorem 1 The raw application of the output of U pdate(K, U, ∅) to K implements a rational change operation. The complexity of our algorithm depends on the parameters (language, rules, ordering). When considering an infinite number of constants in our language, we could have an infinite number of rule instances or a rule instance with an infinite size (when ∃ in a DED stands before a free variable). However, for our parameters, we can limit Σ to the finite set of all the names appearing in K or U , plus an extra auxiliary constant. The intuition behind this choice is that our ordering guarantees that no solution involving more than one auxiliary constant names (i.e., names not in K or U ) could ever be selected for implementation (per ). This fact and Theorem 2 below guarantee termination of the algorithm:

74

G. Konstantinidis et al. / A Formal Approach for RDF/S Ontology Evolution

Theorem 2 Consider the language and the ordering defined in section 2, an ontology K and an update U . Then, if we restrict the set of constants of our language to a finite set, before calling U pdate(K, U, ∅), the latter terminates.

3.2

Special-purpose algorithms

In practice, our general-purpose algorithm will be applied for a particular language and ordering. For such a case, it makes sense to trade generality for computational efficiency and develop special-purpose algorithms that would produce the same output as the generalpurpose one for the particular set of parameters. For this purpose, we singled out a number of useful change operations and developed a special-purpose algorithm for each one (for the particular language and ordering at hand). Since there is an infinite number of possible updates, this effort is inherently incomplete, and we will necessarily have to resort to the general-purpose algorithm for unconsidered operations. This approach may seem to reintroduce the drawbacks of ad-hoc approaches mentioned earlier, but this is not the case: the specialpurpose algorithms have been formally proven to be equivalent to the general algorithm (i.e., they are rational change operators) for the specific operation they tackle. This can be proved by running the general-purpose algorithm for all relevant states of an ontology and verifying its output against the output of the special-purpose algorithms. Thus, special-purpose algorithms are more efficient than the general-purpose algorithm, but use the same general approach as a formal foundation; moreover, any unforseen operation can anyway be dealt with by the general algorithm. As an example, Table 6 shows such an algorithm for adding a domain to a property, and Prop. 3 shows the relevant result. Prop. 3 Consider the language and the ordering defined in section 2, an ontology K and the update U = {Domain(P, D)} (for any P, D ∈ Σ). Then, the output of the algorithm in Table 5 with the above inputs is identical to the output of the algorithm in Table 6 with the same inputs.

Table 6.

Add Domain Algorithm

If PS(P) doesn’t exist does not already exist in K: Add to output {P S(P ), Range(P, rdf s : Resource)} If CS(D) doesn’t exist does not already exist in K: Add to output {CS(D)} Remove the old Domain, e.g., add to output {¬Domain(P, A)} Add the new Domain, i.e., add to output {Domain(P, D)} If P is instantiated by a property instance, say P I(a, b, P ) Verify that (the new domain) D has as its instance the resource a If not, add to output an instance relationship from a to D .

4

CONCLUSIONS

In this paper, we studied the problem of updating an ontology in the face of new information. We criticized the currently used paradigm of selecting a number of operations to support and determining the proper effects of each operation on a per-case basis, and proposed a formal framework to describe updates and their effects, as well as a general-purpose algorithm to perform those updates. Our methodology is inspired by the general belief revision principles of Validity, Success and Minimal Change [3]. The end result is an algorithm that is highly parameterizable, both in terms of the language used and in terms of the implementation of the Principle of Minimal Change.

Our methodology exhibits a “faithful” behavior with respect to the various choices involved, regardless of the particular ontology or update at hand. It lies on a formal foundation, issuing a solid and customizable method to handle any type of change on any ontology, including operations not considered at design time. In addition, it avoids resorting to the error-prone per-case reasoning of other systems, as all the alternatives regarding an update’s side-effects can be derived from the language’s rules themselves, in an exhaustive and provably correct manner. Although our general algorithm can be applied to a variety of languages, in this paper we elaborated on a specific fragment of RDF. This restriction allowed the development of special-purpose algorithms which provably exhibit behavior identical to the general-purpose one (so they are rational change operators), but also enjoy much better computational properties. Our approach was recently implemented in a large scale real-time system, as part of the ICS-FORTH Semantic Web Knowledge Middleware (SWKM), which includes a number of web services for managing RDF/S knowledge bases6 . Future work includes experimental evaluation of the algorithms’ performance and the incorporation of techniques and heuristics that would further speed up our algorithms. We also plan to evaluate the feasibility of applying the same methodology in richer languages, such as OWL Lite (notice that for highly complex languages, the development of a complete set of validity rules may be difficult).

REFERENCES [1] S. Bechhofer, I. Horrocks, C. Goble, and R. Stevens, ‘OilEd: A Reasonable Ontology Editor for the Semantic Web’, Ki 2001: Advances in AI: Joint German/Austrian Conference on AI, Vienna, Austria, September 19-21, 2001: Proceedings, (2001). [2] T. Gabel, Y. Sure, and J. Voelker, ‘KAON–ontology management infrastructure’, SEKT informal deliverable, 3(1). [3] P. G¨ardenfors, ‘Belief Revision: An Introduction’, Belief Revision, 29, 1–28, (1992). [4] P. Haase and Y. Sure, ‘D3.1.1.b state of the art on ontology evolution’, Technical report, (2004). [5] M. Klein and N.F. Noy, ‘A component-based framework for ontology evolution’, Workshop on Ontologies and Distributed Systems at IJCAI, (2003). [6] G. Konstantinidis, G. Flouris, G. Antoniou, and V. Christophides, ‘Ontology evolution: A framework and its application to rdf’, in Proceedings of the Joint ODBIS & SWDB Workshop on Semantic Web, Ontologies, Databases (SWDB-ODBIS-07), (2007). [7] S. Munoz, J. Perez, and C. Gutierrez, ‘Minimal deductive systems for rdf’, in Proceedings of the 4th European Semantic Web Conference, (2007). [8] N. Noy, R. Fergerson, and M. Musen, ‘The knowledge model of Prot´eg´e-2000: Combining interoperability and flexibility’, Lecture Notes in Artificial Intelligence (LNAI), 1937, 17–32. [9] J.Z. Pan and I. Horrocks, ‘Metamodeling Architecture of Web Ontology Languages’, Proceedings of the Semantic Web Working Symposium, 149, (2001). [10] G. Serfiotis, I. Koffina, V. Christophides, and V. Tannen, ‘Containment and minimization of rdf/s query patterns’, in Proceedings of the 4th International Semantic Web Conference (ISWC-05), (2005). [11] L. Stojanovic, A. Maedche, B. Motik, and N. Stojanovic, ‘User-driven ontology evolution management’, Proceedings of the 13th European Conference on Knowledge Engineering and Knowledge Management EKAW, (2002). [12] L. Stojanovic and B. Motik, ‘Ontology Evolution within Ontology Editors’, Proceedings of OntoWeb-SIG3 Workshop, 53–62, (2002). [13] Y. Sure, J. Angele, and S. Staab, ‘OntoEdit: Multifaceted Inferencing for Ontology Engineering’, Journal on Data Semantics, 1(1), 128–152. [14] Y. Theoharis, Y. Tzitzikas, D. Kotzinos, and V. Christophides, ‘On graph features of semantic web schemas’, IEEE Transactions on Knowledge and Data Engineering, 20(5), (2008). 6

http://athena.ics.forth.gr:9090/SWKM

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-75

75

Modular Equivalence in General Tomi Janhunen1 Abstract. The notion of modular equivalence was recently introduced in the context of a module architecture proposed for logic programs under answer set semantics [12, 6, 13]. In this paper, the module architecture is abstracted for arbitrary knowledge bases, KBfunctions for short, giving rise to a universal notion of modular equivalence. A further objective of this paper is to study modular equivalence in the contexts of SAT-functions, i.e., propositional theories with a module interface, and their logic programming counterpart, known as LP-functions [6]. As regards SAT-functions, we establish the full compositionality of classical semantics. This proves modular equivalence a proper congruence relation for SAT-functions. Moreover, we address the interoperability of SAT-functions and LPfunctions in terms of strongly faithful transformations in both directions. These considerations justify the proposed design of KBfunctions in general and pave the way for hybrid KB-functions.

module theorems in the context of LP-functions [12, 6] and they play an essential role in view of the congruence property of modular equivalence. Moreover, our goal is to study the interrelationships of SAT-functions and LP-functions in terms of translations. To this end, translations proposed in the literature [11, 10, 5] provide a good starting point but extra care is required as regards their modularity. The technical objectives discussed above reflect our interest in bringing principles of good software engineering practise such as program encapsulation, modularization, and so forth, to the realm of ASP. In this paper, we propose similar ideas for satisfiability checking and thus foster the cross-fertilization of SAT and ASP technologies. To this end, our long-term goals comprise hybrid formalisms for knowledge representation that combine best features of ASP, SAT checking, and related approaches. In addition to background theory, we intend to develop solvers and/or compilers to deal with hybrid representations. An example of such a representation is given below.

1 INTRODUCTION

Example 1 Consider the problem of determining Hamiltonian cycles (HCs) for a directed graph G = V, E with E ⊆ V × V given as input. To represent the HC problem, we propose the follown ing combination of a SAT-function Πn cyc and an LP-function Πrch :

The study of equivalence relations has become a vivid line of research in answer-set programming (ASP). These relations are tightly connected to the optimization of answer set programs as they formalize which (versions of) programs are viewed equivalent. A number of proposals such as strong/weak equivalence [9], uniform equivalence [2], and the relativized versions of strong and uniform equivalence [16] have been considered in the literature. Amongst these, strong equivalence appears to be too restrictive to cover practical needs such as hiding auxiliary atoms whereas others fail to be congruence relations with respect to program union ∪, i.e., Π1 ≡ Π2 does not imply Π1 ∪ Π ≡ Π2 ∪ Π in general, which pre-empts prospects for modular verification. To the contrary, the notion of modular equivalence [12, 6] overcomes these deficiencies. Firstly, it is a proper congruence relation for a program composition operator —enabling substitutions of equivalent modules as well as modular verification. Secondly, it provides logic programs with an adequate input/output interface using which interaction of program modules can be defined. One of the main objectives of this paper is to generalize the notion of modular equivalence for arbitrary knowledge bases in the propositional case. Conceptually, this implies the introduction of KB-functions—in analogy to (D)LP-functions [3, 6]—to provide the basic mechanism for the encapsulation of theories of interest. The general form of modular equivalence is then obtained by viewing KB-functions as modules. In addition to LP-functions, we consider SAT-functions based on sets of propositional clauses as instances of KB-functions. Although this implies a simpler syntax and semantics compared to LP-functions, the customization of modular equivalence for SAT-functions involves a number of concerns such as the compositionality of classical semantics. Such results are known as 1

Helsinki University of Technology TKK, P.O. Box 5400, FI-02015 TKK, Finland. Email: [email protected]

W W Πn cyc : { 1≤j≤n cij , 1≤j≤n cji | 1 ≤ i ≤ n} ∪ {¬cij ∨ ¬cik , ¬cji ∨ ¬cki | 1 ≤ i ≤ n, 1 ≤ j < k ≤ n} ∪ {¬cij ∨ eij | 1 ≤ i, j ≤ n}. Πn rch :

{r1 ←} ∪ {rj ← ri , cij | 1 ≤ i, j ≤ n} ∪ {f ← ¬ri , ¬f | 1 < i ≤ n}.

The aim of Πn cyc , parameterized by n = |V | > 0, is to create a cycle on G whereas Πn rch checks for the reachability of vertices. An input th vertex to atom eij of Πn cyc denotes an edge in E leading from the i th n the j . The clauses of Πcyc pick for every vertex 1 ≤ i ≤ n, exactly one outgoing edge i, j and incoming edge j, i, denoted by cij and cji , respectively. The rules of Πn rch formalize the reachability of vertices from the first and, if this is not the case, disqualify the cycle. We proceed as follows. The general module architecture based on KB-functions is introduced in Section 2. A particular instance, i.e., SAT-functions, is then fixed in Section 3. The notion of modular equivalence is formulated for KB-functions to define which of them are equivalent and which not. Section 4 is devoted for modularity properties: The full compositionality of classical semantics is established and formalized as a module theorem which implies the desired congruence property of modular equivalence. A brief comparison with LP-functions follows in Section 5. The interconnections of SAT-functions and LP-functions plus their combinations are then worked out in Section 6. The prospects for verifying modular equivalence are addressed in Section 7. Finally, the paper is concluded with a brief discussion of the results and related work in Section 8.

76

T. Janhunen / Modular Equivalence in General

2 GENERAL MODULE ARCHITECTURE For a start, we introduce a general module architecture for logical theories to be used for knowledge representation in the propositional case. They are formalized as sets of expressions built of propositional atoms (or just atoms for short). We term such sets of expressions, which typically formalize some kind of knowledge, as knowledge bases (KBs). Since we are interested in the construction of KBs in a modular fashion, we distinguish an input/output interface for each theory. To this end, we generalize notions introduced in the context of logic programming, see e.g. [3, 6]. Given a KB, i.e., a set E of expressions pertaining to some syntax, we write At(E) for the signature of E, i.e., the set of propositional atoms having occurrences in the expressions of E. We adopt the notion of a module based on E from [6] where the case of disjunctive logic programs is of interest. Definition 1 A KB-function Π is a quadruple E, I, O, H where I, O, and H are mutually disjoint sets of input atoms, output atoms, and hidden atoms, respectively, and E is a set of expressions such that At(E) ⊆ I ∪ O ∪ H. Given a KB-function Π = E, I, O, H, the (overall) signature At(Π) of Π is defined as I ∪ O ∪ H. Atoms in I ∪ O are visible and accessible to other KB-functions attached with Π; either to produce input for Π or to exploit the output of Π. We are prepared to allow arbitrary input/output relationships between modules as long as the semantics of modules supports that. The hidden atoms in H formalize some auxiliary concepts of Π which may not make sense in other contexts but may enable a more succinct representation as follows. Example 2 Consider a KB-function Πn = En , In , On , Hn  where n ≥ 0 and En = {¬b0 } ∪ E1 ∪ . . . ∪ En where each set Ei consists of four clauses bi ∨ ai ∨ ¬bi−1 , bi ∨ ¬ai ∨ bi−1 , ¬bi ∨ ai ∨ bi−1 , and ¬bi ∨ ¬ai ∨ ¬bi−1 . The signature of Πn comprises of In = {a1 , . . . , an }, On = {bn }, and Hn = {b0 , . . . , bn−1 }. The purpose of this KB-function is to check whether the number of true atoms among a1 , . . . , an is odd. Intuitively, the meaning of bi is the odd parity property for a1 , . . . , ai and it is captured recursively by bi ↔ (¬ai ∧ bi−1 ) ∨ (ai ∧ ¬bi−1 ) for which Ei is a clausal representation for each 0 < i ≤ n. The only output atom bn denotes the overall result for a1 , . . . , an . Any attempts to remove hidden atoms b0 , . . . , bn−1 from the definition of Πn cause a substantial increase in the length of En : There are 2n−1 truth assignments to a1 , . . . , an for which bn is supposed to be true and false, respectively, and there is no simple logical pattern to make a distinction between the two cases using clauses. In contrast, the length of Πn is linear in n which nicely demonstrates the favorable effects of auxiliary atoms. For notational compatibility with [6], we denote the visible and hidden parts of At(Π) by Atv (Π) = I ∪ O and Ath (Π) = H = At(Π) \ Atv (Π), respectively. Additionally, Ati (Π) and Ato (Π) denote the sets I and O of input, and respectively, output atoms of Π. To access the set of expressions in Π, we define E = Expr(Π). For any set S ⊆ At(Π), the projections of S on Ati (Π), Ato (Π), Atv (Π), and Ath (Π), are denoted by Si , So , Sv , and Sh , respectively. Having fixed the syntax of KB-functions, few words on their semantics follow. From the mathematical point of view, a KB-function Π = E, I, O, H provides a mapping from truth assignments (represented as subsets of I) to sets of truth assignments (represented as subsets of O ∪ H). However, the exact mapping depends on the semantics assigned to the set of expressions E. The case of SATfunctions, which involves a set of clauses C and classical truth assignments, is deferred until Section 3 whereas a brief account of

LP-functions under stable model semantics [4] follows in Section 6. Meanwhile we formalize conditions for building more complex KBfunctions out of simpler ones. We say that KB-functions Π1 and Π2 respect the I/O interfaces of each other iff At(Π1 ) ∩ Ath (Π2 ) = ∅, At(Π2 ) ∩ Ath (Π1 ) = ∅, and Ato (Π1 ) ∩ Ato (Π2 ) = ∅. Definition 2 (Composition) The composition Π1 ⊕ Π2 of two KBfunctions Π1 = E1 , I1 , O1 , H1  and Π2 = E2 , I2 , O2 , H2  which respect the I/O interface of each other is a KB-function E1 ∪ E2 , (I1 \ O2 ) ∪ (I2 \ O1 ), O1 ∪ O2 , H1 ∪ H2 . Although Π1 and Π2 must not share hidden atoms nor output atoms, they may share input atoms, i.e., I1 ∩ I2 = ∅ is allowed. An input atom of Π1 becomes an output atom in Π1 ⊕ Π2 if it appears as an output atom in Π2 , i.e., Π2 provides the input for Π1 in this setting. The input atoms of Π2 are treated in a symmetric fashion. The hidden atoms of Π1 and Π2 retain their statuses in Π1 ⊕ Π2 .

3 SAT-FUNCTIONS In this section, we define the class of SAT-functions and their semantics. A (propositional) clause is an expression of the form a1 ∨ · · · ∨ an ∨ ¬b1 ∨ · · · ∨ ¬bm ,

(1)

where n, m ≥ 0, and a1 , . . . , an and b1 , . . . , bm are propositional atoms. The logical intuition behind (1) is that at least one of a1 , . . . , an is true or at least one of b1 , . . . , bm is false. Since the order of atoms does not matter, we write A ∨ ¬B as a shorthand for (1) using the idea that A = {a1 , . . . , an } and ¬B = {¬b1 , . . . , ¬bm }. When both A and ¬B are empty, we have an empty disjunction, written ⊥. Any propositional theory can be transformed into clausal form in linear time using Tseitin’s technique with auxiliary atoms. A SAT-function Π = C, I, O, H is a KB-function where C is a finite set of clauses. For simplicity, the class of SAT-functions SF spans over a fixed (at most denumerable) signature At(SF) so that At(Π) ⊆ At(SF) holds for each SAT-function Π ∈ SF. As regards semantics, an interpretation for a SAT-function Π is defined as an arbitrary subset M of At(Π). Then, an atom a ∈ At(Π) is true in M , denoted M |= a, iff a ∈ M . For a negative literal ¬a, we have M |= ¬a iff M |= a. A set of literals L is satisfied by M , denoted M |= L, iff M |= l for every W l ∈ L. To make a disjunctive interpretation, we define M |= L, providing M |= l for some l ∈ L. The classical semantics of SAT-functions is defined next. Definition 3 An interpretation M ⊆ At(Π) is a (classical) model of a SAT-function Π = C, I, O, H, denoted W M |= Π, iff W M |= C, i.e., for every clause A ∨ ¬B ∈ C, M |= A or M |= ¬B. The set of all classical models of Π is denoted by CM(Π). Let us now clarify the role of input atoms in SAT-functions. Given a SATfunction Π and any interpretation M ⊆ At(Π), the projection Mi can be viewed as the actual input for Π. We use partial evaluation to pre-interpret input atoms appearing in Π with respect to Mi . Definition 4 For a SAT-function Π = C, I, O, H and an actual input Mi ⊆ I for Π, the instantiation of Π with respect to Mi , denoted by Π/Mi , is the SAT-function C  , ∅, O, H where C  contains a reduced clause (A \ I) ∨ ¬(B \ I) for each clause A ∨ ¬B ∈ Π such that Mi |= Ai ∨ ¬Bi , i.e., Mi ∩ Ai = ∅ and Bi ⊆ Mi . There are no (occurrences of) input atoms in the reduced SATfunction Π/Mi and the visibility of atoms is not affected by instantiation. Partial evaluation is fully compatible with classical semantics.

T. Janhunen / Modular Equivalence in General

Proposition 1 Let Π be a SAT-function and M ⊆ At(Π) an interpretation that defines an actual input Mi ⊆ Ati (Π) for Π. For all N ⊆ At(Π) such that Ni = Mi , N |= Π ⇐⇒ No ∪ Nh |= Π/Mi . Therefore, we may characterize the overall semantics of Π by CM(Π) = {M ⊆ At(Π) | Mo ∪ Mh |= Π/Mi } which accentuates the dependence of CM(Π) on all possible input interpretations. Example 3 Recall the SAT-function Πn from Example 2 in the case n = 3. For an input interpretation Mi = {a2 }, we obtain Π3 /Mi = {¬b0 , b1 ∨ ¬b0 , ¬b1 ∨ b0 , b2 ∨ b1 , ¬b2 ∨ ¬b1 , b3 ∨ ¬b2 , ¬b3 ∨ b2 }. There is a unique classical model N = {a2 , b2 , b3 } of Π3 /Mi satisfying Ni = Mi . It captures the correct solution to the parity problem formalized by Π3 : the output atom b3 is true indicating that the number of true atoms among a1 , . . . , a3 is odd. Thus Π maps the input interpretation Mi = {a2 } to the set of interpretations {N ⊆ At(Π) | N |= Π3 /Mi and Ni = Mi } = {{a2 , b2 , b3 }}. In general, the set CM(Π/Mi ) need not be a singleton and the number of output interpretations may vary. Imagine, for instance, a SAT-function Πn gc that captures the famous n-coloring problem of graphs. Then there are input interpretations Mi corresponding to graphs without n-colorings, i.e., CM(Πn gc /Mi ) = ∅, or with several n-colorings for which |CM(Πn gc /Mi )| > 1 holds accordingly. Hidden atoms play no special role in Definition 3 which determines the semantics of a SAT-function Π. To the contrary, they become highly important when SAT-functions are compared with each other. To this end, we adopt the notion of modular equivalence originally proposed in the context of ASP [12, 6]. Roughly speaking, the idea is to neglect hidden atoms when comparing interpretations assigned to KB-functions by projecting them with respect to visible atoms. Additionally, a strict correspondence of models is required. Definition 5 Two KB-functions Π1 and Π2 are modularly equivalent, denoted Π1 ≡m Π2 , iff Ati (Π1 ) = Ati (Π2 ) and Ato (Π1 ) = Ato (Π2 ), and there is a bijection f : SEM1 (Π1 ) → SEM2 (Π2 ) so that for all M ∈ SEM1 (Π1 ), M ∩ Atv (Π1 ) = f (M ) ∩ Atv (Π2 ). The definition of ≡m is applicable to all KB-functions given the appropriate semantic operators SEM1 and SEM2 which map Π1 and Π2 to subsets of 2At(Π1 ) and 2At(Π2 ) , respectively. Within SF, one would of course set SEM1 (Π1 ) = CM(Π1 ) and SEM2 (Π2 ) = CM(Π2 ) in Definition 5. The design criteria of ≡m go back to ASP where a bijective relationship between answer sets and the solutions of a problem is highly desirable. In this way, the numbers of “solutions” associated with modularly equivalent programs are the same. This is analogous to parsimonious reductions employed in complexity theory and, in particular, in the context of counting problems [15].

4 MODULARITY OF SAT-FUNCTIONS Our next objective is to address the modularity properties of classical semantics as set out by Definition 4 and Proposition 1. The first result, i.e., module theorem to be established below, links classical models assigned to a composition Π1 ⊕ Π2 with classical models assigned to the component SAT-functions Π1 and Π2 separately. Given two KB-functions Π1 and Π2 in general, we say that respective interpretations M1 ⊆ At(Π1 ) and M2 ⊆ At(Π2 ) are mutually compatible, or just compatible for short, iff M1 ∩ Atv (Π2 ) = M2 ∩ Atv (Π1 ), i.e., M1 and M2 agree about the truth values of their joint visible atoms. If the join Π1 ⊕Π2 is defined, there may be shared input atoms in Ati (Π1 ) ∩ Ati (Π2 ), or atoms in Ato (Π1 ) ∩ Ati (Π2 )

77

and Ati (Π1 ) ∩ Ato (Π2 ) that are output atoms in one SAT-function and input atoms in the other. According to Definition 2, latter atoms end up in Ato (Π1 ⊕ Π2 ). To enable a concise formulation, we adopt a kind of a natural join (in database terminology) for arbitrary sets of interpretations S1 ⊆ 2At(Π1 ) and S2 ⊆ 2At(Π2 ) . The set S1 1 S2 contains I1 ∪ I2 for each compatible pair I1 ∈ S1 and I2 ∈ S2 . Theorem 1 (Module Theorem) Let Π1 and Π2 be SAT-functions such that their composition Π1 ⊕ Π2 is defined. Then, the set of classical models CM(Π1 ⊕ Π2 ) = CM(Π1 ) 1 CM(Π2 ). Theorem 1 indicates that classical semantics fully supports modularization. The result is an analog of [6, Proposition 3] but formulated here for sets of clauses rather than disjunctive rules. As regards a generalization for several SAT-functions, a sequence M1 , . . . , Mn of interpretations for Π1 , . . . , Πn , respectively, is considered to be compatible, iff for all 1 ≤ i, j ≤ n, the interpretations Mi and Mj are pairwise compatible. Likewise, a finite composition Π1 ⊕. . .⊕Πn is defined, iffL the respective compositions Πi ⊕Πj are defined for i = j. n Thus CM( n i=1 Πi ) =1i=1 CM(Πi ) holds if Π1 ⊕. . .⊕Πn is defined. Generally speaking, Theorem 1 and its generalization are not very restrictive when splitting a set of clauses C into modules. Two aspects, however, require attention. First, each hidden atom must remain local to some module and hence it cannot occur in any other modules. Second, each clause should be local to a particular module. Definition 6 The join, Π1  Π2 , of two SAT-functions Π1 and Π2 is Π1 ⊕ Π2 , iff Π1 ⊕ Π2 is defined and Expr(Π1 ) ∩ Expr(Π2 ) = ∅. In view of algebraic properties, we have Π1  Π2 ∈ SF (closure), Π1  ∅ = ∅  Π1 = Π1 for the empty module ∅ = ∅, ∅, ∅, ∅ (identity), Π1  Π2 = Π2  Π1 (commutativity), and Π1  (Π2  Π3 ) = (Π1  Π2 )  Π3 (associativity) given SAT-functions Π1 , Π2 , and Π3 that pairwise respect the I/O interface of each other and share no clauses. In addition, we can establish that ≡m is a congruence relation for  which enables ≡m -preserving substitutions under . Theorem 2 Let Π1 , Π2 , and Π be SAT-functions. If Π1 ≡m Π2 and both Π1  Π and Π2  Π are defined, then Π1  Π ≡m Π2  Π.

5 BRIEF ACCOUNT OF LP-FUNCTIONS Next we highlight the main differences and similarities with respect to LP-functions corresponding to the case of normal programs presented in [12]. The general module system formalized as KBfunctions is almost applicable as such. As regards LP-functions, expressions are rules of the form h ← a1 , . . . , an , ¬b1 , . . . , ¬bm , abbreviated as h ← A, ¬B using sets of atoms as in Section 3. Intuitively, one derives the head h using h ← A, ¬B if one can derive all atoms in A but none in B. Given a set of rules R in an LP-function Π = R, I, O, H, the set Hd(R) = {h | h ← A, ¬B ∈ R} of head atoms must be contained in O ∪ H. This ensures that an LPfunction cannot redefine its input atoms. The composition operator ⊕ from Definition 2 is directly applicable to LP-functions but the definition of the join operator  becomes much more involved. In [12, 6], the strongly connected components (SCCs) of the positive dependency graph DG+ (Π) = Ato (Π) ∪ Ath (Π), ≤1  are exploited. Here a ≤1 h holds iff Π has a rule h ← A, ¬B with a ∈ Ao ∪ Ah . The positive dependency relation ≤ is (≤1 )∗ . The join Π1  Π2 of two LP-functions Π1 and Π2 is defined iff Π1 ⊕ Π2 is defined and, for each SCC S of DG+ (Π1 ⊕ Π2 ), S ∩ Ato (Π1 ) = ∅ or S ∩ Ato (Π2 ) = ∅ [12], i.e., Π1 and Π2 are ≤-independent.

78

T. Janhunen / Modular Equivalence in General

Given an LP-function Π = R, I, O, H and an interpretation M ⊆ At(Π) which includes an input interpretation Mi = M ∩ I, the reduct RM,I generalizes that from [4] as it contains a reduced rule h ← (A \ I) whenever there is a rule h ← A, ¬B such that M |= Ai ∪ ¬B.2 An interpretation M ⊆ At(Π) is a stable model of Π iff M \ I is the least (classical) model of RM,I . The operator SM(Π) = {M ⊆ At(Π) | Mo ∪ Mh = LM(RM,Ati (Π) )} covers the class N of LP-functions based on normal programs. For N , modular equivalence ≡m is obtained from Definition 5 by substituting SM(·) for SEM1 (·) and SEM2 (·). An analog of Theorem 1 can also be established for LP-functions under  but not under ⊕ [12]. This is crucial for ≡m being a congruence relation for .

6 INTERCONNECTIONS The goal of this section is to address the relationship of SATfunctions with LP-functions in terms of translations. Due to known complexity results, we shall restrict ourselves to the case of normal (non-disjunctive) programs—to enable translations feasible in polynomial time. In addition to translations, we consider direct combinations of SAT-functions and LP-functions in Section 6.1. Any set C of propositional clauses can be translated into a normal logic program using, e.g., the translation provided by Niemel¨a [11]. Extra care, however, is required to ensure the modularity of the transformation when embedding SAT-functions in LP-functions. To this end, we have to pay special attention to input atoms and localize the choice of truth values. Thus we introduce a clause-specific new atom f(A,B) for each clause A∨¬B and similarly a(A,B) and b(A,B) for every non-input atom a ∈ A and b ∈ B, respectively. An entire clause A ∨ ¬B ∈ C is translated into a set of rules TrLP (A, B) = {f(A,B) ← B, ¬A, ¬f(A,B) } ∪ {a ← ¬a(A,B) ; a(A,B) ← ¬a | a ∈ Ao ∪ Ah } ∪ {b ← ¬b(A,B) ; b(A,B) ← ¬b | b ∈ Bo ∪ Bh }.

(2)

The choice of a truth value for a non-input atom a ∈ At(C) may concern several pairs of rules a ← ¬a(A,B) and a(A,B) ← ¬a in translations TrLP (A, B) for different clauses A∨¬B. These choices are synchronized by a which will therefore have a unique truth value. Definition 7 Given a SAT-function Π = C, I, O, H, the translation TrLP (Π) of Π into an LP-function is R, I, O, H   where R = S {TrLP (A, B) | A ∨ ¬B ∈ C} ∪ {a ← ¬a; a ← ¬a | a ∈ (O ∪ H) \ At(C)} and H  consists of H, {a | a ∈ (O ∪ H) \ At(C)}, and for each A ∨ ¬B ∈ C, Ao(A,B) ∪ Ah(A,B) ∪ Bo(A,B) ∪ Bh(A,B) ∪ {f(A,B) }. An arbitrary truth value is chosen for any atom in (O ∪ H) \ At(C) having no occurrences in C. Otherwise, such atoms would become false by default. Generally speaking, a translation function Tr(·) is (i) strongly faithful iff rev(Π) ≡m Tr(rev(Π)) where rev(Π) = E, I, O ∪ H, ∅ is Π = E, I, O, H with H revealed, (ii) modular iff Tr(Π1 )  Tr(Π2 ) = Tr(Π1  Π2 ), and (iii) -preserving iff Tr(Π1 )  Tr(Π2 ) is defined whenever Π1  Π2 is [13]. Also, note that strong faithfulness implies faithfulness, i.e., Π ≡m Tr(Π). The semantics of SAT-functions is accurately conveyed by TrLP . Theorem 3 The translation function TrLP from SAT-functions to LP-functions is strongly faithful, modular, and -preserving. 2

Thus all input literals and negative literals get evaluated from rule bodies.

Obtaining a similar translation in the other direction, i.e., from LPfunctions to SAT-functions, is much more elaborate. Formal counterexamples (cf. [11]) pre-empt a fully modular translation that could be applied rule by rule in analogy to TrLP defined above. In spite of this, non-modular alternatives have been proposed [10, 5]. For instance, Janhunen [5] removes positive body conditions from rules using a systematic translation TrAT which is not reviewed herein due to space limitations. When combined with Clark’s completion procedure TrCC , a strongly faithful translation from LP-functions to SATfunctions is obtained. Albeit non-modularity in general, both translations behave modularly given an appropriate level of granularity. To make TrAT modular, it must be applied within each LP-function Π = R, I, O, H to sets of rules RS = {h ← A, ¬B ∈ R | h ∈ S} associated with the SCCs S of DG+ (Π). For TrCC , it is sufficient to distinguish Ra = {h ← A, ¬B | h = a} for each a ∈ Ato (Π) ∪ Ath (Π). The time complexity of TrAT is of the order of R × log2 (|O ∪ H|) [5] for Π = R, I, O, H whereas the procedure TrCC can be kept linear using new (hidden) atoms. Theorem 4 The translation function TrCC ◦TrAT from LP-functions to SAT-functions is strongly faithful, modular, and -preserving.

6.1 Composing hybrid functions In light of translations TrLP and TrCC ◦ TrAT , SAT-functions and LP-functions appear to be equally expressive although LP-functions tend to be more concise (cf. Examples 1 and 4). Furthermore, modular equivalence ≡m can be used for intra-class as well as inter-class comparisons of SAT/LP-functions and ≡m partitions the respective classes into equivalence classes that comfortably respect joins of SAT/LP-functions, i.e., ≡m is a congruence for . Given such a harmonic view over SF and N , we will now consider the possibility of building hybrids of SAT-functions and LP-functions. A cross-section of the theory presented in Sections 2–5 follows. The composition Π1 ⊕ Π2 of a SAT-function Π1 = C, I1 , O1 , H1  with an LP-function Π2 = R, I2 , O2 , H2  can be formed by Definition 2. As regards the join Π1  Π2 , no special restrictions arise from Definition 6: clauses and rules have different syntax and remain separate in C ∪ R. This view is supported if we translate Π1 into TrLP (Π1 ) and check the implications of the conditions on which TrLP (Π1 )Π2 is defined. Whenever Π1 Π2 is defined, it is natural to set SEM(Π1  Π2 ) = CM(Π1 ) 1 SM(Π2 ) in order to combine classical and stable models associated with the respective functions. This paves the way for lifting the module theorem to the level of hybrid SAT/LP-functions and showing ≡m a congruence for . Example 4 Recall the Hamiltonian cycle problem from Example 1. Let Cn be the set of clauses that selects cycles for the graph G given as input and Rn the set of rules that captures the universal reachability of vertex 1. The respective SAT- and LP-functions are n Πn cyc = Cn , In , On , ∅ and Πrch = Rn , On , ∅, Hn ∪ {f } where n Ati (Πcyc ) = In = {eij | 1 ≤ i, j ≤ n} is used to describe G. The n signature Ato (Πn cyc ) = On = {cij | 1 ≤ i, j ≤ n} = Ati (Πrch ) expresses which edges are on a cycle. Auxiliary atoms of Hn = {ri | 1 ≤ i ≤ n} which are hidden in Πn rch denote reachable vertices. n n The semantics of the hybrid function Πn cyc  Πrch is CM(Πcyc ) 1 n SM(Πrch ) which determines all Hamiltonian cycles for all directed graphs based on V = {1, . . . , n}. Theorems 3 and 4 and the congruence properties of ≡m imply that modularly equivalent formaln izations are obtained as partial translations TrLP (Πn cyc )  Πrch and n n Πcyc TrCC (TrAT (Πrch )) and the respective stable/classical semann tics for the joins. But the hybrid Πn cyc  Πrch is the most concise.

T. Janhunen / Modular Equivalence in General

7 VERIFYING MODULAR EQUIVALENCE In what follows, we analyze the task of verifying the modular equivalence of two SAT-functions, say Π1 and Π2 , given for inspection. Recalling Definition 5, it is straightforward to check that the input and output signatures of Π1 and Π2 coincide but the bijective relationship of models renders the question difficult. A brute-force approach would check all input interpretations Mi ⊆ I = Ati (Π1 ) = Ati (Π2 ) to ensure that CM(Π1 /Mi ) and CM(Π2 /Mi ) have equally many models which coincide up to Atv (Π1 ) = Atv (Π2 ). Thus, the verification of ≡m involves a counting problem which can be highly complex in general, i.e., hard for the complexity class #P. The computational cost of verifying ≡m is reduced if the use of hidden atoms is forbidden. Then ≡m reduces to weak equivalence ≡ for SAT-functions defined as follows: Π1 ≡ Π2 iff Ati (Π1 ) = Ati (Π2 ), Ato (Π1 ) = Ato (Π2 ), and CM(Π1 ) = CM(Π2 ). This notion is very close to classical equivalence but the compatibility of the input/output interfaces is additionally required. Deciding whether Π1 ≡ Π2 holds is coNP-complete for SAT-functions by the respective complexity results for LP-functions [7] and the polynomial time reductions involved in Theorems 3 and 4. 3 This fits perfectly with the idea of testing Π1 ≡ Π2 using a SAT solver: SAT-functions Π1 and Π2 are translated into a set of clauses EQT(Π1 , Π2 ) that is unsatisfiable iff CM(Π1 ) ⊆ CM(Π2 ). A preliminary implementation, namely SATEQ4 , produces such a translation but does not yet support input atoms due to lack of symbols in the DIMACS format, i.e., the de facto standard for representing propositional formulas in CNF. As regards benchmarking, we use two orthogonal formulations n of the classical n-queens problem [7]: Πn qx and Πqy correspond to column-wise, and respectively row-wise, placement of n queens on an n × n chess board. It is rather easy to find models for these instances: MINISAT version 1.14 solves an instance with n = 128 queens in roughly 5 seconds on a 2GHz AMD Athlon 64/3200+ CPU n with 2GB memory. In contrast, verifying Πn qx ≡ Πqy , i.e., showing n n EQT(Πqx , Πqy ) unsatisfiable, gets increasingly difficult as n grows. For n = 5 . . . 13, the running times of MINISAT for the respective problem instances are 0.010, 0.014, 0.025, 0.070, 0.403, 2.26, 16.9, 173, and 2430 seconds—all averaged over 10 runs. The numbers of atoms and clauses in these instances vary in the respective ranges 304 . . . 5828 (≈ 2.5 × n3 ) and 844 . . . 17112 (≈ 7.5 × n3 ). For comparison, average running times become 9.0, 7.7, 8.0, 6.5, 4.3, and 2.0 times higher when LPEQ [7] and SMODELS are used to solve the respective problem instances with 8 . . . 13 queens. As shown in [7], the problem of verifying modular equivalence Π1 ≡m Π2 is in coNP for normal programs having enough visible atoms, i.e, the EVA property. For an LP-function Π = R, I, O, H, the point is that the hidden part Rh /Mv = {h ← Ah , ¬Bh | h ← A, ¬B ∈ R and Mv |= Av ∪ ¬Bv } must have a unique stable model given any interpretation Mv for the visible atoms in I ∪ O. In the presence of the EVA property, there is no need to count models when verifying ≡m . This suggests a future extension to the translation EQT(·, ·) described above, but it is not yet clear how to generalize the EVA property for SAT-functions like Πn in Example 2.

8 CONCLUSION AND DISCUSSION In this paper, we present a general module architecture, involving the notion of a KB-function, for knowledge bases. In case of 3 4

Also, recall that checking the validity of φ ↔ ψ for two propositional formulas φ and ψ is a coNP-complete decision problem. See http://www.tcs.hut.fi/Software/sateq/ for details.

79

propositional theories, represented as sets of clauses, the architecture based on SAT-functions is fully compatible with classical semantics as made precise by Theorem 1. Furthermore, the compositionality of classical semantics guarantees that ≡m is a proper congruence for SAT-functions. The tight interconnections with LP-functions addressed in Section 6 justify the presented architecture and pave the way for hybrid functions as illustrated in Examples 1 and 4. Few words on related work follow. In contrast with ≡m , the notions of equivalence formulated in [8] are entailment-based, although the idea of hiding auxiliary atoms is present in the sense of forgetting. Moreover, forgetting auxiliary atoms from logic programs does not preserve (stable) models in general which, in turn, is central in our approach. The ideas underlying satisfiability modulo theories [14] suggest yet another mode of reasoning in which one is interested in the answer to the satisfiability question but not in values of individual variables. To support this kind of reasoning, we would have to revise the way in which models are combined by the operator 1 under joins. In the future, we intend to study automated decomposition of propositional theories using ≡m (cf. [1]). To fully exploit our implementation of EQT(·, ·) in the modular verification of ≡m , we have to somehow represent module interfaces in the DIMACS format.

9 ACKNOWLEDGEMENTS This research has been supported by the Academy of Finland (grant #122399). The author thanks reviewers for their helpful comments.

REFERENCES [1] E. Amir and S. McIlraith, ‘Improving the efficiency of reasoning through structure-based reformulation’, in Abstraction, Reformulation, and Approximation, pp. 247–259. Springer, (2000). LNCS 1864. [2] T. Eiter and M. Fink, ‘Uniform equivalence of logic programs under the stable model semantics’, in Proceedings of ICLP’03, pp. 224–238. Springer, (2003). LNCS 2916. [3] M. Gelfond, ‘Representing knowledge in a-prolog’, in Logic Programming and Beyond, Essays in Honour of Robert A. Kowalski, Part II, LNCS 2408, pp. 413–451. Springer, (2002). [4] M. Gelfond and V. Lifschitz, ‘The stable model semantics for logic programming’, in Proceedings of ICLP’88, pp. 1070–1080, (1988). [5] T. Janhunen, ‘Representing normal programs with clauses’, in Proceedings of ECAI’04, pp. 358–362, Valencia, Spain, (2004). IOS Press. [6] T. Janhunen, E. Oikarinen, H. Tompits, and S. Woltran, ‘Modularity aspects of disjunctive stable models’, in Proceedings of LPNMR’07, pp. 697–744, Tempe, AZ, U.S.A., (2007). Springer. LNAI 4483. [7] T. Janhunen and T. Oikarinen, ‘Automated Verification of Weak Equivalence within the Smodels System’, TPLP, 7(6), 697–744, (2007). [8] J. Lang, P. Liberatore, and P. Marquis, ‘Propositional independence: Formula-variable independence and forgetting’, Journal of Artificial Intelligence Research, 18, 391–443, (2003). [9] V. Lifschitz, D. Pearce, and A. Valverde, ‘Strongly equivalent logic programs’, ACM TOCL, 2(4), 526–541, (2001). [10] F. Lin and J. Zhao, ‘On tight logic programs and yet another translation from normal logic programs to propositional logic’, in Proceedings of IJCAI’03, pp. 853–858, Acapulco, Mexico, (2003). [11] I. Niemel¨a, ‘Logic programs with stable model semantics as a constraint programming paradigm’, AMAI, 25(3–4), 241–273, (1999). [12] E. Oikarinen and T. Janhunen, ‘Modular equivalence for normal logic programs’, in Proceedings of ECAI’06, pp. 412–416. IOS Press, (2006). [13] E. Oikarinen and T. Janhunen. Achieving compositionality of the stable model semantics for SMODELS programs, February 2008. Submitted. [14] S. Ranise and C. Tinelli. SMT-LIB – The satisfiability modulo theories library. http://goedel.cs.uiowa.edu/smtlib/, 2008. [15] L. Valiant, ‘The complexity of computing the permanent’, Theoretical Computer Science, 8, 189–201, (1979). [16] S. Woltran, ‘Characterizations for relativized notions of equivalence in answer set programming’, in Proceedings of JELIA’04, pp. 161–173, Lisbon, Portugal, (2004). Springer. LNAI 3229.

80

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-80

Description Logic Rules Markus Krötzsch and Sebastian Rudolph and Pascal Hitzler1 Abstract. We introduce description logic (DL) rules as a new rulebased formalism for knowledge representation in DLs. As a fragment of the Semantic Web Rule Language SWRL, DL rules allow for a tight integration with DL knowledge bases. In contrast to SWRL, however, the combination of DL rules with expressive description logics remains decidable, and we show that the DL SROIQ – the basis for the ongoing standardisation of OWL 2 – can completely internalise DL rules. On the other hand, DL rules capture many expressive features of SROIQ that are not available in simpler DLs yet. While reasoning in SROIQ is highly intractable, it turns out that DL rules can be introduced to various lightweight DLs without increasing their worst-case complexity. In particular, DL rules enable us to significantly extend the tractable DLs EL++ and DLP.

1 INTRODUCTION The development of description logics (DLs) has been driven by the desire to push the expressivity bounds of these knowledge representation formalisms while still maintaining decidability and implementability. This has lead to very expressive DLs such as SHOIN, the logic underlying the Web Ontology Language OWL DL, SHOIQ, and more recently SROIQ [6] which is the basis for the ongoing standardisation of OWL 22 as the next version of the Web Ontology Language. On the other hand, more lightweight DLs for which most common reasoning problems can be implemented in (sub)polynomial time have also been sought, leading, e.g., to the tractable DL EL++ [1]. Another popular paradigm of knowledge representation are rulebased formalisms – ranging from logic programming to deductive databases. Similar to DLs, the expressivity and complexity of rule languages has been studied extensively [3], and many decidable and tractable formalisms are known. Yet, reconciling DLs and rule languages is not easy, and many works have investigated this problem. In this paper, we introduce DL rules as an expressive new rule language for combining DLs with first-order rules in a rather natural way that admits tight integration with existing DL systems. Since DLs can be considered as fragments of function-free first-order logic with equality, an obvious approach is to combine them with firstorder Horn-logic rules. This is the basis of the Semantic Web Rule Language SWRL [7], proposed as a rule extension to OWL. However, reasoning becomes undecidable for the combination of OWL and SWRL, and thus more restricted rule languages have been investigated. A prominent example are DL-safe rules [12], which restrict the applicability of rules to a finite set of named individuals to retain decidability. Similar safety conditions have already been proposed for CARIN [11] in the context of the DL ALCNR, where also 1 2

Universität Karlsruhe (TH), Germany, [mak|sru|phi]@aifb.uni-karlsruhe.de OWL 2 is the forthcoming W3C recommendation updating OWL, based on the OWL 1.1 member submission, cf. http://www.w3.org/2007/OWL.

acyclicity of rules and Tboxes was studied as an alternative for retaining decidability. Another basic approach is to identify the Horn-logic rules directly expressible in OWL DL (i.e. SHOIN), and this fragment has been called Description Logic Programs DLP [5]. DL rules in turn can be characterised as a decidable fragment of SWRL, which corresponds to a large class of SWRL rules indirectly expressible in SROIQ. They are based on the observation that DLs can express only tree-like interdependencies of variables. For example, the concept expression ∃ worksAt.University  ∃ supervises .PhDStudent that describes all people working at a university and supervising some PhD student corresponds to the following first-order formula: ∃y.∃z.worksAt(x, y)∧ University(y)∧ supervises(x, z)∧ PhDStudent(z) Here variables form the nodes of a tree with root x, where edges are given by binary predicates. Intuitively, DL rules are exactly those SWRL rules, where premises (rule bodies) consist of one or more of such tree-shaped structures. One could, for example, formulate the following rule: worksAt(x, y) ∧ University(y) ∧ supervises(x, z) ∧ PhDStudent(z) → profOf(x, z)

Since SWRL allows the use of DL concept expressions in rules, we obtain SROIQ rules, EL++ rules, or DLP rules as extensions of the respective DLs. For the case of SROIQ, DL rules have independently been proposed in [4], where a tool for editing such rules was presented. As shown below, DL rules are indeed “syntactic sugar” in this case, even though rule-based presentations are often significantly simpler due to the fact that many rules require the introduction of auxiliary vocabulary for being encoded in SROIQ. On the other hand, we also consider the light-weight DLs EL++ and DLP for which DL rules truly extend expressivity, and we show that the polynomial complexity of these DLs is preserved by this extension. After giving some notation in Section 2, we introduce DL rules in Section 3. Section 4 shows how DL rules can be internalised in SROIQ, while Section 5 employs a novel reasoning algorithm to process EL++ rules. Section 6 introduces DLP 2 and shows the tractability of this DL-based rule language. Most proofs are omitted and can be found in [9].

2 PRELIMINARIES In this section, we briefly introduce our notation based on the DL SROIQ [6]. More detailed definitions and introductory remarks can be found in [9]. As usual, the DLs considered in this paper are based on three disjoint sets of individual names NI , concept names NC , and role names NR containing the universal role U ∈ NR . Definition 1 A SROIQ Rbox for NR is based on a set R of roles defined as R  NR ∪ {R− | R ∈ NR }, where we set Inv(R)  R−

M. Krötzsch et al. / Description Logic Rules

and Inv(R− )  R to simplify notation. In the sequel, we will use the symbols R, S , possibly with subscripts, to denote roles. A generalised role inclusion axiom (RIA) is a statement of the form S 1 ◦ . . . ◦ S n R, and a set of such RIAs is a SROIQ Rbox. An Rbox is regular if there is a strict partial order ≺ on R such that • S ≺ R iff Inv(S ) ≺ R, and • every RIA is of one of the forms: R ◦ R R, R− R, S 1 ◦ . . . ◦ S n R, R ◦ S 1 ◦ . . . ◦ S n R, or S 1 ◦ . . . ◦ S n ◦ R R such that R ∈ NR is a (non-inverse) role name, and S i ≺ R for i = 1, . . . , n. The set of simple roles for some Rbox is defined inductively: • If a role R occurs only on the right-hand-side of RIAs of the form S R such that S is simple, then R is also simple. • The inverse of a simple role is simple. Definition 2 Given a SROIQ Rbox R, the set of concept expressions C is defined as follows: • NC ⊆ C, ∈ C, ⊥ ∈ C, • if C, D ∈ C, R ∈ R, S ∈ R a simple role, a ∈ NI , and n a nonnegative integer, then ¬C, C  D, C  D, {a}, ∀R.C, ∃R.C, ∃S .Self, ≤n S .C, and ≥n S .C are also concept expressions. Throughout this paper, the symbols C, D will be used to denote concept expressions. A SROIQ Tbox is a set of general concept inclusion axioms (GCIs) of the form C D. An individual assertion can have any of the following forms: C(a), R(a, b), ¬R(a, b), a  b, with a, b ∈ NI individual names, C ∈ C a concept expression, and R, S ∈ R roles with S simple. A SROIQ Abox is a set of individual assertions. A SROIQ knowledge base KB is the union of a regular Rbox R, and an Abox A and Tbox T for R. The standard semantics of the above constructs is recalled in [9].

3 DESCRIPTION LOGIC RULES We introduce DL rules as a syntactic fragment of first-order logic. Definition 3 Consider some description logic L with concept expressions C, individual names NI , roles R (possibly including inverse roles), and let V be a countable set of first-order variables. Given terms t, u ∈ NI ∪ V, a concept atom (role atom) is a formula of the form C(t) (R(t, u)) with C ∈ C (R ∈ R). To simplify notation, we often use finite sets S of (role and concept)  atoms for representing the conjunction S . Given such a set S of atoms and terms t, u ∈ NI ∪ V, a path from t to u in S is a non-empty sequence R1 (x1 , x2 ), . . . , Rn (xn , xn+1 ) ∈ S where x1 = t, xi ∈ V for 2 ≤ i ≤ n, xn+1 = u, and xi  xi+1 for 1 ≤ i ≤ n. A term t in S is initial (resp. final) if there is no path to t (resp. no path starting at t). Given sets B and H of atoms, and a set x ⊆ V of all variables in   B∪H, a description logic rule (DL rule) is a formula ∀x. B → H such that (R1) for any u ∈ NI ∪ V that is not initial in B, there is a path from exactly one initial t ∈ NI ∪ V to u in B, (R2) for any t, u ∈ NI ∪ V, there is at most one path in B from t to u, (R3) if H contains an atom C(t) or R(t, u), then t is initial in B. Here ∀x for x = {x1 , . . . , xn } abbreviates an arbitrary sequence ∀x1 . . . . ∀xn . Since we consider only conjunctions with all variables  quantified, we will often simply write B → H instead of ∀x. B →  H. A rule base RB for some DL L is a set of DL rules for L.

81

The semantics of DL rules in the context of a description logic knowledge base is given by interpreting both the rules and knowledge base as first-order theories in the usual way, and applying the standard semantics of predicate logic. This has been discussed in the context of SWRL in [7], and we will not repeat the details here. Definition 3 ensures that role atoms in rule bodies essentially form a (set of) directed trees, starting at initial elements. Using the wellknown equivalence of formulae {p → q1 ∧ q2 } and {p → q1 , p → q2 }, one can transform any rule into an equivalent set of rules without conjunctions in rule heads. This can be done in linear time, so we assume without loss of generality that all DL rules are of this form. Since all DLs considered herein support nominals, we can also assume that all terms in rules are variables. Indeed, any atom C(a) with a ∈ NI can be replaced by C(x)∧{a}(x) with x ∈ V a new variable. Using inverse roles, role atoms with individual names can be replaced by concept atoms as follows: R(x, a) becomes ∃R.{a}(x), R(a, y) becomes ∃ Inv(R).{a}(y), and R(a, b) becomes ∃R.{b}(x)∧{a}(x). A similar transformation is possible for rule heads, where generated concept atoms {a}(x) are again added to the rule body. Before considering the treatment of DL rules in concrete DLs, we highlight some relevant special applications of DL rules. Concept products Rules of the form C(x) ∧ D(y) → R(x, y) can encode concept products (sometimes written C × D R) asserting that all elements of two classes must be related [13]. Examples include statements such as Elephant(x) ∧ Mouse(y) → biggerThan (x, y) or Alkaline(x) ∧ Acid(y) → neutralises (x, y). Local reflexivity, universal role Rules of the forms C(x) → R(x, x) and R(x, x) → C(x) can replace the SROIQ Tbox expression C ∃R.Self and ∃R.Self C. The universal role U of SROIQ can be defined as (x) ∧ (y) → U(x, y). Hence, a DL that permits such rules does not need to explicitly introduce those constructs. Qualified RIAs DL rules of course can express arbitrary role inclusion axioms, but they also can state that a RIA applies only to instances of certain classes. Examples include Woman(x) ∧ hasChild(x, y) → motherOf(x, y) and trusts(x, y) ∧ Doctor(y) ∧ recommends(y, z) ∧ Medicine(z) → buys(x, z).

4 DL RULES IN SROIQ We now show how knowledge bases of such rules can be internalised into the DL SROIQ. Since SROIQ supports inverse roles, it turns out that one can relax condition (R1) of DL rules as follows: (R1’) for any u ∈ NI ∪ V that is not initial in B, there is a path from one or more initial elements t ∈ NI ∪ V to u in B. On the other hand, we need to adopt the notions of regularity and simplicity to DL rule bases in SROIQ, which again restricts the permissible rule bases: Definition 4 Consider a rule base RB and a knowledge base KB for SROIQ. The set of simple roles of KB ∪ RB is the smallest set of roles containing every role R for which the following hold: • If R or Inv(R) occur on the right-hand-side of some RIA of KB, then this RIA is of the form S R or S Inv(R), and S is simple. • If R or Inv(R) occur in some rule head of the form R(x, y) or Inv(R)(x, y) in RB, then the according rule body is of the form S (x, y) with S simple, or of the form C(x) where x = y. Note that this is indeed a proper inductive definition, where roles that do not occur on the right of either RIAs or rules form the base case.

82

M. Krötzsch et al. / Description Logic Rules

The extended knowledge base KB ∪ RB is admissible for SROIQ if all roles S (i) occurring in concept (sub)expressions of the form ≤n S .C, ≥n S .C, ∃S .Self, and Dis(S 1 , S 2 ), and in role atoms of the form S (x, x) (x ∈ V) are simple. An extended knowledge base KB ∪ RB is regular if there is a strict partial order ≺ on R such that • S ≺ R iff Inv(S ) ≺ R, • the role box of KB is regular w.r.t. ≺, and • for any rule B → R(x, y), each S (z, v) ∈ B satisfies one of the following: – S ≺ R, or – there is no path from v to y, or – S = R, there is no other R(z , v ) ∈ B with a path from v to y, and we find that: either x = z and there is no C(x) ∈ B, or y = v and there is no C(y) ∈ B. Note that RIAs in regular SROIQ knowledge bases are allowed to have two special forms for transitivity and symmetry, which we do omit for the definition of regularity in DL rules to simplify notation. Since S in S (x, x) is simple, we can replace such role atoms by concept atoms C(x) where C is a new concept name for which a new axiom C ≡ ∃S .Self is added. We will thus assume that no role atoms of this form occur in admissible knowledge bases. One can now show that checking the satisfiability of extended SROIQ knowledge bases that are admissible and regular is decidable, and has the same worst-case complexity as reasoning in SROIQ. This is achieved by a polynomial transformation of rule bases into SROIQ axioms. The first step of doing this is to replace “dead branches” of the tree-shaped query body by DL concepts. The proof is a variation of the “rolling-up” technique used for conjunctive query answering [2]. Lemma 5 Any DL rule B → H for SROIQ can be transformed into a semantically equivalent rule B → H such that all paths in B are contained in a single maximal path. If H = R(x, y), then y is the final element of that maximal path, and if H = C(x) then there are no paths in B. A rule with these properties is called linearised. As an example, the DL rule that was given in the introduction can be simplified to yield (using “ , ” instead of “∧” for brevity): ∃ worksAt.University(x), supervises(x,z), PhDStudent(z)→profOf(x,z) The above transformation allows us to reduce tree-shaped rules to rules of only linear structure that are much more similar to RIAs in SROIQ. But while all role atoms now belong to a single maximal path, rules might still contain disconnected concept atoms. The rule R(x, y) ∧ S (u, v) ∧ C(z) → T (x, v), e.g., is rewritten to ∃R. (x) ∧ S (u, v) ∧ C(z) → T (x, v). Now it can be shown that DL rules in SROIQ can be internalised. Theorem 6 Consider a rule base RB and a knowledge base KB for SROIQ, such that RB∪KB is admissible. There is a SROIQ knowledge base KBRB that can be computed in time polynomial in the size of RB, such that KB ∪ RB and KB ∪ KBRB are equisatisfiable. Moreover, if KB ∪ RB is regular, then KB ∪ KBRB is also regular. Proof. We can assume rules in RB to have the form as in Lemma 5, since the transformation given in [9] preserves regularity and simplicity in KB ∪ RB. We assume w.l.o.g. that no rule in RB has the universal role U in its head: such rules would be tautological. We

further assume that all variables occurring in rule heads also occur in their body – atoms of the form (x) can safely be added to that end. For any rule B → R(x, y), Lemma 5 asserts that B contains at most one maximal path with final element y, and all role atoms of B (if any) are part of that path. Let z be the initial element of this path if it exists, and let z be y otherwise. If x  z, then x occurs in B only in concept atoms C(x), and we can add a role atom U(x, z) to B without violating (R1)–(R3). This change preserves the semantics of the rule since U(x, z) is true for any variable assignment (mapping free variables to domain elements of I; sometimes also called variable binding [7]) in any interpretation. Regularity of the role base is preserved since we can assume w.l.o.g. U to be the least element of ≺ (exploiting that U does not occur in rule heads). Simplicity is no concern as R by assumption is not a simple role in KB∪RB. In summary, we may transform the body of any rule with head R(x, y) to contain exactly one maximal path, leading from x to y. We describe the step-wise computation of KBRB . Initially, we set KBRB  ∅, and define the set of remaining rules as RB  RB. The reduction proceeds iteratively until RB is empty. In every step, we select some rule B → H ∈ RB. As discussed above, there is only a single maximal path of roles in B, all role atoms in B are part of that path, and all but adjacent variables in the path are distinct (no cycles). We distinguish five cases: (1) If B contains atoms D(z) and D (z) for some variable z, then these atoms are replaced in B by a new atom (D  D )(z). (2) Otherwise, if H = C(x) and B = D(x), then B → H is removed from RB , and a Tbox axiom D C is inserted into KBRB . (3) Otherwise, if H = R(x, y) and B is of the form {R1 (x, x2 ), . . . , Rn (xn , y)}, then B → H is removed from RB , and an Rbox axiom R1 ◦ . . . ◦ Rn R is inserted into KBRB . (4) Otherwise, if H = R(x, y), and there is D(z) ∈ B such that z occurs in a role atom of B or H (in first or second argument position), then the following is done. First, a new role name S is introduced, and the Tbox axiom D ≡ ∃S .Self is added to KBRB . Second, a new variable z ∈ V is introduced, the role atom S (z, z ) is added to B, every role atom T (x , z) ∈ B is replaced by T (x , z ), and every role atom T (z, y ) ∈ B is replaced by T (z , y ). Finally, D(z) is removed from B, and if z = y then the rule head is replaced by R(x, z ). (5) Otherwise, if H = C(x) or H = R(x, y), and there is some D(z) ∈ B such that z occurs neither in H nor in any role atom of B, then the following is done. If B contains some atom of the form R(x, t) so there is no atom D (x) ∈ B, then define u  y; otherwise define u  x. Now D(z) in B is replaced by the atom ∃U.D(u). The correctness of this procedure can be established by verifying the following claims: 1. 2. 3. 4. 5.

The cases distinguished by the algorithm are exhaustive. The algorithm terminates after a polynomial number of steps. After termination, KB ∪ KBRB is a SROIQ knowledge base. After termination, KB ∪ RB and KB ∪ KBRB are equisatisfiable. If KB ∪ RB is regular, then so is KB ∪ KBRB .

Details on the proofs of those claims can be found in [9].



Considering again our introductory example, we arrive at the following SROIQ axioms (where S 1 , S 2 are new auxiliary roles): S 1 ◦ supervises ◦ S 2 profOf ∃ worksAt.University ≡ ∃S 1 .Self PhDStudent ≡ ∃S 2 .Self Based on Theorem 6, we conclude that the problem of checking the

M. Krötzsch et al. / Description Logic Rules

satisfiability of SROIQ knowledge bases extended with DL rules is decidable, as long as the extended knowledge base is admissible and regular. Since the internalisation is possible in polynomial time, the worst-case complexity for this problem is the same as for checking satisfiability of SROIQ knowledge bases.

5 DL RULES IN EL++ In this section, we investigate DL rules for the DL EL++ [1], for which many typical inference problems can be solved in polynomial time. As EL++ cannot internalise DL rules, they constitute a true extension of expressivity. We therefore take a different approach than in SROIQ: instead of considering rule bases as an auxiliary set of axioms that is successively reduced and internalised, we introduce DL rules as core expressive mechanism to which all other EL++ axioms can be reduced. While EL++ rule bases offer many expressive features formerly unavailable in EL++ , we show that the complexity of core inference problems remains tractable. We simplify our presentation by omitting concrete domains from EL++ – they are not affected by our extension and can be treated as shown in [1]. Definition 7 A role of EL++ is a (non-inverse) role name. An EL++ Rbox is a set of generalised role inclusion axioms, and an EL++ Tbox (Abox) is a SROIQ Tbox (Abox) that contains only the following concept constructors: , ∃, , ⊥, as well as nominal classes {a}. An EL++ knowledge base is the union of an EL++ Rbox, Tbox and Abox. An EL++ rule base is a set of DL rules for EL++ that do not contain atoms of the form R(x, x) in the body. Note that we do not have any requirement for regularity or simplicity of roles in the context of EL++ . It turns out that neither is relevant for obtaining decidability or tractability. The case of R(x, x) in bodies is not addressed by the below algorithm – [10] significantly extends the below approach to cover this and other features. Since it is obvious that both concept and role inclusion axioms can directly be expressed by DL rules, we will consider only EL++ rule bases without any additional EL++ knowledge base axioms. We can restrict our attention to EL++ rules in a certain normal form: Definition 8 An EL++ rule base RB is in normal form if all concept atoms in rule bodies are either concept names or nominals, all variables in a rule’s head also occur in its body, and all rule heads are of one of the following forms: A(x) ∃R.A(x) R(x, y) where A ∈ NC ∪ {{a} | a ∈ NI } ∪ { , ⊥} and R ∈ NR . A set B of basic concept expressions for RB is defined as B  {C | C ∈ NC , C occurs in RB} ∪ {{a} | a ∈ NI , a occurs in RB} ∪ { , ⊥}. Proposition 9 Any EL++ rule base can be transformed into an equisatisfiable EL++ rule base in normal form. The transformation can be done in polynomial time. When checking satisfiability of EL++ rule bases, we can thus restrict to rule bases in the above normal form. A polynomial algorithm for checking class subsumptions in EL++ knowledge bases has been given in [1], and it was shown that other standard inference problems can easily be reduced to that problem. We now present a new algorithm for checking satisfiability of EL++ rule bases, and show its correctness and tractability. Clearly, subsumption checking can be reduced to this problem: given a new individual a ∈ NI , the rule base RB ∪ {C(a), {a}(x)  D(x) → ⊥(x)} is unsatisfiable iff RB entails C D. Instance checking in turn is directly reducible to subsumption checking in the presence of nominals.

83

Algorithm 10 The algorithm proceeds by computing two sets: a set E of inferred “domain elements”, and a set S of relevant subclass inclusion axioms that are entailed by RB. The elements of E are represented by basic concept expressions of RB, i.e. E ⊆ B, and the inclusion axioms in S are of the form C D or C ∃R.D, where C, D ∈ E. Thus E and S are polynomially bounded by the size of RB. Initially, we set E  {{a} | {a} ∈ B} ∪ { } and S  ∅. Now a DL rule is applied whenever we find that there is a match with the rule body. Given a rule B → H, a match θ is a mapping from all variables in B to elements of E, such that the following hold: • for every C(y) ∈ B, θ(y) C ∈ S, and • for every R(y, z) ∈ B, θ(y) ∃R.θ(z) ∈ S. The algorithm now proceeds by applying the following rules until no possible rule application further modifies the set E or S: (EL1) If C ∈ E, then S  S ∪ {C C, C }. (EL2) If there is a rule B → E(x) ∈ RB, and if there is a match θ for B with θ(x) = θx , then S  S ∪ {θx E}. In this case, if E = C or E = ∃R.C, then E  E ∪ {C}. (EL3) If there is a rule B → R(x, y) ∈ RB, and if there is a match θ for B with θ(x) = θx and θ(y) = θy , then S  S∪{θx ∃R.θy }. (EL4) If {C {a}, D {a}, D E} ⊆ S then S  S ∪ {C E}. Here we assume that C, D, D ∈ B, E ∈ B ∪ {∃R.C | C ∈ B}, and R ∈ NR . After termination, the algorithm returns “unsatisfiable” if ⊥ ∈ E, and “satisfiable” otherwise. The correctness of the above algorithm is shown by using the sets E and S for constructing a model, which is indeed possible whenever ⊥  E (see [9] for details). EL++ has a small model property (a consequence of the proofs for [1]) that allows us to consider at most one individual for representing the members of each class. The set E thus is used to record classes which must have some element, and matches θ use these class names to represent (arbitrary) individuals to which some DL rule might be applied. Assuming that all steps of Algorithm 10 are computable in polynomial time, it is easy to see that the algorithm also terminates in polynomial time, since there are only polynomially many possible elements for E and S, and each case adds new elements to either set. However, it also has to be verified that individual steps can be computed efficiently, and this is not obvious for the match-checks in (EL2) and (EL3). Indeed, finding matches in query graphs is known to be NP-complete in general, and the tree-like structure of queries is crucial to retain tractability. Moreover, even tree-like rule bodies admit exponentially many matches. But note that Algorithm 10 does not consider all matches but only the (polynomially many) possible values of θx (and θy ). It turns out that there is indeed a algorithm that checks in polynomial time whether a match θ as in (EL2) and (EL3) exists, but without explicitly considering all possible matches. This task is closely related to the problem of testing the existence of homomorphisms between trees and graphs. Proposition 11 Consider a rule of the form B → C(x) (B → R(x, y)), sets E and S as in Algorithm 10, and an element θx ∈ E (elements θx , θy ∈ E). There is an algorithm that decides whether there is a match θ such that θ(x) = θx (θ(x) = θx and θ(y) = θy ), running in polynomial time w.r.t. the size of the inputs. Theorem 12 Algorithm 10 is a sound and complete procedure for checking satisfiability of EL++ rule bases. Satisfiability checking, instance retrieval, and computing class subsumptions for EL++ rule bases is possible in polynomial time in the size of the rule base.

84

M. Krötzsch et al. / Description Logic Rules

6 DLP 2 Description Logic Programs (DLP) have been proposed as a tractable formalism for bridging the gap between DL and (Horn) logic programming [5]. DLP can naturally be extended with DL rules and various other features of SROIQ, and the resulting tractable rule language might be dubbed DLP 2 in analogy to the ongoing standardisation of OWL 2. A detailed syntactic characterisation for DLP was given in [14], and a yet more general formulation can be obtained from [8]. Here, we adopt a much simpler definition that focusses on the essential expressive features only: Definition 13 Roles of DLP are defined as in SROIQ, including inverse roles. A DLP body concept is any SROIQ concept expression that includes only concept names, nominals, , ∃, , and ⊥. A DLP head concept is any SROIQ concept expression that includes only concept names, nominals, , ∀, , ⊥, and expressions of the form ≤1.C where C is a DLP body concept. A DLP knowledge base is a set of Rbox axioms of the form R S and R ◦ R R, Tbox axioms of the form C D, and Abox axioms of the form D(a) and R(a, b), where C ∈ C is a body concept, D ∈ C is a head concept, and a, b ∈ NI are individual names. A DLP rule base is a set of DL rules such that all concepts in rule bodies are body concepts, and all concepts in rule heads are head concepts. A DLP 2 knowledge base consists of a DLP knowledge base that additionally might contain Rbox axioms of the form Dis(R, S ) and Asy(R), together with some DLP rule base. Dis and Asy assert role disjointness and asymmetry as explained in [9]. Note that neither regularity nor simplicity restrictions apply in DLP. Combining “rolling-up” as in Lemma 5 with a decomposition of the remaining single paths to multiple rules, any DLP 2 knowledge base can be transformed into an equisatisfiable set of function-free first-order Horn rules with at most five variables per formula, and this transformation is possible in polynomial time. Since this fragment of Horn-logic is tractable, we can conclude the following:

Theorem 14 Satisfiability checking, instance retrieval, and computing class subsumptions for DLP 2 knowledge bases is possible in polynomial time in the size of the knowledge base.

7 CONCLUSION We have introduced DL rules as a rule-based formalism for augmenting description logic knowledge bases. For all DLs considered in this paper – SROIQ, EL++ , and DLP – the extension with DL rules does not increase the worst-case complexity. In particular, EL++ rules and the extended DLP 2 allow for polynomial time reasoning for common inference tasks, even though DL rules do indeed provide added expressive features in those cases. The main contributions of this paper therefore are twofold. Firstly, we have extended the expressivity of two tractable DLs while preserving their favourable computational properties. The resulting formalisms of EL++ rules and DLP 2 are arguably close to being maximal tractable fragments of SROIQ. In particular, note that the union of EL++ and DLP is no longer tractable, even when disallowing number restrictions and inverse roles: this follows from the fact that this DL contains the DL Horn-FLE which was shown to be ExpTimecomplete in [8]. Secondly, while DL rules do not truly add expressive power to SROIQ, our characterisation and reduction methods for DL rules

provides a basis for developing ontology modelling tools. Indeed, even without any further extension, the upcoming OWL 2 standard would support all DL rules. Hence OWL-conformant tools can choose to provide rule-based user interfaces (as done for Protégé in [4]), and rule-based tools may offer some amount of OWL support. We remark that in the case of DLP and EL++ , the conditions imposed on DL rules can be checked individually for each rule without considering the knowledge base as a whole. Moreover, in order to simplify rule editing, the general syntax of DL rules can be further restricted without sacrificing expressivity, e.g. by considering only chains rather than trees for rule bodies. We thus argue that DL rules can be a useful interface paradigm for many application fields. Our treatment of rules in EL++ and DLP 2 – used only for establishing complexity bounds in this paper – can be the basis for novel rule-based reasoning algorithms for those DLs, and we leave it for future research to explore this approach.

ACKNOWLEDGEMENTS Research reported herein was supported by the EU in the IST projects ACTIVE (IST-2007-215040) and NeOn (IST-2006-027595), and by the German Research Foundation under the ReaSem project.

REFERENCES [1] Franz Baader, Sebastian Brandt, and Carsten Lutz, ‘Pushing the EL envelope’, in Proc. 19th Int. Joint Conf. on Artificial Intelligence (IJCAI 2005), Edinburgh, UK, (2005). Morgan-Kaufmann Publishers. [2] Diego Calvanese, Giuseppe De Giacomo, and Maurizio Lenzerini, ‘On the decidability of query containment under constraints’, in Proc. 17th ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS’98), pp. 149–158. ACM Press, (1998). [3] Evgeny Dantsin, Thomas Eiter, Georg Gottlob, and Andrei Voronkov, ‘Complexity and expressive power of logic programming’, ACM Computing Surveys, 33, 374–425, (2001). [4] Francis Gasse, Ulrike Sattler, and Volker Haarslev. Rewriting rules into SROIQ axioms. Poster at 21st Int. Workshop on DLs (DL-08), 2008. [5] Benjamin N. Grosof, Ian Horrocks, Raphael Volz, and Stefan Decker, ‘Description logic programs: combining logic programs with description logic’, in Proc. 12th Int. Conf. on World Wide Web (WWW 2003), pp. 48–57. ACM, (2003). [6] Ian Horrocks, Oliver Kutz, and Ulrike Sattler, ‘The even more irresistible SROIQ’, in Proc. 10th Int. Conf. on Principles of Knowledge Representation and Reasoning (KR2006), pp. 57–67. AAAI Press, (2006). [7] Ian Horrocks and Peter F. Patel-Schneider, ‘A proposal for an OWL rules language’, in Proc. 13th Int. Conf. on World Wide Web (WWW 2004), eds., Stuart I. Feldman, Mike Uretsky, Marc Najork, and Craig E. Wills, pp. 723–731. ACM, (2004). [8] Markus Krötzsch, Sebastian Rudolph, and Pascal Hitzler, ‘Complexity boundaries for Horn description logics’, in Proc. 22nd AAAI Conf. (AAAI’07), (2007). [9] Markus Krötzsch, Sebastian Rudolph, and Pascal Hitzler, ‘Description logic rules (extended technical report)’, Technical report, Universität Karlsruhe, Germany, (FEB 2008). Available at http://korrekt. org/page/SROIQ_Rules. [10] Markus Krötzsch, Sebastian Rudolph, and Pascal Hitzler, ‘ELP: Tractable rules for OWL 2’, Technical report, Universität Karlsruhe, Germany, (May 2008). http://korrekt.org/page/ELP. [11] Alon Y. Levy and Marie-Christine Rousset, ‘Combining Horn rules and description logics in CARIN’, Artificial Intelligence, 104, 165–209, (1998). [12] Boris Motik, Ulrike Sattler, and Rudi Studer, ‘Query answering for OWL-DL with rules’, J. Web Sem., 3(1), 41–60, (2005). [13] Sebastian Rudolph, Markus Krötzsch, and Pascal Hitzler, ‘All elephants are bigger than all mice’, in Proc. 21st Int. Workshop on Description Logics (DL-08), (2008). [14] Raphael Volz, Web Ontology Reasoning with Logic Databases, Ph.D. dissertation, Universität Karlsruhe (TH), Germany, 2004.

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-85

85

Conflicts between Relevance-Sensitive and Iterated Belief Revision Pavlos Peppas and Anastasios Michael Fotinopoulos and Stella Seremetaki1 Abstract. The original AGM paradigm focuses only on one-step belief revision and leaves open the problem of revising a belief state with whole sequences of evidence. Darwiche and Pearl later addressed this problem by introducing extra (intuitive) postulates as a supplement to the AGM ones. A second shortcoming of the AGM paradigm, seemingly unrelated to iterated revision, is that it is too liberal in its treatment of the notion of relevance. Once again this problem was addressed with the introduction of an extra (also very intuitive) postulate by Parikh. The main result of this paper is that Parikh postulate for relevance-sensitive belief revision is inconsistent with each of the Darwiche and Pearl postulates for iterated belief revision.

1

INTRODUCTION

The original AGM paradigm for belief revision, [1, 3, 11], focuses only on one-step transitions leaving open the problem of how to revise a belief state with a whole sequence of evidence. This problem was later addressed by Darwiche and Pearl who formulated four intuitive new postulates (known as the DP postulates) to regulate iterated revisions. Possible world semantics were introduced to characterize the new postulates, and with some adjustments (see section 4) the DP postulates were shown to be compatible with the original AGM ones.2 Although Darwiche and Pearl’s work has received some criticism, it remains very influential in the literature of iterated belief revision and has served as a basis for further developments in the area [7, 5]. A shortcoming of a different nature of the original AGM paradigm is that it neglects the important role of relevance in belief revision. As noted by Parikh, [8], when a belief state ψ is revised by new information μ, only the part of ψ that is related to μ should be effected; the rest of ψ should remain the same. Parikh proceeded to formulate a postulate, called (P), that captures this intuition (albeit in limited cases). Postulate (P) was later shown to be consistent with the AGM postulates and possible-world semantics were introduced to characterize it, [10]. The main contribution of this paper is to show that, in the presence of the AGM postulates, Parikh postulate for relevance-sensitive belief revision is inconsistent with each of the (seemingly unrelated) Darwiche and Pearl postulates for iterated belief revision. This of course is quite disturbing. Both the concept of relevance and the process of iteration are key notions in Belief Revision and 1

2

Dept of Business Administration, University of Patras, Patras 26500, Greece, emails: [email protected], [email protected], [email protected] To be precise, the DP postulates were shown to be compatible with the reformalization of the AGM postulates introduced by Katsuno and Mendelzon in [6].

we can do away with neither. Moreover, the encoding of these notions proposed by Parikh, Darwiche, and Pearl appears quite natural and it is not obvious how one should massage the postulates in order to reconcile them. Further to this point, subsequent postulates introduced to remedy problems with the (DP) ones, [7, 5], are also shown to be incompatible with postulate (P) (see section 6). On the positive side, these incompatibility results reveal a hitherto unknown connection between relevance and iteration that deepens our understanding of the belief revision process. The paper is structured as follows. The next section introduces some notation and terminology. Following that we briefly review (Katsuno and Mendelzon’s re-formalization of) the AGM postulates, Darwiche and Pearl’s approach for iterated revisions, and Parikh’s proposal for relevance-sensitive belief revision (sections 3, 4, and 5). Section 6 contains our main incompatibility results. Finally in section 7 we make some concluding remarks.

2

PRELIMINARIES

Throughout this paper we shall be working with a finitary propositional language L. We shall denote the (finite) set of all propositional variables of L by A. For a set of sentences Γ of L, we denote by Cn(Γ) the set of all logical consequences of Γ, i.e. Cn(Γ) = {ϕ ∈ L: Γ % ϕ}. A theory K of L is any set of sentences of L closed under %, i.e. K = Cn(K). We shall denote the set of all theories of L by TL . A theory K of L is complete iff for all sentences ϕ ∈ L, ϕ ∈ K or ¬ϕ ∈ K. As it is customary in Belief Revision, herein we shall identify consistent complete theories with possible worlds. We shall denote the set of all consistent complete theories of L by ML . If for a sentence ϕ, Cn(ϕ) is complete, we shall also call ϕ complete. For a set of sentences Γ of L, [Γ] denotes the set of all consistent complete theories of L that contain Γ. Often we shall use the notation [ϕ] for a sentence ϕ ∈ L, as an abbreviation of [{ϕ}]. When two sentences ϕ and χ are logically equivalent we shall often write ϕ ≡ χ as an abbreviation of % ϕ ↔ χ. Finally, the symbols  and ⊥ will be used to denote an arbitrary (but fixed) tautology and contradiction of L respectively.

3

THE KM POSTULATES

In the AGM paradigm belief sets are represented as logical theories, new evidence as sentences of L, and the process of belief revision is modeled as a function mapping a theory K and a sentence μ to a new theory K ∗ μ. Moreover, eight postulates for ∗ are proposed, known as the AGM postulates, that aim to capture the notion of rationality in belief revision. Katsuno and Mendelzon in [6] slightly reshaped the AGM constituents to make the formalization more amendable to implementa-

86

P. Peppas et al. / Conflicts Between Relevance-Sensitive and Iterated Belief Revision

tion. In particular, the object language L is set to be a finitary propositional one,3 and belief sets are defined as finite sets of sentences of L. Since a belief set contains only finitely many elements, one can in fact represent it as a single sentence ψ; namely the conjunction of all its elements. This is the representation eventually adopted in [6] and the one we shall use herein. To emphasize the finiteness of the new representation we shall call ψ a belief base and reserve the term belief set for the closure of ψ, i.e. the theory Cn(ψ). With the above reformulation, a revision function becomes a function ∗ mapping a sentence ψ and a sentence μ to a new sentence ψ ∗ μ; i.e. ∗ : L × L → L. Moreover in the new formalization the AGM postulates are equivalent to the following six, known as the KM postulates: (KM1) (KM2) (KM3) (KM4) (KM5) (KM6)

4

ψ ∗ μ % μ. If ψ ∧ μ is satisfiable then ψ ∗ μ ≡ ψ ∧ μ. If μ is satisfiable then ψ ∗ μ is also satisfiable. If ψ1 ≡ ψ2 and μ1 ≡ μ2 then ψ1 ∗ μ1 ≡ ψ2 ∗ μ2 . (ψ ∗ μ) ∧ ϕ % ψ ∗ (μ ∧ ϕ). If (ψ ∗μ)∧ϕ is satisfiable then ψ ∗(μ∧ϕ) % (ψ ∗μ)∧ϕ.

ITERATED BELIEF REVISION

One thing to notice about the KM postulates is that they all refer to single-step revisions; no constraints are placed on how the revision policy at the initial belief base ψ may relate to the revision policies at it descendants (i.e. at the belief bases resulting from ψ via a sequence of revisions). Darwiche and Pearl’s solution to this problem came in the form of four additional postulates, known as the DP postulates, listed below, [2]: (DP1) (DP2) (DP3) (DP4)

If ϕ % μ then (ψ ∗ μ) ∗ ϕ = ψ ∗ ϕ. If ϕ % ¬μ then (ψ ∗ μ) ∗ ϕ = ψ ∗ ϕ. If ψ ∗ ϕ % μ then (ψ ∗ μ) ∗ ϕ % μ. If ψ ∗ ϕ % ¬μ then (ψ ∗ μ) ∗ ϕ % ¬μ.

The DP postulates are very intuitive and their intended interpretation, is loosely speaking as follows (see [2, 7] for details). Postulate (DP1) says that if the subsequent evidence ϕ is logically stronger than the initial evidence μ then ϕ overrides whatever changes μ may have made. (DP2) says that if two contradictory pieces of evidence arrive sequentially one after the other, it is the later that will prevail. (DP3) says that if revising ψ by ϕ causes μ to be accepted in the new belief base, then revising first by μ and then by ϕ can not possibly block the acceptance of μ. Finally, (DP4) captures the intuition that “no evidence can contribute to its own demise” [2]; if the revision of ψ by ϕ does not cause the acceptance of ¬μ, then surely this should still be the case if ψ is first revised by μ before revised by ψ. An initial problem with the DP postulates (more precisely with (DP2)) was that they were inconsistent with the KM postulates. However this inconsistency was not deeply rooted and was subsequently resolved. There are in fact (at least) two ways of removing it. The first, proposed by Darwiche and Pearl themselves, [2], involves substituting belief bases with belief states, and adjusting the KM postulates accordingly. The second, proposed by Nayak, Pagnucco, and Peppas, [7], keeps belief bases as the primary objects of change, but modifies the underlying assumptions about the nature of ∗. 3

In the original AGM paradigm, the object language L is not necessarily finitary nor propositional. The details of L are left open and only a small number of structural constraints are assumed of L and its associated entailment relation  (see [3, 11]).

In particular, notice that, with the exception of (KM4), the KM postulates apply only to a single initial belief base ψ; no reference to other belief bases are made. Even postulate (KM4) can be weakened to comply with this policy: (KM4)’

If μ1 ≡ μ2 then ψ ∗ μ1 ≡ ψ ∗ μ2 .

With the new version of (KM4), the KM postulates allow us to define a revision function as a unary function ∗ : L → L, mapping the new evidence μ to a new belief base ∗ψ (μ), given the initial belief base ψ as background. This is the first modification made by Nayak et al. The second is to make revision functions dynamic. That is, revision functions may change as new evidence arrives. With this relaxation it is possible for example to have one revision function ∗ψ associated initially with ψ, and a totally different one after the revision of ψ by a sequence of evidence that have made the full circle and have converted ψ back to itself.4 Notice that the weakening of (KM4) to (KM4)’ is consistent with such dynamic behavior. As shown by Nayak et al., these two modifications suffice to reconcile the DP postulates with the KM ones, and it is these modifications we shall adopt for the rest of the paper.5 Hence for the rest of the paper, unless specifically mentioned otherwise, we shall use the term “KM postulates” to refer to (KM1)-(KM6) with (KM4) replaced by (KM4)’, and we shall assume that revision function are unary and dynamic. We close this section with a remark on notation. Although we assume that revision functions are unary (relative to some background belief base ψ), for ease of presentation we shall keep the original notation and denote the revision of ψ by μ as ψ ∗ μ rather than ∗ψ (μ).

5

RELEVANCE-SENSITIVE BELIEF REVISION

Leaving temporarily aside the issue of iterated belief revision, we shall now turn back to one-step revisions to review the role of relevance in this process. As already mentioned in the introduction, Parikh in [8] pointed out that the AGM/KM postulates fail to capture the intuition that during the revision of a belief base ψ by μ, only the part of ψ that is related to μ should be effected, while everything else should stay the same. Of course determining the part of ψ that is relevant to some new evidence μ is not a simple matter. There is however at least one special case where the role of relevance can be adequately formalized; namely, when it is possible to break down ψ into two (syntactically) independent parts such that only the fist of the two parts is syntactically related to the new evidence μ. More precisely, for a sentence ϕ of L, we shall denote by Aϕ the smallest set of propositional variables, through which a sentence that is logically equivalent to ϕ can be expressed. For example, if ϕ is the sentence (p ∨ q ∨ z) ∧ (p ∨ q ∨ ¬z), then Aϕ = {p, q}, since ϕ is logically equivalent to p ∨ q, and no sentence with fewer propositional variables is logically equivalent to ϕ. We shall denote by Lϕ the propositional sublanguage built from Aϕ via the usual boolean connectives. By Lϕ we shall denote the sublanguage built from the complement of Aϕ , i.e. from A − Aϕ . Parikh proposed the following postulate to capture the role of relevance in belief revision (at least for the special case mentioned above):6 4 5 6

In such cases, although the sequence of evidence does not effect the beliefs of the agent, it does however change the way the agent reacts to new input. It should be noted though that our results still hold even if Darwiche and Pearl’s proposal of switching to belief states was adopted. The formulation of (P) in [8] is slightly different from the one presented below since Parikh was working with theories rather than belief bases. The two version are of course equivalent.

P. Peppas et al. / Conflicts Between Relevance-Sensitive and Iterated Belief Revision

(P)

If ψ ≡ χ ∧ ϕ where χ, ϕ are sentences of disjoint sublanguages Lχ , Lϕ respectively, and Lμ ⊆ Lχ , then ψ ∗ μ ≡ (χ ◦ μ) ∧ ϕ, where ◦ is a revision function of the sublanguage Lχ .

According to postulate (P), whenever it is possible to break down the initial belief base ψ into two independent parts χ and ϕ, and moreover it so happens that the new evidence μ can be expressed entirely in the language of the first part, then during the revision of ψ by μ, it is only the first part that is effected; the unrelated part ϕ crosses over to the new belief base verbatim. Notice that of the nature of the “local” revision operator ◦ and its relationship with the “global” revision operator ∗ is not clearly stated in (P). Peppas et al., [10], therefore proposed a re-formulation of axiom (P) in terms of two new conditions (R1) and (R2) that do not refer to a “local” revision operator ◦. Only the first of these two conditions will be needed herein: (R1)

If ψ ≡ χ∧ϕ, Lχ ∩Lϕ = ∅, and Lμ ⊆ Lχ , then Cn(ψ)∩Lχ = Cn(ψ ∗ μ) ∩ Lχ .

At first (R1) looks almost identical to (P) but it is in fact strictly weaker than it (see [10] for details). It is essentially condition (R1) that we will be using to derive our incompatibility results.

6

INCOMPATIBILITY RESULTS

As already announced in the introduction, we shall now prove that in the presence of the KM postulates, (R1) – and therefore (P) – is inconsistent with each of the postulates (DP1)-(DP4). The proof relies on the semantics characterization of these postulates so we shall briefly review it before presenting our results. We start with Grove’s seminal representation result [4] and its subsequent reformulation by Katsuno and Mendelzon [6]. Let ψ be a belief base and ≤ψ a total preorder in ML . We denote the strict part of ≤ψ by <ψ . We shall say that ≤ψ is faithful iff the minimal elements of ≤ψ are all the ψ-worlds:7 (SM1) (SM2)

If r ∈ [ψ] then r ≤ψ r for all r ∈ ML . If r ∈ [ψ] and r ∈ [ψ] then r <ψ r .

Given a belief base ψ and a faithful preorder ≤ψ associated with it, one can define a revision function ∗ : L → L as follows: (S*)

ψ ∗ μ = γ(min([μ], ≤ψ ).

In the above definition min([μ], ≤ψ ) is the set of minimal μworld with respect to ≤ψ , while γ is a function that maps a set of possible worlds S to a sentence γ(S) such that [γ(S)] = S. The preorder ≤ψ essentially represents comparative plausibility: the closer a world is to the initial worlds [ψ], the more plausible it is. Then according to (S*), the revision of ψ by μ is defined as the belief base corresponding to the most plausible μ-worlds. In [4, 6] it was shown that the function induced from (S*) satisfies the KM postulates and conversely, every revision function ∗ that satisfies the KM postulates can be constructed from a faithful preorder by means of (S*).8 7

8

In [6] a third constraint was required for faithfulness, namely that logically equivalent sentences are assigned the same preorders. This is no longer necessary given the new version of (KM4). We note that the weakening of (KM4) to (KM4)’ does not effect these results since it is accommodated by a corresponding weakening of the notion of faithfulness.

87

This correspondence between revision functions and faithful preorders can be preserved even if extra postulates for belief revision are introduced, as long as appropriate constraints are also imposed on the preorders. In particular, Darwiche and Pearl proved that the following four constraints (SI1)-(SI4) on faithful preorders correspond respectively to the four postulates (DP1)-(DP4). (SI1) (SI2) (SI3) (SI4)

If r, r  ∈ [μ] then r ≤ψ r iff r ≤ψ∗μ r . If r, r  ∈ [¬μ] then r ≤ψ r iff r ≤ψ∗μ r . If r ∈ [μ] and r ∈ [¬μ] then r <ψ r entails r <ψ∗μ r . If r ∈ [μ] and r ∈ [¬μ] then r ≤ψ r entails r ≤ψ∗μ r .

Notice that all of the above constraints make associations between the preorder ≤ψ related to the initial belief base ψ and the preorder ≤ψ∗μ related to the belief base that results from the revision of ψ by μ. The semantic constraint(s) corresponding to postulate (P) have also been fully investigated in [10].9 Herein however we shall focus only on condition (R1); in fact we shall be even more restrictive and consider only the semantic counterpart of (R1) in the special case of consistent and complete belief bases: (PS)

If Diff(ψ, r) ⊂ Diff(ψ, r ) then r <ψ r .

In the above condition, for any two worlds w, w , Diff(w, w ) represents the set of propositional variables that have different truth values in the two worlds; in symbols, Diff(w, w ) = {p ∈ A : w % p and w % p} ∪ {p ∈ A : w % p and w % p}. Whenever a sentence ψ is consistent and complete, we use Diff(ψ, w ) as an abbreviation of Diff(Cn(ψ), w ). Intuitively, (PS) says that the plausibility of a world r depends on the propositional variables in which it differs from the initial (complete) belief base ψ: the more the propositional variables in Diff(ψ, r) the less plausible r is. In [10] it was shown that, for the special case of consistent and complete belief bases, (PS) is the semantic counterpart of (R1); i.e. given a consistent and complete belief base ψ and a faithful preorder ≤ψ , the revision function ∗ produced from ≤ψ via (S*) satisfies (R1) iff ≤ψ satisfies (PS). Although it is possible to obtain a fully-fledged semantic characterization of postulate (P) by generalizing (PS) accordingly (see [10]) the above restricted version suffices to establish the promised results: Theorem 1 In the presence of the KM postulates, postulate (P) is inconsistent with each of the postulates (DP1)-(DP4). Proof. Since (P) entails (R1) it suffices to show that (R1) is inconsistent with each of (DP1)-(DP4). Assume that the object language L is built from the propositional variable p, q, and z. Moreover let ψ be the complete sentence p∧q∧z and let ≤ψ be the following preorder in ML : pqz <ψ pqz <ψ pqz <ψ pqz <ψ pqz <ψ pqz <ψ pqz <ψ pqz In the above definition of ≤ψ , and in order to increase readability, we have used sequences of literals to represent possible worlds (namely the literals satisfied by a world), and we have represented the negation of a propositional variable v by v. Notice that ≤ψ satisfies (PS). In what follows we shall construct sentences μ1 , μ2 , μ3 , and μ4 , such that no preorder satisfying (PS) and related to ψ ∗ μ1 (respectively to ψ ∗ μ2 , ψ ∗ μ3 , ψ ∗ μ4 ) can also 9

In the same paper axiom (P) was shown to be consistent with all the AGM/KM postulates.

88

P. Peppas et al. / Conflicts Between Relevance-Sensitive and Iterated Belief Revision

satisfy (SI1) (respectively (SI2), (SI3), (SI4)). Given the correspondence between (R1) and (PS) on one hand, and the correspondence between (DP1)-(DP4) and (SI1)-(SI4) on the other, this will suffice to prove the theorem. Inconsistency of (PS) and (SI1): Let μ1 be the sentence q ∨ z. According to the definition of ≤ψ , there is only one minimal μ1 -world, namely pqz, and therefore by (S*), ψ ∗μ1 ≡ p∧ q ∧z. Consider now the possible worlds w = pqz and w = pqz. Clearly, Diff(ψ ∗ μ1 , w ) = {q} ⊂ {q, z} = Diff(ψ ∗ μ1 , w). Consequently, no matter what the new preorder ≤ψ∗μ1 is, as long as it satisfies (PS) it holds that w <ψ∗μ1 w. On the other hand, since w, w ∈ [μ1 ] and w ≤ψ w , (SI1) entails that w ≤ψ∗μ1 w . Contradiction. Inconsistency of (PS) and (SI2): Let μ2 be the sentence p∧q∧z. Once again there is only one minimal μ2 -world, namely pqz, and therefore ψ ∗ μ2 ≡ p ∧ q ∧ z. Let w and w be the possible worlds pqz and pqz respectively. It is not hard to verify that Diff(ψ ∗ μ2 , w ) ⊂ Diff(ψ ∗ μ2 , w) and therefore (PS) entails w <ψ∗μ1 w. On the other hand, given that w, w ∈ [¬μ2 ] and w ≤ψ w , (SI2) entails that w ≤ψ∗μ1 w , leading us to a contradiction. Inconsistency of (PS) and (SI3): Let μ3 be the sentence (p ∧ q ∧ z) ∨ (p ∧ q ∧ z). Given the above definition of ≤ψ it is not hard to verify that ψ ∗ μ3 ≡ p ∧ q ∧ z. Once again, define w and w to be the possible worlds pqz and pqz respectively. Clearly Diff(ψ∗μ3 , w ) ⊂ Diff(ψ∗μ3 , w) and therefore (PS) entails w <ψ∗μ3 w. On the other hand notice that w ∈ [μ3 ], w ∈ [¬μ3 ] and w <ψ w . Hence (SI3) entails that w <ψ∗μ3 w . Contradiction. Inconsistency of (PS) and (SI4): Let μ4 = μ3 = (p ∧ q ∧ z) ∨ (p ∧ q ∧ z) and assume that w, w are as previously defined. Then clearly, like before, (PS) entails w <ψ∗μ4 w. On the other hand, since w ∈ [μ4 ], w ∈ [¬μ4 ] and w ≤ψ w , (SI4) entails that w ≤ψ∗μ4 w . A contradiction once again. 2 The above inconsistencies extend to subsequent developments of the Darwiche and Pearl approach as well. Herein we consider two such extensions in the form of two new postulates introduced to rectify anomalies with the original DP approach. The first postulate, called the Conjunction Postulate, was introduced by Nayak, Pagnucco, and Peppas in [7]: (CNJ)

If μ ∧ ϕ % ⊥, then ψ ∗ μ ∗ ϕ ≡ ψ ∗ (μ ∧ ϕ).

As shown in [7], in the presence of the KM postulates, (CNJ) entails (DP1), (DP3), and (DP4). Hence the following result is a direct consequence of Theorem 1: Corollary 1 In the presence of the KM postulates, postulate (P) is inconsistent with postulate (CNJ). The last postulate we shall consider herein is the Independence Postulate proposed by Jin and Thielscher in [5]: (Ind)

If ¬μ ∈ ψ ∗ ϕ then μ ∈ ψ ∗ μ ∗ ϕ.

Jin and Thielscher proved that, although weaker than (CNJ), (Ind) still entails (DP3) and (DP4). Hence, from Theorem 1, it follows: Corollary 2 In the presence of the KM postulates, postulate (P) is inconsistent with postulate (Ind).

7

CONCLUSION

In this paper we have proved the inconsistency of Parikh’s postulate (P) for relevance-sensitive belief revision, with each of the four postulates (DP1)-(DP4) proposed by Darwiche and Pearl for iterated belief revision. This result suggests that a major refurbishment may be due in our formal models for belief revision. Both relevance and iteration are central to the process of belief revision and neither of them can be sacrificed. Moreover, the formalizations of these notions by postulates (P) and (DP1)-(DP2) respectively seem quite intuitive and it is not clear what amendments should be made to reconcile them. On a more positive note, the inconsistencies proved herein reveal a hitherto unknown connection between relevance and iteration, which will eventually lead to a deeper understanding of the intricacies of belief revision.

REFERENCES [1] C. Alchourron and P. Gardenfors and D. Makinson, “On the Logic of Theory Change: Partial Meet Functions for Contraction and Revision”, Journal of Symbolic Logic, vol 50, pp 510-530, 1985. [2] A. Darwiche and J. Pearl, “On the Logic of Iterated Belief Revision”, Artificial Intelligence, vol. 89, pp 1-29, 1997. [3] P. Gardenfors, Knowledge in Flux. MIT Press, 1988. [4] A. Grove, “Two modellings for theory change”, Journal of Philosophical Logic vol. 17, pp. 157-170, 1988. [5] Y. Jin and M. Thielscher, “Iterated Revision, Revised”, Proceedings of the 19th International Joint Conference in Artificial Intelligence, pp 478-483, Edinburgh, 2005. [6] H. Katsuno and A. Mendelzon, “Propositional Knowledge Base Revision and Minimal Change”, in Artificial Intelligence, vol. 52, pp 263-294, 1991. [7] A. Nayak, M. Pagnucco, and P. Peppas, “Dynamic Belief Revision Operators”, Artificial Intelligence, vol. 146, pp 193-228, 2003. [8] R. Parikh, “Beliefs, Belief Revision, and Splitting Languages”, in J. Lawrence Moss, M. de Rijke, (eds)., Logic, Language, and Computation, vol 2, pp 266–268, CSLI Lecture Notes No. 96,. CSLI Publications, 1999. [9] P. Peppas, N. Foo, and A. Nayak, “Measuring Similarity in Belief Revision”, Journal of Logic and Computation, vol. 10(4), 2000. [10] P. Peppas, S. Chopra, and N. Foo, “Distance Semantics for RelevanceSensitive Belief Revision”, Proceedings of the 9th International Conference on the Principles of Knowledge Representation and Reasoning (KR2004), Whistler, Canada, June 2004. [11] P. Peppas, “Belief Revision”, in F. van Harmelen, V. Lifschitz, and B. Porter (eds), Handbook in Knowledge Representation, Elsevier Publications, 2007. [12] M. Winslett, “Reasoning about Action using a Possible Models Approach”, in Proceedings of the 7th National (USA) Conference on Artificial Intelligence (AAAI’88), pp. 89-93, 1988.

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-89

89

Conservativity in Structured Ontologies1 Oliver Kutz2 and Till Mossakowski3 Abstract. Using category theoretic notions, in particular diagrams and their colimits, we provide a common semantic backbone for various notions of modularity in structured ontologies, and outline a general approach for representing (heterogeneous) combinations of ontologies through interfaces of various kinds, based on the theory of institutions. This covers theory interpretations, (definitional) language extensions, symbol identifications, and conservative extensions. In particular, we study the problem of inheriting conservativity between sub-theories in a diagram to its colimit ontology, and apply this to the problem of localisation of reasoning in ‘modular ontology languages’ such as DDLs or E-connections.

1

Introduction

In this paper, we propose to use the category theoretic notions of diagram and colimit in order to provide a common semantic backbone for various notions of modularity in ontologies.4 At least three commonly used notions of ‘module’ in ontologies can be distinguished, depending on the kind of relationship between the ‘module’ and its supertheory (or superontology): (1) a module can be considered a ‘logically independent’ part within its superontology—this leads to the definition of module as a part of a larger ontology which is a conservative extensions of it; (2) a module can be a part of a larger ‘integrated ontology’, where the kind of integration determines the relation between the modules—this is the approach followed by modular ontology languages (e.g. DDLs, E-connections etc.); (3) a ‘part’ of a larger theory can be considered a module for reasons of elegance, re-use, tradition, etc.—in this case, the relation between a module and its supertheory might be a language extension, theory extension/interpretation, etc. The main contributions of the present paper are the following: (i) building on the theory of institutions, diagrams, and colimits, we show how these different notions of module can be considered simultaneously using the notion of a module diagram; (ii) we show how conservativity properties can be traced and inherited to the colimit of a diagram; (iii) we show how this applies to the composition problem in modular ontology languages such as DDLs and E-connections. Section 2 introduces institutions, Section 3 the diagrammatic view of modules, and Section 4 studies the problem of conservativity in diagrams. Finally, we sketch heterogeneous diagrams and apply this to modular ontology languages in Section 5.5 1

Work on this paper has been supported by the Vigoni program of the DAAD and by the DFG-funded collaborative research centre ‘Spatial Cognition’. 2 SFB/TR 8 Spatial Cognition, University of Bremen, Germany. [email protected] 3 DFKI GmbH, Bremen, Germany, [email protected] 4 This paper extends the results of [18]. 5 Proofs for the results of this paper can be found in [19].

2

Institutions

The study of modularity principles can be carried out to a quite large extent independently of the details of the underlying logical system that is used. The notion of institutions was introduced by Goguen and Burstall in the late 1970s exactly for this purpose (see [14]). They capture in a very abstract and flexible way the notion of a logical system by describing how, in any logical system, signatures, models, sentences (axioms) and satisfaction (of sentences in models) are related. The importance of the notion of institutions lies in the fact that a surprisingly large body of logical notions and results can be developed in a way that is completely independent of the specific nature of the underlying institution.6 An institution I = (Sign, Sen, Mod, |=) consists of (i) a category Sign of signatures; (ii) a functor Sen : Sign −→ Set giving, for each signature Σ, the set of sentences Sen(Σ), and for each signature morphism σ : Σ −→ Σ , the sentence translation map Sen(σ) : Sen(Σ) −→ Sen(Σ ), where Sen(σ)(ϕ) is abbreviated σ(ϕ); (iii) a functor Mod : Signop −→ CAT giving, for each signature Σ, the category of models Mod(Σ), and for each signature morphism σ : Σ −→ Σ , the reduct functor Mod(σ) : Mod(Σ ) −→ Mod(Σ), where Mod(σ)(M  ) is abbreviated M  |σ ; (iv) a satisfaction relation |=Σ ⊆ |Mod(Σ)| × Sen(Σ) for each Σ ∈ |Sign|, such that for each σ : Σ −→ Σ in Sign the following satisfaction condition holds: ()

M  |=Σ σ(ϕ) iff M  |σ |=Σ ϕ

for each M  ∈ |Mod(Σ )| and ϕ ∈ Sen(Σ), expressing that truth is invariant under change of notation and enlargement of context. The only condition governing the behaviour of institutions is thus the satisfaction condition ().7 A theory in an institution is a pair T = (Σ, Γ) consisting of a signature Sig(T ) = Σ and a set of Σ-sentences Ax(T ) = Γ, the axioms of the theory. The models of a theory T are those Sig(T )-models that satisfy all axioms in Ax(T ). Logical consequence is defined as usual: T |= ϕ if all T -models satisfy ϕ. Theory morphisms, also called interpretations of theories, are signature morphisms that map axioms to logical consequences. Examples of institutions include first- and higher-order classical logic, description logics, and various (quantified) modal logics [19].

3

Modules as Diagrams

Several approaches to modularity in ontologies have been discussed in recent years, including the introduction of various so-called ‘modular ontology languages’. The module system of the Web Ontology 6 7

For an extensive treatment of the model theory in this setting, see [10]. Note, however, that non-monotonic formalisms can only indirectly be covered this way, but compare, e.g., [16].

90

O. Kutz and T. Mossakowski / Conservativity in Structured Ontologies

Language OWL itself is as simple as inadequate [9]: it allows for importing other ontologies, including cyclic imports. The language C ASL, originally designed as a first-order algebraic specification language, is used for ontologies in [21]. Beyond imports, it allows for renaming, hiding and parameterisation. Other languages envisaging more involved integration and modularisation mechanisms than plain imports include DDLs [6], E-connections [17], and P-DLs [4]. We will use the formalism of colimits of diagrams as a common semantic backbone for these languages.8 The intuition behind colimits is explained as follows:

structure are just tree-shaped diagrams, while both shared parts and cyclic imports lead to arbitrary graph-shaped diagrams. The translation of C ASL (without hiding) to so-called development graphs detailed in [7] naturally leads to diagrams as well. Finally, the diagrams corresponding to modular languages like DDLs and E-connections will be studied in Sect. 5. Thus, diagrams can be seen as a uniform mathematical formalism where properties of all of these module concepts can be studied. An important such property is conservativity.

“Given a species of structure, say widgets, then the result of interconnecting a system of widgets to form a super-widget corresponds to taking the colimit of the diagram of widgets in which the morphisms show how they are interconnected.” [13]

Conservative diagrams are important because they imply that the combined ontology does not add new facts to the individual ontologies. Indeed, the notion of an ontology module of an ontology T has been defined as any “subontology T  such that T is a conservative extension of T  ” [12]—this perfectly matches our notion of conservative diagram below.

The notion of diagram is formalised in category theory. Diagrams map an index category (via a functor) to a given category of interest. They can be thought of as graphs in the category. A cocone over a diagram is a kind of “tent”: it consists of a tip, together with a morphism from each object involved in the diagram into the tip, such that the triangles arising from the morphisms in the diagram commute. A colimit is a universal, or minimal cocone. For details, see [1]. In the sequel, we will assume that the signature category has all finite colimits, which is a rather mild assumption; in particular, it is true for all the examples of institutions mentioned above. Moreover, we will rely on the fact that colimits of theories exist in this case as well; the colimit theory is defined as the union of all component theories in the diagram, translated along the signature morphisms of the colimiting cocone. Definition 1 A module diagram of ontologies is a diagram of theories such that the nodes are subdivided into ontology nodes and interface nodes.

4

Conservative Diagrams and Composition

Definition 2 A theory morphism σ : T1 −→ T2 is prooftheoretically conservative, if T2 does not entail anything new w.r.t. T1 , formally, T2 |= σ(ϕ) implies T1 |= ϕ. Moreover, σ : T1 −→ T2 is model-theoretically conservative, if any T1 -model M1 has a σ-expansion to T2 , i.e. a T2 -model M2 with M2 |σ = M1 . It is easy to show that conservative theory morphisms compose. Moreover, model-theoretic implies proof-theoretic conservativity. However, the converse is not true in general, compare [22] for an example. Definition 3 A (proof-theoretic, model-theoretic) conservative module diagram of ontologies is a diagram of theories such that the theory morphism of any ontology node into the colimit of the diagram is (proof-theoretically resp. model-theoretically) conservative. By conservativity, the definition immediately yields:

Composition of module diagrams is simply their union. Proposition 1 The colimit ontology of a proof-theoretic (modeltheoretic) conservative module diagram is consistent (satisfiable)9 if any of the component ontologies is.

Example 1 Consider the union of the diagrams T1  Σ1

-

T2

T2  Σ2

-

T3

where the Σi are interfaces and the Ti are ontologies. Think of e.g. T12 as being an ontology that imports T1 and T2 , where Σ1 contains all the symbols shared between T1 and T2 . Then T12 (and T23 ) can be obtained as pushouts, and so can the overall union T123 (which typically is then further extended with new concepts etc.). A “c” means “conservative”; this will be explained in Sect. 4. c c

-

T12 

T1 

c

c c

Σ1

T123 

-

T23 

T2 

c

-

T3

Σ2

Notice that Example 1 is closely related to the composition (or combination) of ontology alignments, as introduced in [26], and further studied in [20]. In general, it is clear that theories with an import 8

However, note that hiding is not covered by this approach.

Thus, in particular, in a conservative module diagram, an ontology node Oi can only be consistent (satisfiable) if all other ontology nodes Oj , j = i, are consistent (satisfiable) as well. The main question is how to ensure these conservativity properties in the united diagram. To study this, we introduce some notions from model theory, namely various notions of interpolation (for proof-theoretic conservativity) and amalgamation (for modeltheoretic conservativity). Craig interpolation plays a crucial role in connection with proof systems in structured theories. The most common formulation, i.e. Craig (or Arrow) interpolation, however, relies on a connective → being present in the institution. A slightly more general formulation, often called turnstile interpolation is as follows: if ϕ |= ψ, then there exists some χ that only uses symbols occurring in both ϕ and ψ, with ϕ |= χ and χ |= ψ. This, of course, follows from Craig interpolation in the presence of a deduction theorem. For the general study of module systems, we need to generalise such definitions in at least two important ways. The first concerns the 9

Contrary to the terminology used in DL, we distinguish here proof-theoretic (syntactic) consistency of a theory T (which means T |= ϕ for some sentence ϕ) from model-theoretic (semantic) satisfiability (which means M |= T for some model M ).

91

O. Kutz and T. Mossakowski / Conservativity in Structured Ontologies

rather implicit use of signatures in the standard definitions. Making signatures explicit means to assume that ϕ lives in a signature Σ1 , ψ lives in a signature Σ2 , the entailment ϕ |= ψ lives in Σ1 ∪ Σ2 , and the interpolant in Σ1 ∩ Σ2 . Since we do not want to go into the technicalities for equipping an institution with unions and intersections (see [11] for details), we replace Σ1 ∩ Σ2 with a signature Σ, and Σ1 ∪ Σ2 with Σ such that Σ is obtained as a pushout from the other signatures via suitable signature morphisms (cf. the diagram below). Secondly, we move from single sentences to sets of sentences. This is useful since we want to support DLs and TBox reasoning, and DLs like (sub-Boolean) EL do not allow to rewrite ‘conjunctions of subsumptions’, i.e., we cannot collapse a TBox into a single sentence. (In case of compact logics, the use of sets is equivalent to the use of finite sets.) This leads to the following definition. In the sequel, fix an arbitrary institution I = (Sign, Sen, Mod, |=): Definition 4 The institution I has the Craig-Robinson interpolation property (CRI for short), if for any pushout θ1 -

Σ1  σ1



Σ 

θ2 σ2 -

Σ2

Σ any set Γ1 of Σ1 -sentences and any sets Γ2 , Δ2 of Σ2 -sentences with θ1 (Γ1 ) ∪ θ2 (Δ2 ) |= θ2 (Γ2 ), there exists a set of Σ-sentences Γ (called the interpolant) such that

Institution EL ALC ms ALC ALCO ALCQO SHOIN FOLms QS5 Figure 1.

Proposition 2 A compact institution with implication has CRI iff it has Craig interpolation. Here, an institution I has implication if for any two Σ-sentences ϕ, ψ, there exists a Σ-sentence χ such that, for any Σ-model M ,

Definition 5 An institution I is (weakly) exact if, for any diagram of signatures, any compatible family of models (i.e. compatible with the reducts induced by the involved signature morphisms) can can be amalgamated to a unique (or weakly amalgamated to a not necessarily unique) model of the colimit. For pushouts, this amounts to the following (we use notation as in Def. 4): any pair (M1 , M2 ) ∈ Mod(Σ1 )×Mod(Σ2 ) that is compatible (in the sense that M1 and M2 reduce to the same Σ-model) can be amalgamated to a (unique) Σ -model M (i.e., there exists a (unique) M ∈ Mod(Σ ) that reduces to M1 and M2 , respectively).

CRI + + + + -

Weak exactness for these institutions follows with standard methods, see [10]. The same holds for exactness for the many-sorted variants. Exactness, however, obviously fails for the single-sorted logics as well as for QS5, because in these logics, the implicit universe resp. the implicit set of worlds leads to the phenomenon that the empty signature has many different models. The following propositions are folklore in institutional model theory, see [10]. Theorem 1 1. In an institution with CRI proof-theoretic conservativity is preserved along pushouts. 2. In an institution that is weakly exact, model-theoretic conservativity is preserved along pushouts. We now give necessary conditions for the preservation of conservativity when taking the colimit of the union of conservative diagrams. Firstly, a diagram is thin, or a preorder, if its index category is thin (i.e., there is at most one arrow between two given objects). Consider the following non-thin union diagram (assuming that the two arrows in the union are inherited from two different ontologies), where {P  } ⊆ T1 and {C1 ≡ ¬C2 } ⊆ T2 : T1

P → C1

- T3 ⊇ C ≡ ¬C - T2 ...........

P → C2

Although the individual ontologies are conservative, the union is not because in the colimit C1 and C2 are identified. Next, a preorder is finitely bounded inf-complete if any two elements with a common lower bound have an infimum. Consider the following, not finitely bounded inf-complete union diagram (assume that it is obtained as the union of its upper and its lower half): P  Q...... .......... .......... .......... c .... c Q P ≡Q .. c .......... c .. .. .. ...... ................ QP

-

M |= χ iff (M |= ϕ implies M |= ψ) Moreover, I is compact if T |= ϕ implies T  |= ϕ for a finite subtheory T  of T . Since for modal logics, the deduction theorem (for the global consequence relation |=) generally fails, these logics do not have implication in the above sense, and we cannot apply Prop. 2. However, various more specialised criteria can be given, see [19]. Some results are summarised in Fig. 1. The amalgamation property (called ‘exactness’ in [11]) is a major technical assumption in the study of specification semantics, see [23].

exact + + -

(Weak) exactness and Craig-Robinson interpolation

Γ1 |= σ1 (Γ) and Δ2 ∪ σ2 (Γ) |= Γ2 . CRI, in general, is strictly stronger than Craig interpolation. However, for almost all logics typically used in knowledge representation, they are indeed equivalent. We give a criterion that applies to institutions generally, taken from [10]:

weakly exact + + + + + + + +

P 

Again, the individual ontologies are conservative, but the colimit of the union is not. Hence, call a diagram tame if it does not show these sources of inconsistency/non-conservativity, i.e. if it is thin and finitely bounded inf-complete. Theorem 2 1. Assume institution I has an initial signature10 and has CRI (is weakly exact). If the involved ontologies are consistent (satisfiable), then composition of module diagrams via union preserves proof-theoretic (model-theoretic) conservativity if the diagram resulting from the union of the individual diagrams and their colimits is tame. 2. If the union is a disjoint union, the tameness assumption can be dropped. 10

Usually, the empty signature is initial.

92

O. Kutz and T. Mossakowski / Conservativity in Structured Ontologies

Note that consistency of the involved ontologies can be replaced with connectedness of the united diagram. The above examples and Example 2 below show that the conditions from the theorem are essentially optimal. See Example 1 for a conservative union of conservative diagrams.

5

ms C E (T1ms , T 2 )

M  |=JΦ(Σ) αΣ (ϕ) ⇔ βΣ (M  ) |=IΣ ϕ. Here, Φ(Σ) is the translation of signature Σ from institution I to institution J, αΣ (ϕ) is the translation of the Σ-sentence ϕ to a Φ(Σ)sentence, and βΣ (M  ) is the translation (or perhaps: reduction) of the Φ(Σ)-model M  to a Σ-model. The definitions and results of the previous sections also apply to the heterogeneous case. However, special care is needed in obtaining CRI or (weak) exactness [10]. Heterogeneous knowledge representation was also a major motivation for the definition of modular languages, E-connections in particular [17]. We here show how the integration of ontologies via ‘modular languages’ can be re-formulated in module diagrams. In the following, we will assume basic acquaintance with the syntax and semantics of both, DDLs and E-connections, which we reformulate as many-sorted theories. Details have to remain sketchy for lack of space. It should be clear that DDLs or E-connections can essentially be considered as many-sorted heterogeneous theories: component ontologies can be formulated in different logics, but have to be built from many-sorted vocabulary, and link relations are interpreted as relations connecting the sorts of the component logics (compare [3] who note that this is an instance of a more general co-comma construction). To be more precise, assume a DDL D = (S1 , S2 ) is given. Knowledge bases for D can contain bridge rules of the form: 

Ci −→ Cj

(into rule)



Ci −→ Cj

(onto rule)

where Ci and Cj are concepts from Si and Sj (i = j), respectively (we consider here only DDL in its most basic form without individual correspondences etc.). An interpretation I for a DDL knowledge base is a pair ({Ii }i≤n , R), where each Ii is a model for the corresponding Si , and R is a function associating with every pair (i, j), i = j, a binary relation rij ⊆ Wi × Wj between the domains Wi and Wj of Ii and Ij , respectively. In the many-sorted re-formulation of DDLs, the relation rij is now interpreted as a relation between the -sort of S1 and the -sort of S2 . Bridge rules are expressed as existential restrictions of the form () ∃rij .Ci  Cj

and

ms T1ms ( T2

T1

T1ms 

c

c

c

-

T2ms 

∅

Figure 2.

c

T2

E-connections and DDLs many-sorted

a discussion of related issues). In fact, the main difference between DDLs and various E-connections now lies in the expressivity of this ‘link language’ L connecting the different sorts of the ontologies. In basic DDL as defined above, the only expressions allowed are those given in (), so the link language of basic DDL is a certain, very weak sub-Boolean fragment of many sorted ALC, namely the one given through (). In E-connections, expressions of the form ∃rij .Ci are again concepts of Sj , to which Booleans (or other operators) of Sj as well as restrictions using relations rji can be applied. Thus, the basic link language of E-connections is sorted ALCI ms (relative to the now richer languages of Si ).11 Such many-sorted theories can easily be represented in a diagram as shown in Figure 2. Here, we first (conservatively) obtain a disjoint union T1ms ( T2ms as a pushout, where the component ontologies have been turned into sorted variants (using an institution comorphism from the single-sorted to the many-sorted logic), and the empty interface guarantees that no symbols are shared at this point. An E-connection KB in language C E (T1ms , T2ms ) or a DDL KB in language DDL(T1ms , T2ms ) is then obtained as a (typically not conservative) theory extension. When connecting ontologies via bridges, or interfaces, this typically is not conservative everywhere, but only for some of the involved ontologies. We give a criterion for a single ontology to be conservative in the combination. While the theorem can be applied to arbitrary interface nodes, when applied to E-connections or DDLs, we assume that bridge nodes contain DDL bridge rules or E-connection assertions. Theorem 3 Assume that we work in an institution that has CRI (is weakly exact). Let ontologies T1 , . . . , Tn be connected via bridges Bij , i < j. If Ti is proof-theoretically (model-theoretically) conservative in Bij for j > i, then T1 is proof-theoretically (modeltheoretically) conservative in the resulting colimit ontology T . The diagram in Fig. 3 illustrates Theorem 3 for the case n = 3. As concerns the applicability of the theorem, we have given an overview of logics being (weakly) exact or having CRI in Fig. 1. Of course, the conservativity assumptions have to be shown additionally. We next give an example of the failure of the claim of the theorem in case we work in a logic that lacks Craig-Robinson interpolation. Example 2 The presence of nominals in description or modal logics generally destroys (standardly formulated) Craig interpolation [2]. Here is a counterexample for the logic ALCO. Let

∃rij .Ci ' Cj

The fact that bridge rules are atomic statements in a DDL knowledge base now translates to a restriction on the grammar governing the usage of the link relation rij in the multi-sorted formalism (see [5] for

-

-

Heterogeneity and Modular Languages

As [24] argue convincingly, relating or integrating ontologies may happen across different institutions as ontologies are written in many different formalisms, like relation schemata, description logics, firstorder logic, and modal logics. Heterogeneous specification is based on some graph of logics and logic translations, formalised as institutions and so-called institution comorphisms, see [15]. The latter are again governed by the satisfaction condition, this time expressing that truth is invariant also under change of notation across different logical formalisms:

DDL(T1ms , T2ms )

11

Γ

:=

{  ∃S.C  ∃S.¬C} and

Δ

:=

{∀S.(D  i)  ∃S.D}

But can be weakened to ALC ms or the link language of DDLs, or strengthened to more expressive many-sorted DLs such as ALCQI ms .

O. Kutz and T. Mossakowski / Conservativity in Structured Ontologies

c

T1 c

-

- B13 

T3

c B12 

- ? - T   6

-

B23



c T2

Figure 3. Colimit integration along bridges for n = 3

where i is a nominal. Clearly, Γ |= Δ, for in every model M |= Γ, every point has at least two S-successors. But i can only be true in at most one of those successors, which entails M |= Δ. Now, (using bisimulations) it can be shown that in ALCO there is no Δ built from shared concept names alone (there are none) such that Γ |= Δ and Δ |= Δ. Assume now ontologies T1 , T2 , T3 are formulated in the DL ALCO with signatures Sig(T1 ) ⊆ {S, B, D, i}, Sig(T2 ) ⊆ {C1 , C2 }, and Sig(T3 ) ⊆ {B1 , B2 }. Also, assume {∃S.D} ⊆ T1 . Consider now the situation depicted in Fig. 3 with B12

⊇

{  ∃S.∃R1 .C1 ,   ∃S.∃R1 .¬C2 },

B13

⊇

{B1 ≡ ∃R3−1 .B, B2 ≡ ∃R3−1 .B},

B23

⊇

{C1 ≡ ∃R2 .B1 , C2 ≡ ∃R2 .B2 }.

Here, the roles R1 , R2 , R3 can be seen as link relations, and since we apply existential restrictions ∃S to ∃R2 .C1 etc., the example can be understood as a composition of (binary) E-connections. The reader can check that Ti is conservative in Bij for j > i. However, in the colimit (union) of this diagram, ∀S.D  i  ∃S.D follows, while this does not follow in T1 , and thus T1 is not conservative in the colimit ontology. Thus, if the assumptions of the theorem are satisfied, reasoning over the signature of T1 can be performed within T1 , i.e. without considering the overall integration T . This, however, can not be guaranteed for logics lacking CRI. In the light of this example, it should now come as no surprise that attempts to localise reasoning in DDLs in a peer-to-peer like fashion whilst remaining sound and complete have been restricted to logics lacking nominals [25].

6

Discussion and Outlook

Diagrams and their colimits offer the right level of abstraction to study conservativity issues in different languages for modular ontologies. We have singled out conditions that allow for lifting conservativity properties from individual diagrams to their combinations. An interesting point is the question whether proof-theoretic or model-theoretic conservativity should be used. The model-theoretic notion ensures ‘modularity’ in more logics than the proof-theoretic one since the lifting theorem for the former only depends on mild amalgamation properties. By contrast, for the latter one needs CraigRobinson interpolation which fails, e.g., for some description logics with nominals, and also for QS5—but these logics are used in practice for ontology design. Moreover, when relating ontologies across different institutions, the model-theoretic notion is more feasible. Finally, it has the ad-

93

vantage of being independent of the particular language, which implies avoidance of examples like the one presented in [22], where a given ontology extension is proof-theoretically conservative in EL but not in ALC. Of course, model-theoretic conservativity generally is harder to decide, but it can be ensured by syntactic criteria, and the work related to this is promising [8].

REFERENCES [1] J. Ad´amek, H. Herrlich, and G. Strecker, Abstract and Concrete Categories, Wiley, New York, 1990. [2] C. Areces and M. Marx, ‘Failure of interpolation in combined modal logics’, Notre Dame Journal of Formal Logic, 39(2), 253–273, (1998). [3] F. Baader and S. Ghilardi, ‘Connecting Many-Sorted Theories’, The Journal of Symbolic Logic, 72(2), 535–583, (2007). [4] J. Bao, D. Caragea, and V. Honavar, ‘On the Semantics of Linking and Importing in Modular Ontologies’, in Proc. of ISWC. Springer, (2006). [5] A. Borgida, ‘On Importing Knowledge from DL Ontologies: some Intuitions and Problems’, in Proc. of DL, (2007). [6] A. Borgida and L. Serafini, ‘Distributed Description Logics: Assimilating Information from Peer Sources’, Journal of Data Semantics, 1, 153–184, (2003). [7] CoFI (The Common Framework Initiative), C ASL Reference Manual, LNCS Vol. 2960 (IFIP Series), Springer, 2004. Freely available at http://www.cofi.info. [8] B. Cuenca Grau, I. Horrocks, Y. Kazakov, and U. Sattler, ‘Modular Reuse of Ontologies: Theory and Practice’, Journal of Artificial Intelligence Research (JAIR), 31, (2008). to appear. [9] B. Cuenca Grau, B. Parsia, and E. Sirin, ‘Combining OWL Ontologies Using E-Connections’, Journal of Web Semantics, 4(1), 40–59, (2006). [10] R. Diaconescu, Institution-independent Model Theory, Studies in Universal Logic, Birkh¨auser, 2008. [11] R. Diaconescu, J. Goguen, and P. Stefaneas, ‘Logical Support for Modularisation’, in 2nd Workshop on Logical Environments, 83–130, CUP, New York, (1993). [12] S. Ghilardi, C. Lutz, and F. Wolter, ‘Did I Damage My Ontology? A Case for Conservative Extensions in Description Logics’, in Proc. of KR-06, pp. 187–197, (2006). [13] J. A. Goguen, ‘A categorical manifesto’, Mathematical Structures in Computer Science, 1, 49–67, (1991). [14] J. A. Goguen and R. M. Burstall, ‘Institutions: Abstract Model Theory for Specification and Programming’, J. of the ACM, 39, 95–146, (1992). [15] J. A. Goguen and G. Ros¸u, ‘Institution morphisms’, Formal aspects of computing, 13, 274–307, (2002). [16] S. Guerra, ‘Composition of Default Specifications’, Journal of Logic and Computation, 11(4), 559–578, (2001). [17] O. Kutz, C. Lutz, F. Wolter, and M. Zakharyaschev, ‘E-Connections of Abstract Description Systems’, Artificial Intelligence, 156(1), 1–73, (2004). [18] O. Kutz and T. Mossakowski, ‘Modules in Transition: Conservativity, Composition, and Colimits’, in 2nd Int. Workshop on Modular Ontologies (WoMO-07), (2007). [19] O. Kutz and T. Mossakowski, ‘Conservativity in Structured Ontologies’, Technical report, University of Bremen, www.informatik. uni-bremen.de/˜okutz/OntoStruc-TR.pdf, (2008). [20] O. Kutz, T. Mossakowski, and M. Codescu, ‘Shapes of Alignments: Construction, Combination, and Computation’, in Int. Workshop on Ontologies: Reasoning and Modularity (WORM-08), (2008). [21] K. L¨uttich, C. Masolo, and S. Borgo, ‘Development of Modular Ontologies in CASL’, in 1st Workshop on Modular Ontologies 2006, eds., P. Haase, V. Honavar, O. Kutz, Y. Sure, and A. Tamilin, volume 232 of CEUR Workshop Proceedings. CEUR-WS.org, (2006). [22] C. Lutz and F. Wolter, ‘Conservative Extensions in the Lightweight Description Logic EL’, in Proc. of CADE-07, pp. 84–99. Springer, (2007). [23] D. Sannella and A. Tarlecki, ‘Specifications in an arbitrary institution’, Information and Computation, 76, 165–210, (1988). [24] M. Schorlemmer and Y. Kalfoglou, ‘Institutionalising Ontology-Based Semantic Integration’, Journal of Applied Ontology., (2008). To appear. [25] A. Tamilin, Distributed Ontological Reasoning: Theory, Algorithms, and Applications, Ph.D. dissertation, University of Trento, 2007. [26] A. Zimmermann, M. Kr¨otzsch, J. Euzenat, and P. Hitzler, ‘Formalizing Ontology Alignment and its Operations with Category Theory’, in Proc. of FOIS, pp. 277–288, (2006).

94

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-94

1

1

2

Card Σ M ax GM ax

•

•

•

Σ

1

L

P 2

→ ↔ ϕ

M ax

Cn

¬ ∧ ∨ L

95

J. Hué et al. / Removed Sets Fusion: Performing off the Shelf

Ψ = {ϕ1 , . . . , ϕn } Ψ ϕ1 , . . . , ϕn 

n

n

• V

W

ϕ

Ψ = {ϕ}

Δ

•

Δ(Ψ) Δ(Ψ) = W ϕ ∈Ψ

V ϕ ∈Ψ

•

ϕi

1

n

short(X) ← size(Y ), X < Y X Y X
•

Δ(Ψ) Δ(Ψ)

Δ(Ψ) =

#domain possible(X) possible(1..n) X

h ← k {a1 , . . . , an } l l

k

{a1 , . . . , an }

minimize{.} maximize{.} minimize{a1 , · · · , an }

ϕi

{a1 , · · · , an }

Σ Ψ

Card

Σ

M ax

GM ax

M ax Ψ

P c ← c, ai (1 ≤ i ≤ n), bj (1 ≤ a1 , . . . , an , not b1 , . . . , not bm not j ≤ m) r head(r) = body(r) = {a1 , · · · , an , b1 , · · · , bm } c body + (r) = {a1 , · · · , an } body − (r) = {b1 , · · · , bm } body(r) = body + (r) ∪ body − (r) + r head(r) ← body + (r) r r X Π r ∈ Π head(r) ∈ X body(r) ⊆ X Π ΠX

CN(Π) Π X = {r + | r ∈ Π and body − (r) ∩ X = ∅} X Π

ϕn

Ψ = {ϕ1 , . . . , ϕn }

ϕ1  . . .  ϕ1  . . . 

ϕn ϕ1  . . .  ϕn

P

ΠX


CN(ΠX ) = X Ψ

ϕ1 . . .ϕn Ψ

P

Ψ = {ϕ1 , . . . , ϕn } X ⊆ ϕ1  . . .  ϕn Ψ (ϕ1  . . .  ϕn )\X

X ≤P Y P

≤P

P X

Y

Ψ = {ϕ1 , . . . , ϕn } X ⊆ ϕ1 . . .ϕn P X Y ⊆ ϕ1  . . .  ϕn Y

Ψ = {ϕ1 , . . . , ϕn } ΔP (Ψ) W ΔP (Ψ) = X∈F R(Ψ) {Cn((ϕ1  . . .  ϕn )\X)}

Ψ

96

J. Hué et al. / Removed Sets Fusion: Performing off the Shelf

Card, Σ, M ax, GM ax GM ax

ϕi

LΨ X

X ≤lex

pi (X) = |X ∩ ϕi | (pi (X))1≤i≤n

•

f ≡ f1 ∨ . . . ∨ fj ρf 1 , . . . , ρ f

•

f ≡ f 1 ∧. . .∧f j ..., rfi ← ρf

a ← not b ← not b 1 ra∨b ← not a, not b ρ1 ← not a ρ2 ← b 3 r¬a∧¬b ←b

X ≤P Y |X| ≤ |Y | Σ1≤i≤n | X ∩ ϕi |≤ Σ1≤i≤n |Y ∩ ϕi | max1≤i≤n | X ∩ ϕi |≤ max1≤i≤n | Y ∩ ϕi | Ψ LΨ X ≤lex LY

Ψ = {ϕ1 , ϕ2 , ϕ3 } ϕ1 = ϕ3 = {¬a ∧ ¬b, ¬a ∨ ¬b} {a ∨ b, b} ϕ2 = {a ↔ b, ¬b} FΣ R(Ψ) = {{a∨b, b}} ΔΣ (Ψ) = Cn(a ↔ b, ¬b, ¬a∧¬b, ¬a∨ ¬b) FM ax R(Ψ) = {{b, a ↔ b, ¬a ∧ ¬b}} ΔM ax (Ψ) = Cn(a ∨ b, ¬b, ¬a ∨ ¬b)

ΠΨ ΠΨ

P

Ψ

ΠΨ

ρ2 = ρ¬a∧¬b

a

← not a c ← not c 1 rb ← not b ρ1 ← not b 2 ← b r¬b 3 r¬a∨¬b ← a, b

ΠΨ ΠΨ S P

∀S  ⊆ ΠΨ S 

S S

S

IS

s(Ψ)

ϕi

ΠΨ R

+

=

{rfi |f

f ∈ ϕi

a←

Ψ V+

∈ ϕi } F0 (rfi )

rfi a

ρf

ΠΨ

ΠΨ

f≡a

• 3

f ≡ ¬f

a→b a

Ψ

X ⊆ (ϕ1  . . .  ϕn ) X P P F0 (S ∩ R+ ) = X Card Σ M ax

rfi minimize f

ΠΨ

← not a rfi ← not ρf 1

¬a∨b

M ax

S

V+

rfi 1

ΠΨ

ΠΨ 1 2 2 3 {a , b , ρ1 , ra∨b , rb1 } {a , b, ρ1 , ρ2 , r¬b , ra↔b , r¬a∧¬b }  1 2 3 2 3 3 {a, b , ρ1 , ρ2 , rb , ra↔b , r¬a∧¬b} {a, b, ρ2 , r¬b , r¬a∧¬b, r¬a∨¬b } M ax 2 3 {a, b , ρ1 , ρ2 , rb1 , ra↔b , r¬a∧¬b } 1 Σ {a , b , ρ1 , ra∨b , rb1 } Ψ M ax Σ

fj

f ∀f ∈ ϕi •

IS = {a | a ∈ S} ∪

Ψ = {ϕ1 , . . . , ϕn } S IS V+ (ϕ1  . . .  ϕn )\F0 (S ∩ R+ )

(ϕ1  . . .  ϕn ) ΠΨ

ΠΨ

S ΠΨ

ΠΨ

V−

ϕi ∀rfi ∈ R+ , F0 (rfi ) = f

a∈V+ a ← a



f, f 1 , . . . , f n

P

S

ΠΨ Card Σ Ψ = {ϕ1 , . . . , ϕn }

b ← not b c ← not c 2 ra↔b ← ρ1 , ρ2 ρ2 ← a 3 r¬a∧¬b ←a

P

S ∩ V+  {¬a | a ∈ S}

P

P

rfi ← ρf 1

ρ1 = ρa∧b a

P Card Σ M ax GM ax

rfi ←

a ↔ b (a∧b)∨(¬a∧¬b) f j ≡ a ρf

i i + ΠΣ Ψ = minimize{rf | rf ∈ R } ∪ ΠΨ

Σ Σ

ΠΨ

ΠΣ Ψ Σ

97

J. Hué et al. / Removed Sets Fusion: Performing off the Shelf

Card

+

ΠCard = minimize{rfi Ψ

R } ∪ Ps(Ψ)

Card

Σ

ΠΨ

Ψ

Card

: : : : :

S ( =

β1

δ4 β1 β2

: : :

S

Σ

#domain possible(U ). #domain base(V ). possible(1..m). base(1..n). size(U ) ← U {rfV |F0 (f ) ∈ ϕV }U.

9 > > > = > > > ;

δi

α rfV

U

Πbound Ψ

S

M ax

ϕV

U size(U )

)

#domain possible(W ). negmax(W ) ← size(U ), U > W. max(U ) ← size(U ), not negmax(U ).

U >W max(U ) U size(U ) ax M ax ΠM = Πsize ∪ Πbound ∪ ΠΨ ∪ Ψ Ψ Ψ minimize[max(1) = 1, . . . , max(m) = m] M ax

ΠΨ

M ax

= { γ1 : size(V, U ) ← U {rem(V, 1), . . . , rem(V, m)} U. }

m γ1

bound

ΠΨ

Σ

ax ΠM Ψ M ax

O(m × n) GM ax |X ∩ ϕi | size

GM ax

O(mn × nn )

W

size(U )

ΠΨ

ΠΨ

minimize{}

S

8 δ0 > > > < δ1 δ2 = > > > δ3 : α

m

Πbound Ψ

ax ΠGM Ψ GM ax

Card

S max1≤i≤n (|{rfi |rfi ∈ S}|) minimize{} M ax

size(U )

GM ax

rfi ∈

ΠCard Ψ

M ax

Πsize Ψ

|

8 α0 > > > > > > > > < α1 = > > > > αi > > > > : αn

ϕV ϕV : : : :

U

max(X1 , . . . , Xn ) ← size(Y1 , X1 ), . . . , size(Yn , Xn ), X1 >= X2 , . . . , Xn−1 >= Xn , neq(Y1 , . . . , Yn ). max1 (X1 ) ← max(X1 , . . . , Xn ) , X1 >= X2 , . . . , Xn−1 >= Xn . ... max3 (X3 ) ← max(X1 , . . . , Xn ), X1 >= X2 , . . . , Xn−1 >= Xn .

9 > > > > > > > > =

M ax

> > > > > > > > ;

X1

size() Xn minimize[] Xn +Xn−1 ×m+. . .+X1 ×mn−1 size bound ax GM ax ΠGM = ΠΨ ∪ ΠΨ ∪ Ψ ΠΨ ∪ minimize[maxn (1) = 1, maxn (2) = 2, . . . , maxi (1) = mn−i , maxi (2) = 2 × mn−i , . . . , max1 (n) = n × mn−1 ].

(nf )

(na)

98

J. Hué et al. / Removed Sets Fusion: Performing off the Shelf

Σ

M ax

(nf )/(na)

Card Σ M ax GM ax

minimize

12th

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-99

99

A Coherent Well-founded Model for Hybrid MKNF Knowledge Bases ´ Alferes1 and Pascal Hitzler 2 Matthias Knorr1 and Jos´e Julio Abstract. With the advent of the Semantic Web, the question becomes important how to best combine open-world based ontology languages, like OWL, with closed-world rules paradigms. One of the most mature proposals for this combination is known as Hybrid MKNF knowledge bases [11], which is based on an adaptation of the stable model semantics to knowledge bases consisting of ontology axioms and rules. In this paper, we propose a well-founded semantics for such knowledge bases which promises to provide better efficiency of reasoning, which is compatible both with the OWL-based semantics and the traditional well-founded semantics for logic programs, and which surpasses previous proposals for such a well-founded semantics by avoiding some issues related to inconsistency handling.

1

Introduction

The Web Ontology Language OWL3 is a recommended standard by the W3C for modeling Semantic Web knowledge bases. It is essentially based on Description Logics (DLs) [1], and thus adheres to the open-world assumption. It is apparent, however, and frequently being voiced by application developers, that it would be favorable to have closed-world modeling as an additional feature for ontology-based systems. This need has led to several investigations into combinations of closed-world rules paradigms with DLs, which can still be considered to be in their early stages, and the proposed solutions differ substantially. We base our work on the claim that the integration should be as tight as possible, in the sense that conclusions from the rules affect the conclusions from the ontology and vice-versa. Among such proposals are several whose semantics is based on stable model semantics (SMS) [5] (e.g. [2, 3, 6, 11, 13]), and only few which are based on the well-founded semantics (WFS) [14], like [4, 8]. Though these WFS-based approaches are in general weaker in their derivable consequences, their faster computation (data complexity P vs. NP) should be more suitable for the intended application area, the WWW. One of the currently most mature proposals for a tight integration is known as Hybrid MKNF knowledge bases [11], which draws on the logic of Minimal Knowledge and Negation as Failure (MKNF) [10]. [11]’s proposal evaluates knowledge bases under a stable model semantics, resulting in unfavorable computational complexities. In this paper, we therefore define a new semantics, restricted to non-disjunctive rules, which soundly approximates the semantics of [11] and is in a strictly lower complexity class. The semantics furthermore yields the original DL-semantics when no rules are present, and the original well-founded semantics if the DL-component is empty. 1 2 3

CENTRIA, Universidade Nova de Lisboa, Portugal AIFB, Universit¨at Karlsruhe, Germany http://www.w3.org/2004/OWL/

The semantics is furthermore coherent in the sense of [12], i.e. whenever any formula is first-order false then it is also non-monotonically false. It also allows for detecting inconsistencies between interacting ontologies and rules, and in fact does this without any substantial additional computational effort. Due to this inconsistency handling, our proposal is superior to that of [8], which also attempted to define a WF semantics, but resulted in some unintuitive behavior in the presence of inconsistencies. The paper is structured as follows. We first recall preliminaries on Hybrid MKNF knowledge bases in Section 2. We then introduce a running modeling example in Section 3 before introducing our wellfounded semantics in Section 4. Section 5 is devoted to some basic properties, especially regarding consistency. In Section 6 we briefly compare with most similar approaches, and conclude. More details, including proofs, can be found in [9].

2

Preliminaries

At first we present the syntax of MKNF formulas taken from [11]. A first-order atom P (t1 , . . . , tn ) is an MKNF formula where P is a predicate and the ti are function-free first-order terms. If ϕ is an MKNF formula then ¬ϕ, ∃x : ϕ, K ϕ and not ϕ are MKNF formulas and likewise ϕ1 ∧ ϕ2 and ϕ1 ← ϕ2 for MKNF formulas ϕ1 , ϕ2 . The symbols ∨, ≡, and ∀ represent the usual boolean combinations of the previously introduced constructors. Substituting the free variables xi in ϕ by terms ti is denoted ϕ[t1 /x1 , . . . , tn /xn ]. Then, given a (first-order) formula ϕ, K ϕ is called a modal K-atom and not ϕ a modal not-atom. The signature Σ contains, apart from the constants occurring in the formulas, a countably infinite supply of constants not occurring in the formulas and the Herbrand universe of such a signature is denoted by . Moreover, the equality predicate ≈ in Σ is interpreted as an equivalence relation on . As in [11], hybrid MKNF knowledge bases can contain any firstorder fragment DL satisfying the following conditions: (i) each knowledge base O ∈ DL can be translated into an equivalent formula π(O) of function-free first-order logic with equality, (ii) it supports A-Boxes-assertions of the form P (a1 , . . . , an ) for P a predicate and ai constants of DL and (iii) satisfiability checking and instance checking (i.e. entailment of the form O |= P (a1 , . . . , an )) are decidable4 . We now recall hybrid MKNF knowledge bases of [11]. Definition 1 Let O be a DL knowledge base. A first-order functionfree atom P (t1 , . . . , tn ) over Σ such that P is ≈ or it occurs in O is called a DL-atom; all other atoms are called non-DL-atoms. A (nondisjunctive) MKNF rule r has the following form where H, Ai , 4

For more details on DL notation we refer to [1].

100

M. Knorr et al. / A Coherent Well-Founded Model for Hybrid MKNF Knowledge Bases

and Bi are first-order function free atoms: K H ← K A1 , . . . , K An , not B1 , . . . , not Bm

(1)

The sets {K H}, {K Ai }, and {not Bi } are called the rule head, the positive body, and the negative body, respectively. A rule is positive if m = 0; r is a fact if n = m = 0. A program P is a finite set of MKNF rules. A hybrid MKNF knowledge base K is a pair (O, P). The semantics of such a knowledge base K is obtained by translating it into the MKNF formula π(K) = K π(O) ∧ π(P) where P is transformed by universally quantifying all the variables in each rule. An MKNF rule r is DL-safe if every variable in r occurs in at least one non-DL-atom K B occurring in the body of r. A hybrid MKNF knowledge base K is DL-safe if all its rules are DL-safe. Given a hybrid MKNF knowledge base K = (O, P), the ground instantiation of K is the KB KG = (O, PG ) where PG is obtained by replacing in each rule of P all variables with constants from K in all possible ways.

3

Example Scenario

Consider an online store selling, among other things, CDs. Due to the fact that many newly published CDs are simply compilations of already existing music, the owners decide to offer their customers a special service: whenever somebody likes the compilation of a certain artist he can search specifically for more music of that artist published on albums. The service shall however deny offering other compilations or products which are too similar to the already owned CD. Similarity can be defined in various ways but we assume for simplicity that this is handled internally, e.g. by counting the number of identical tracks, and encoded by predicate Dif(x, y). The internal database is organized as a hybrid MKNF knowledge base including an ontology containing all available discs, their tracks and so on and whether they are albums or compilations. The following shall provide the considered service5 : Comp



¬Offer

K Offer(x)

←

not owns(x), K owns(y), K Dif(x, y), K artist(x, z), K artist(y, z).

(2) (3)

Given the input of CDs the customer owns, rule (3) offers an album x in case the customer does not own it, which is sufficiently different to a CD y he owns, where the artist z of x is the same as the artist of y. Additionally, (2) is a DL statement (translatable into ∀x : Comp(x) → ¬Offer(x)) enforcing that any CD which is a compilation shall never be offered.

4

Three-valued Semantics

We start by defining three-valued structures which serve as a means for evaluating hybrid MKNF knowledge bases. Definition 2 A three-valued (partial) MKNF structure (I, M, N ) consists of a Herbrand first-order interpretation I and two pairs M = M, M1  and N = N, N1  of sets of Herbrand first-order interpretations where any first-order atom which occurs in all elements in M (resp. N ) also occurs in all elements of M1 (resp. N1 ). It is called total if M = M, M  and N = N, N . 5

Capital letters represent DL-atoms and objects/individuals while the other represent non-DL atoms and variables. Note that rule (3) is in fact DL-safe.

Set I is intended to interpret the first-order formulas while the pairs M and N evaluate the modal operators K and not . MKNF formulas are thus interpreted with respect to the set {t, u, f } of truth values with f < u < t where the operator max (resp. min) chooses the greatest (resp. least) elementjwith respect to this ordering: t iff p(t1 , . . . , tn ) ∈ I (I, M, N )(p(t1 , . . . , tn )) = f iff p(t1 , . . . , tn ) ∈ I 8 < t u (I, M, N )(¬ϕ) = : f

iff (I, M, N )(ϕ) = f iff (I, M, N )(ϕ) = u iff (I, M, N )(ϕ) = t

(I, M, N )(ϕ1 ∧ ϕ2 ) = min{(I, M, N )(ϕ1 ), (I, M, N )(ϕ2 )} j (I, M, N )(ϕ1 ← ϕ2 ) =

t iff (I, M, N )(ϕ1 ) ≥ (I, M, N )(ϕ2 ) f otherwise

(I, M, N )(∃x : ϕ) = max{(I, M, N )(ϕ[α/x]) | α ∈ } 8 t iff (J, M, M1 , N )(ϕ) = t > > > > for all J ∈ M < f iff (J, M, M1 , N )(ϕ) = f (I, M, N )(K ϕ) = > > for some J ∈ M1 > > : u otherwise 8 t iff (J, M, N, N1 )(ϕ) = f > > > > for some J ∈ N1 < f iff (J, M, N, N1 )(ϕ) = t (I, M, N )(not ϕ) = > > for all J ∈ N > > : u otherwise Note that first-order atoms, and also first-order (non-modal) formulas, are evaluated with respect to one first-order interpretation, and are, therefore, entirely two-valued. This is intended since this way a rule free hybrid knowledge base shall be interpreted just as any DL base. Moreover, implications are not interpreted in a classical sense: u ← u is true while its classical boolean correspondence, u ∨ ¬u, is undefined. This is needed for the very same reason it is in case of the well-founded semantics of LP: rules propagating undefinedness are true. Without this, it could never be the case that a DL-free hybrid knowledge would coincide with the well-founded semantics of LPs. So, only modal atoms (and thus rules) make use of the third truth value u and we are going to explain the details of this part of the evaluation scheme. Each modal operator is evaluated with respect to a pair of sets of interpretations. The idea is that K ϕ is true if ϕ is true in all elements in M ; otherwise it is either false or undefined depending on M1 . If ϕ is true in all elements in M1 then K ϕ is undefined; otherwise false. The case of not ϕ is handled symmetrically with respect to N, N1 , only now the condition for true modal Katoms yields false modal not-atoms. The restrictions on three-valued MKNF structures guarantee that no modal formula can be true and false at the same time. We now define interpretation pairs which form the basis for a model notion. Definition 3 An interpretation pair (M, N ) consists of two sets of Herbrand interpretations M , N with N ⊆ M , and models a closed MKNF formula ϕ if and only if (I, M, N , M, N )(ϕ) = t for each I ∈ M . If there exists an interpretation pair modeling ϕ then ϕ is consistent. M contains all interpretations which model only truth while N models everything which is true or undefined. Note that the corre-

M. Knorr et al. / A Coherent Well-Founded Model for Hybrid MKNF Knowledge Bases

spondence ¬K being equivalent to not is supported by using the interpretation pair (M, N ) to evaluate both, K and not , simultaneously. The subset relation between M and N does not only guarantee allowed MKNF structures but also that any formula which is true (resp. false), in all elements of M is also true (resp. false) in all elements of N . This will ensure consistency since it prevents modal atoms which are true and false at the same time and, as we will soon see, modal atoms which are undefined though being first-order false. Now we define MKNF models based on a preference relation over interpretation pairs which model the considered formula. Definition 4 Any interpretation pair (M, N ) is a partial (or threevalued) MKNF model for a given closed MKNF formula ϕ if (1) (I, M, N , M, N )(ϕ) = t for all I ∈ M and (2) (I  , M  , N  , M, N )(ϕ) = t for some I  ∈ M  and each interpretation pair (M  , N  ) with M ⊆ M  and N ⊆ N  where at least one of the inclusions is proper. If there is a partial MKNF model of a given closed MKNF formula ϕ then ϕ is called MKNF-consistent, otherwise MKNF-inconsistent. With a fixed evaluation of the modal not-atoms we maximize the sets which evaluate modal K-atoms, checking whether this still yields a true evaluation. By maximizing these sets we naturally obtain less formulas which are true in all elements of these sets and thus less modal K-atoms which are true or undefined. In this sense we deal with a logics of minimal knowledge. Once more, by N  ⊆ M  , we guarantee that only reasonable augmentations are considered. Example 1 Consider the knowledge base from the running example, with the obvious abbreviations, together with the users input owns(C1), and two albums A1, A2, in the database from the same artist, where A1 is sufficiently different from the compilation while A2 is not. Then, restricted to the domain of interest, an interpretation pair (M, N ) modeling the KB and containing owns(C1), Of(A1), and Of(A2) is not an MKNF model since any (M  , N ) such that Of(A2) is not in all elements of M  still models the KB. In fact, the only MKNF model restricted to these three modal atoms would be (M, N ) with M = N = {{owns(C1), Of(A1), Of(A2)}, {owns(C1), Of(A1)}}. One could ask now, what is the point of having u available in this example? The answer is that this simply depends on the intention and design of the reasoning capability: the idea could be only to recommend one disk. For that, we could add not Of(x1), x = x1 to the rule (3) (ensuring additionally DL-safety). Supposing that both, A1 and A2 are sufficiently different from C1 we would obtain two MKNF models of the KB, one with K Of(A1) and one with K Of(A2), and additionally one model which simply does not choose between the two but leaves them both undefined. The advantage of this comes into play when defining a way of calculating a model which incorporates all the minimally necessary true information: it is simpler to compute one slightly less expressive model than to keep track of various of them. Since MKNF models are in general infinite, as in [11] the proper idea for algorithmization is to represent them via a 1st-order formula whose model corresponds to the MKNF model. For that, a partition (T, F ) of true and false modal atoms is provided which allows to determine this first-order formula. Definition 5 Let K = (O, P) be a hybrid MKNF knowledge base. The set of K-atoms of K, written KA(K), is the smallest set that

101

contains (i) all K-atoms occurring in PG , and (ii) a modal atom K ξ for each modal atom not ξ occurring in PG . For a subset S of KA(K), the objective knowledge of S is the forS mula obK,S = O ∪ K ξ∈S ξ, and SbDL = {ξ | K ξ ∈ SDL } where SDL is the subset of DL-atoms of S. A (partial) partition (T, F ) of KA(K) is consistent if obK,T |= ξ for each K ξ ∈ F . Before we continue defining operators which will derive conclusions from knowledge bases, we have to modify MKNF knowledge bases such that we can address the coherence problem: a first-order false formula ϕ (as a consequence of the DL part) has to be connected to not ϕ which cannot be done straightforwardly since not cannot occur in the DL part. Thus, instead of representing the connection directly, we introduce new positive DL atoms which represent the falsity of an already existing DL atom, and a further program transformation which makes these new modal atoms available for reasoning in the respective rules. Definition 6 Let K be a DL-safe hybrid MKNF knowledge base. We obtain K∗ from K by adding an axiom ¬H  N H for every DL atom H(t1 , . . . , tn ) which occurs as head in at least one rule in K where N H is a new predicate not allready occurring in K. Moreover, b from K∗ by adding not N H(t1 , . . . , tn ) to the body of we obtain K each rule with a DL atom H(t1 , . . . , tn ) in the head. The idea is to have N H(t1 , . . . , tn ) available as a predicate representing that ¬H(t1 , . . . , tn ) holds: K∗ makes this connection exb introduces a restriction on each rule with a DL atom in plicit and K the head saying intuitively that the rule can only be used to conclude something if the negation of its head does not hold already. Note that b are still hybrid MKNF knowledge bases, so we only refer K∗ and K ∗ b explicitly when it is necessary. to K and K We now define the monotonic operator TK which allows to draw conclusions from positive hybrid MKNF knowledge bases. Definition 7 For K a positive nondisjunctive DL-safe hybrid MKNF knowledge base, RK , DK , and TK are defined on the subsets of b as follows: KA(K) RK (S)

=

DK (S)

=

TK (S)

=

S ∪ {K H | K contains a rule of the form (1) such that K Ai ∈ S for each 1 ≤ i ≤ n} b and O ∪ SbDL |= ξ}∪ {K ξ | K ξ ∈ KA(K) {K Q(b1 , . . . , bn ) | K Q(a1 , . . . , an ) ∈ S \ SDL , b and K Q(b1 , . . . , bn ) ∈ KA(K), O ∪ SbDL |= ai ≈ bi for 1 ≤ i ≤ n} RK (S) ∪ DK (S)

RK derives immediate consequences from the rules while DK obtains consequences using modal atoms and the statements contained in the DL part. Since TK is monotonic, it has a unique least fixpoint which we denote TK ↑ ω and is obtained in the usual way. Note that TK ↑ ω is in fact order-continuous due to the absence of function symbols in the language and the restriction to known individuals. A transformation for nondisjunctive hybrid MKNF knowledge bases is defined turning them into positive ones, thus allowing the application of the operator TK . Definition 8 Let KG = (O, PG ) be a ground nondisjunctive DLsafe hybrid MKNF knowledge base and S ⊆ KA(KG ). The MKNF transform KG /S = (O, PG /S) is obtained by PG /S containing all rules K H ← K A1 , . . . , K An for which there exists a rule K H ← K A1 , . . . , K An , not B1 , . . . , not Bm in PG with K Bj ∈ S for all 1 ≤ j ≤ m.

102

M. Knorr et al. / A Coherent Well-Founded Model for Hybrid MKNF Knowledge Bases

This resembles the transformation known from stable models of logic programs and the following operator using a fixpoint of TK is thus straightforward to define. Definition 9 Let K = (O, P) be a nondisjunctive DL-safe hybrid b We define: MKNF knowledge base and S ⊆ KA(K). ΓK (S) = TK∗G /S ↑ ω b the apEven though we consider all modal atoms from KA(K), ∗ plied knowledge base KG does not enforce not H(t1 , . . . , tn ) if ¬H(t1 , . . . , tn ) holds. Thus ΓK alone is not sufficient to obtain the intended model and we define an operator similar in appearance but bG. referring to K

5

Properties

In the same manner as done in [7] for the alternating fixpoint of normal logic programs, we restate the iteration for obtaining ΦK and ΨK as: P0 Pn+1 Pω

= = =

∅ Γ SK (Nn ) Pn

N0 Nn+1 Nω

= = =

b KA(K)  Γ (P n) K T Nn

Definition 10 Let K = (O, P) be a nondisjunctive DL-safe hybrid b We define: MKNF knowledge base and S ⊆ KA(K).  ΓK (S) = TKb G /S ↑ ω

It is easy to see that ΦK ↑ 1 = P2 , ΦK ↑ 2 = P4 , i.e. ΦK ↑ i = P2i , and likewise ΨK ↓ i = P2i . In particular, it can be shown that the sequence of Pi (respectively Ni ) is increasing, (respectively decreasing) and without surprise its limits concur with the least fixpoint of ΦK , respectively the greatest fixpoint of ΨK , i.e. Pω = lfp(ΦK ) and Nω = gfp(ΨK )6 . As an overall benefit, we can compute the least fixpoint of ΦK directly from the greatest one of ΨK and vice versa.

Both, Γ and Γ , are shown to be antitonic and we join them as follows to two monotonic operators.

Proposition 1 Let K be a nondisjunctive DL-safe hybrid MKNF knowledge base. Then Pω = Γ(Nω ) and Nω = Γ (Pω ).

Definition 11 Let K = (O, P) be a nondisjunctive DL-safe hybrid b We define: MKNF knowledge base and S ⊆ KA(K).  ΦK (S) = ΓK (ΓK (S)) and ΨK (S) = ΓK (ΓK (S))

It furthermore can be shown that we can even use this computation to check consistency of the KB.

Since both are monotonic we obtain a least and a greatest fixpoint in both cases and the least fixpoint of ΦK and the greatest one of ΨK then define the well-founded partition. Definition 12 Let K = (O, P) be a nondisjunctive DL-safe hybrid MKNF knowledge base and let PK , NK ⊆ KA(K) with PK being the least fixpoint of ΦK and NK the greatest fixpoint of ΨK , both restricted to the modal atoms only occurring in KA(K). Then (PW , NW ) = (PK ∪ {K π(O)}, KA(K) \ NK ) is the well-founded partition of K. Both, PK and NK , are restricted to the modal atoms occurring b are not in K only. Thus, the auxiliary modal atoms introduced via K present in the well-founded partition. But they are not necessary there anyway since their only objective is preventing inconsistencies, i.e. deriving ¬ϕ and not ϕ being undefined. Example 2 Consider the KB K from the example scenario and add the owners compilation C2, i.e. owns(C2) and album A3 and compilation C3 which are both sufficiently different from C2. At first we add a DL statement ¬Of  NOf to the knowledge base to obtain b with head Of(x) receives a K∗ . Additionally, each ground rule in K modal atom not NOf(x) in its body where x here just functions as a placeholder for A3 and C2. When we compute Γ of ∅ then the result contains K NOf(C3), K Of(C3) and of course K Of(A3) and so does Γ applied to this result providing already the least fixpoint of b then only ΦK (due to our simplifications). If we compute Γ of KA(K) K NOf(C3) occurs due to DKb because all rules with modal notatoms are removed from the transform. However, computing Γ of this result derives only additionally K Of(A3) due to not NOf(C3) occurring in the body of the rule with head K Of(C3), and we also obtained the greatest fixpoint of ΨK . So the well-founded partition contains K Of(A3) in PW , i.e. offers A3, but also K Of(C3) in PW and in NW , i.e. C3 is offered and rejected at the same time. This is a clear indication that the knowledge base is inconsistent, as we shall see in the following section, and the (2) and (3) alone are not suitable to provide the intended service.

Proposition 2 Let K be a nondisjunctive DL-safe hybrid MKNF knowledge base and Pω the least fixpoint of ΦK . If Γ (Pω ) ⊂ Γ(Pω ) then K is MKNF-inconsistent. Intuitively, what this statement says is that any inconsistency between rules and the ontology can be discovered by computing the set of non-false modal atoms and then checking whether all false modal atoms are not just enforced to be false by the first-order knowledge although the (unchanged) rules support (at least one of) these false modal atoms. For an inconsistency check this is of course not sufficient, since an inconsistent DL base O is not detected by this method. In fact, in case we want to check for consistency of K we have both to check consistency of O alone, and apply the proposition above. Theorem 1 Let K = (O, P) be a nondisjunctive DL-safe hybrid MKNF knowledge base and Pω the least fixpoint of ΦK . Γ (Pω ) ⊂ Γ(Pω ) or O is inconsistent iff K is MKNF-inconsistent. Normal rules alone cannot be inconsistent, unless we allow integrity constraints as rules whose head is K f (cf. [11]). But then inconsistencies are easily detected since PW or NW contain K f . Example 3 Reconsider the result from Example 2. If we compute Γ of the least fixpoint of ΦK , i.e. Pω then the result now contains K Of(C3) while this is not contained in Γ of Pω . Our assumption that the KB is inconsistent is thus verified. However, in case of a consistent knowledge base, the well-founded partition always yields a three-valued model. Theorem 2 Let K be a consistent nondisjunctive DL-safe hybrid MKNF KB and (PK ∪{K π(O)}, KA(K)\NK ) be the well-founded partition of K. Then (IP , IN ) where IP = {I | I |= obK,PK } and IN = {I | I |= obNK } is an MKNF model – the well-founded MKNF model. In fact, the result is not any three-valued MKNF model but the least one with respect to the following order. 6

ΦK and ΨK are order-continuous for the same reasons as TK .

M. Knorr et al. / A Coherent Well-Founded Model for Hybrid MKNF Knowledge Bases

Definition 13 Let ϕ be a closed MKNF formula and (M1 , N1 ) and (M2 , N2 ) be partial MKNF models of ϕ. Then (M1 , N1 ) ,k (M2 , N2 ) iff M1 ⊆ M2 and N1 ⊇ N2 . This order intuitively resembles the knowledge order where the least element contains the smallest amount of derivable knowledge, i.e. the one which leaves as much as possible undefined. Theorem 3 Let K be a consistent nondisjunctive DL-safe hybrid MKNF KB and (M, N ) be the well-founded MKNF model. Then, for any three-valued MKNF model (M1 , N1 ) of K we have (M1 , N1 ) ,k (M, N ). Moreover, for an empty DL base the well-founded partition corresponds to the well-founded model for (normal) logic programs. Corollary 1 Let K be a nondisjunctive program of MKNF rules, Π a normal logic program obtained from P by transforming each MKNF rule K H ← K A1 , . . . , K An , not B1 , . . . , not Bm into a clause H ← A1 , . . . , An , not B1 , . . . , not Bm of Π, WK = (P, N ) be the well-founded MKNF model, and WΠ be the well-founded model of Π. Then K H ∈ P if and only if H ∈ WΠ and K H ∈ N if and only if not H ∈ WΠ . Finally the data complexity result is obtained basically from the result of TK for positive nondisjunctive MKNF knowledge bases in [11] where data complexity is measured in terms of A-Box assertions and rule facts. Theorem 4 Let K be a nondisjunctive DL-safe hybrid MKNF KB. Assuming that entailment of ground DL-atoms in DL is decidable with data complexity C the data complexity of computing the wellfounded partition is in PC . This means that if the description logic fragment is tractable,7 we end up with a formalism whose model is computed with a data complexity of P.

6

Comparisons and Conclusions

As already said, [11] is the stable model oriented origin of our work. The data complexity for reasoning with (2-valued) MKNF models C in nondisjunctive programs is shown to be E P where E = NP if C ⊆ NP, and E = C otherwise. Thus, computing the well-founded partition generally ends up in a strictly smaller complexity class than deriving one of maybe various MKNF models. However, M is a (two-valued) MKNF model of K iff (M, M ) is a three-valued MKNF model of K and, furthermore, if (M, M ) is the well-founded MKNF model of K, M is the only MKNF model of K. Furthermore, the well-founded partition can also be used in the algorithms presented in [11] for computing a subset of that knowledge which holds in all partitions corresponding to a two-valued MKNF model. The approach presented in [8], though conceptually similar to ours, is based on a different semantics which evaluates K and ¬not (and ¬K and not ) simultaneously, thereby differing from the ideas of [11]. This in particular does not allow to minimize unnecessary undefinedness. Furthermore, in contrast with our approach, [8] does not allow for any form of detection of inconsistencies resulting from the interaction of the DL part and the rules. Instead, it provides a 7

See e.g. the W3C member submission on tractable fragments of OWL 1.1 (now called OWL 2) at http://www.w3.org/Submission/ owl11-tractable/.

103

strange kind of model in these cases which contains undefined modal atoms which are actually first-order false. Therefore, our proposal is more robust than the one in [8] and in fact more closely related to the two-valued one. In summary, here we define a WFS of (tightly integrated) hybrid KBs that is sound wrt. the semantics defined in [11] for MKNF KBs, that has strictly lower complexity, coinciding with it in case there are no rules, and that coincides with the WFS of normal programs [14] in case the DL-part is empty. We also obtain tractable fragments whenever the underlying DL is tractable. Moreover, we define a construction for computing the WF-model that is also capable of detecting inconsistencies. It is worth noting that when inconsistencies come from the combination of the rules with the DL-part (i.e. for inconsistent KBs with a consistent DL-part), the construction still yields some results (e.g. in example 2). This suggests that the method could be further exploited in the direction of defining a paraconsistent semantics for hybrid KBs. This, together with a study of tractable fragments, generalization to disjunctive rules and implementations, are subjects for future work.

REFERENCES [1] The Description Logic Handbook: Theory, Implementation, and Applications, eds., F. Baader, D. Calvanese, D. L. McGuinness, D. Nardi, and P. F. Patel-Schneider, Cambridge University Press, 2 edn., 2007. [2] J. de Bruijn, T. Eiter, A. Polleres, and H. Tompits, ‘Embedding nonground logic programs into autoepistemic logic for knowledge-base combination’, in IJCAI-07. AAAI Press, (2007). [3] T. Eiter, T. Lukasiewicz, R. Schindlauer, and H. Tompits, ‘Combining answer set programming with description logics for the semantic web’, in KR’04, eds., D. Dubois, C. Welty, and M-A. Williams, pp. 141–151. AAAI Press, (2004). [4] T. Eiter, T. Lukasiewicz, R. Schindlauer, and H. Tompits, ‘Wellfounded semantics for description logic programs in the semantic web’, in RuleML’04, eds., G. Antoniou and H. Boley, pp. 81–97. Springer, LNCS, (2004). [5] M. Gelfond and V. Lifschitz, ‘The stable model semantics for logic programming’, in ICLP, eds., R. A. Kowalski and K. A. Bowen. MIT Press, (1988). [6] S. Heymans, D. Van Nieuwenborgh, and D. Vermeir, ‘Guarded open answer set programming’, in LPNMR’05, eds., C. Baral, G. Greco, N. Leone, and G. Terracina, pp. 92–104. Springer, LNAI, (2005). [7] P. Hitzler and M. Wendt, ‘A uniform approach to logic programming semantics’, Theory and Practice of Logic Programming, 5(1–2), 123– 159, (2005). [8] M. Knorr, J. J. Alferes, and P. Hitzler, ‘Towards tractable local closed world reasoning for the semantic web’, in Progress in Artificial Intelligence, eds., J. Neves, M. F. Santos, and J. Machado, pp. 3–14. Springer, (2007). [9] M. Knorr, J. J. Alferes, and P. Hitzler, ‘A coherent well-founded model for hybrid MKNF knowledge bases (extended version)’, Technical report, (2008). Available from the authors. [10] V. Lifschitz, ‘Nonmonotonic databases and epistemic queries’, in IJCAI’91, pp. 381–386, (1991). [11] B. Motik and R. Rosati, ‘A faithful integration of description logics with logic programming’, in Proceedings of the Twentieth International Joint Conference on Artificial Intelligence (IJCAI-07), pp. 477–482. AAAI Press, (2007). [12] L. M. Pereira and J. J. Alferes, ‘Well founded semantics for logic programs with explicit negation’, in ECAI, pp. 102–106, (1992). [13] R. Rosati, ‘Dl+Log: A tight integration of description logics and disjunctive datalog’, in KR’06, eds., P. Doherty, J. Mylopoulos, and C. Welty. AAAI Press, (2006). [14] Allen van Gelder, Kenneth A. Ross, and John S. Schlipf, ‘The wellfounded semantics for general logic programs’, Journal of the ACM, 38(3), 620–650, (1991).

This page intentionally left blank

2. Machine Learning

This page intentionally left blank

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-107

107

Prototype-based Domain Description Fabrizio Angiulli1 Abstract. In this work a novel one-class classifier, namely the Prototype-based Domain Description rule (PDD), is presented. The PDD classifier is equivalent to the NNDD rule under the infinity Minkowski metric for a suitable choice of the prototype set. The concept of PDD consistent subset is introduced and it is shown that computing a minimum size PDD consistent subset is in general not approximable within any constant factor. A logarithmic approximation factor algorithm, called the CPDD algorithm, for computing a minimum size PDD consistent subset is then introduced. The CPDD algorithm has some parameters which allow to tune the trade off between accuracy and size of the model. Experimental results show that the CPDD rule sensibly improves over the CNNDD classifier in terms of size of the subset, while guaranteeing a comparable classification quality.

1

INTRODUCTION

Domain description, or one-class classification, is a classification technique whose goal is to distinguish between objects belonging to a certain class and all the other objects of the space [11]. The Nearest Neighbor Domain Description rule (NNDD) [1] is a one-class classifier accepting test objects whose nearest neighbors distances in a reference data set, assumed to model normal behavior, lie within a certain threshold. In particular, given a data set of objects, also called reference set, and two parameters k and θ, the NNDD associates a feature vector δ(x) ∈ Rk with each object x composed of the distances from x to its first k nearest neighbors in the reference set. The classifier accepts x if and only if δ(x) belongs to the hyper-sphere (according to one of the Lr Minkowski metric, r ∈ {1, 2, . . . , ∞}) centered in the origin of Rk and having radius θ, i.e. if and only if -δ(x)-r ≤ θ. The CNNDD rule is a variant of the NNDD rule using a selected subset of the data set as the reference set [1]. In this work a novel nearest neighbor based one-class classifier, called the Prototype-based Nearest Neighbor classifier (PDD), is introduced. A prototype set is a set of objects xi , also called prototypes, each of which is associated with a radius R(xi ). Given parameter θ, an object y is accepted if it lies within distance θ − R(xi ) from some prototype xi . It is shown that the PDD classifier is equivalent to the NNDD rule under the infinity Minkowski metric (that is for r = ∞) for a suitable choice of the prototype set. Then the concept of PDD consistent subset is introduced, that is a subset of the original prototype set, which, loosely speaking, accepts all the discarded prototypes. It is shown that computing a minimum size PDD consistent subset is in general not approximable within any constant factor. A logarithmic approximation factor algorithm, called the CPDD algorithm, for computing a minimum size PDD consistent subset is then introduced. The CPDD algorithm has some parameters which allow 1

DEIS - University of Calabria, Italy, email: [email protected]

to tune the trade off between accuracy and size of the model. Experimental results show that the CPDD rule sensibly improves over the CNNDD classifier in terms of size of the subset, while guaranteeing a comparable classification quality. Moreover, comparison with the one-class SVM classifier points out that both the compression ratio and the accuracy of the CPDD are comparable to that of the one-class SVM classifier, but with some advantages for the CPDD rule. The rest of the work is organized as follows. Section 2 defines the Prototype-based Domain Description rule (PDD) and the concept of PDD consistent subset. Section 3 investigates the computational complexity of the problem of computing a minimum size PDD consistent subset. Section 4 describes the CPDD rule. Section 5 presents experimental results. Finally, Section 6 presents conclusions and future work.

2

THE PROTOTYPE-BASED DOMAIN DESCRIPTION RULE

In the following U denotes a set of objects, d a distance metric on U , D a set of objects from U , k a positive integer number, θ a positive real number, and r ∈ {1, 2, . . . , ∞} a Minkowski metric Lr . A prototype set P is a set of pairs P = {x1 , r1 , . . . , xn , rn }, where each xi (1 ≤ i ≤ n) is an object of U , also called prototype, and each ri is a real number, also called prototype radius. Given a prototype xi , the prototype radius ri associated with xi is also denoted by R(xi ). Next the Prototype-based Domain Description one-class classifier is defined. Definition 2.1 Given a prototype set P , the Prototype-based Domain Description rule (PDD) according to P , d, and θ, is the function PDDP,d,θ from U to {−1, +1} such that  +1, if ∃x ∈ P such that d(x, y) + R(x) ≤ θ PDDP,d,θ (y) = −1, otherwise The PDD rule accepts an input object y (that is returns the value +1) if y lies within distance R(xi ) from some prototype xi . The PDD rule is a nearest neighbor based one-class classifier. Next the definition of another nearest neighbor based one-class classifier, namely the NNDD rule, is recalled, and then the relationship between there two rules is pointed out. Given an object x of U , the k-th nearest neighbor nnD,d,k (x) of x in D according to d is the object y of D such that there exist exactly k − 1 objects z of D with d(x, z) ≤ d(x, y). If x ∈ D, then nnD,d,1 (x) = x. The k nearest neighbors distances vector δD,d,k (p) of p in D is δD,d,k (p) = (d(p, nnD,d,1 (p)), . . . , d(p, nnD,d,k (p))). Definition 2.2 ([1]) The Nearest Neighbor Domain Description rule (NNDD) according to D, d, k, θ, r, is the function NNDDD,d,k,θ,r

108

F. Angiulli / Prototype-Based Domain Description

from U to {−1, +1} such that

Algorithm CPDD 1. for each object xi in D, determine the distance ri between xi and its k-th nearest neighbor in D 2. for each object xi such that ri ≤ θ, determine the set Ni composed of the objects y of D such that d(xi , y) + ri ≤ θ 3. set P to {xi ∈ D | ri ≤ θ}, and set S and C to the emptyset 4. while |C| ≤ η|P | do

NNDDD,d,k,θ,r (p) = sign(θ − -δD,d,k (p)-r ), where sign(x) = −1 if x < 0, and sign(x) = 1 otherwise. The following definition relates the PDD rule and the NNDD rule. Given a set of objects D, the prototype set P (D, d, k, θ) associated with D w.r.t. d, k, and θ is

(a) determine the object xj of P such that (break ties in favor of the object such that the value rj is minimum)

{x, d(x, nnD,d,k (x)) | x ∈ D ∧ d(x, nnD,d,k (x)) ≤ θ}.

|Nj − C| = max{|Ni − C| : xi ∈ P }

Relationship between the two rules is clarified by the theorem below.

(b) set S to S ∪ {xj , rj }, and C to C ∪ Nj

Theorem 1 Given a set of objects D, and parameters k and θ, it holds that

5. return the set S

(∀x ∈ D)(NNDDD,d,k,θ,+∞ (x) = PDDP (D,d,k,θ),d,θ (x)). Proof. Let x be a generic object of D. If d(x, nnD,d,k (x)) ≤ θ, then NNDDD,d,k,θ,+∞ (x) = sign(θ − -δD,d,k (p)-+∞ ) = sign(θ − d(x, nnD,d,k (x))) = 1. Furthermore, the pair x, d(x, nnD,d,k (x)) belongs to P (D, d, k, θ) and, hence, d(x, x) + R(x) = 0 + d(x, nnD,d,k (x)) ≤ θ and PDDP (D,d,k,θ),d,θ (x) = 1. If d(x, nnD,d,k (x)) > θ, then NNDDD,d,k,θ,+∞ (x) = −1. By contradiction, assume that there exists a pair y, R(y)) in P (D, d, k, θ) such that d(x, y) + R(y) ≤ θ. Since within distance ry = d(x, y) + d(y, nnD,d,k (y)) from x there are at least k + 1 objects of D, it holds that d(x, nnD,d,k (x)) ≤ ry ≤ θ, which contradicts the hypothesis. 2 Thus, from the point of view of the objects belonging to the data set D, the prototype set P (D, d, θ, k) is the analogue for the PDD rule of the data set D for the NNDD rule. When the reference set D is large, space requirements to store D and time requirements to find the nearest neighbors of an object in D increase. In the spirit of the reference set thinning problem for the k-NN-rule [9, 2], the concept of NNDD reference consistent subset was defined in [1]. In the same spirit, next it is provided the definition of PDD consistent subset. Let P be a prototype set and let S be a subset of P . The set S is said to be a PDD consistent subset of P with respect to d and θ, if the following relationship hold (∀x, r ∈ P )(PDDP,d,θ (x) = PDDS,d,θ (x)). Importantly, it also holds that a PDD consistent subset S of the set P (D, d, θ, k) is the analogue for the PDD rule of the data set D for the NNDD rule. It can be finally concluded from the concept of sample compression scheme [7] and from the discussion above that replacing the prototype set P with a consistent subset S of P improves both response time and generalization.

3

COMPLEXITY ANALYSIS

In this section the computational complexity of the problem of computing a minimum size PDD consistent subset is investigated. The reader is referred to [8, 3] for basics on complexity theory, NP optimization problems, and approximation algorithms. Next it is shown that, in the general case, the problem of computing a minimum size PDD consistent subset is not in the APX complexity class, which is, loosely speaking, the class of the NP optimization problems whose optimal solution can be approximated in polynomial time within a fixed factor.

Figure 1. The CPDD algorithm.

Given a prototype set P , distance metric d, and a positive real number θ, the PDD Consistent Subset Problem P, d, θ is defined as follows: compute a PDD consistent subset S ∗ of P with respect to d and θ, also said a minimum size PDD consistent subset, such that, for each PDD consistent subset S of P with respect to d and θ, |S ∗ | ≤ |S|. Given a positive integer m, the decision version P, d, θ, mD of the problem P, d, θ is defined as follows: reply “yes” if there exists a PDD consistent subset S of P with respect to d and θ such that |S| ≤ m, and reply “no” otherwise. Theorem 2 The P, d, θ problem (1) is NP-hard, and (2) is not in APX. Proof sketch. (Point 1) Membership is immediate. As for the hardness the proof is by reduction of the Dominating Set Problem [8]. Let G = (V, E) be an undirected graph, and let m ≤ |V | be a positive integer. The Dominating Set Problem is: is there a subset U ⊆ V , called dominating set of G, with |U | ≤ m, such that for all v ∈ (V − U ) there exists u ∈ U with {u, v} ∈ E ? Define the metric dV on the set V of nodes of G as follows: dV (u, v) = θ, if {u, v} ∈ E, and dV (u, v) = 2θ, otherwise. Let PV be the set {v, 0 | v ∈ V }. It can be proved that G has a dominating set of size m if and only if PV , dV , θ, mD is a “yes” instance. The NP-hardness of the P, d, θ problem follows immediately from the NP-completeness of its decision version. (Point 2) It is known that the Minimum Dominating Set Problem, that is the problem of determining the size of the smallest dominating set of a graph, is not in APX [4]. We note that Point 1 of this theorem defines an AP-reduction from the Minimum Dominating Set Problem to the Minimum PDD Consistent Subset Problem (the reader is referred to [3] for the definition of AP-reduction). As an immediate consequence of this reduction, the latter problem does not belong to APX. 2

4

THE CPDD ALGORITHM

Figure 1 shows the algorithm CPDD. Given a data set D, the CPDD algorithm computes a PDD consistent subset of the prototype set P (D, d, k, θ) associated with D. The algorithm receives in input a data set D, parameters k and θ, and the additional parameters , η ∈ (0, 1], whose use will be

109

F. Angiulli / Prototype-Based Domain Description

(a) θ = 0.2, = 1.00, η = 1.00

(b) θ = 0.2, = 0.75, η = 1.00

(c) θ = 0.2, = 0.75, η = 0.99

(d) θ = 0.1, = 1.00, η = 1.00

(e) θ = 0.1, = 0.75, η = 1.00

(f) θ = 0.1, = 0.75, η = 0.99

Figure 2. Examples of PDD consistent subsets computed by the CPDD algorithm.

discussed in the following (if not otherwise specified, it is assumed that  and η are both set to one). Initially, for each object xi of D, the algorithm determines the distance ri to its k-th nearest neighbor (step 1) and also the set Ni of the objects of D lying within distance θ − ri from it (step 2). The set P built in step 3 is composed of the objects occurring in the prototype set P (D, d, k, θ). Then the algorithm computes the consistent subset S following a greedy strategy (step 4). The set C consists of the objects of P which are correctly classified by the current subset S. At each step, the object xj maximizing the number of objects in Nj −C is selected and inserted in S, until C contains at least the fraction η of the objects in P (until C covers P , if η = 1). Next theorem shows that the the size of the solution returned by the algorithm has an approximation factor. Theorem 3 The CPDD algorithm provides a solution having a 1 + ln(n) approximation factor. Proof. Assume that the parameter  is set to one. We note that the set Ni consists of precisely all the prototypes of P (D, d, θ, k) which are correctly recognized through the PDD rule if xi is included in the PDD consistent subset S. Given a finite set S and a collection C of subsets of S, a set cover for S is a subset C  of C such that every element in S belongs to at least one member of C  . It is clear that the PDD consistent subsets of P are in one-to-one correspondence with the set covers of {Ni | xi ∈ P }. The result hence follows by noting that step 4 of the algorithm CPDD is analogous to the greedy algorithm for the Minimum Set Cover Problem [6], the problem of computing a set cover of minimum size, which achieves an approximation factor of 1 + ln(n), where n is the size of the set to be covered. 2

Note that steps 3-5 compute a PDD consistent subset of any arbitrary prototype set. Figure 2 reports some examples of PDD consistent subsets computed by the CPDD algorithm. The data set (blue points) is composed of ten thousands points in the plane. The parameter k was set to 5, while two distinct values for the parameters θ,  and η were considered, namely 0.1 and 0.2 for θ, 0.75 and 1.0 for , and 0.99 and 1.0 for η. The Euclidean distance was employed as distance function d. Stars (in red color) denote the prototypes belonging to the PDD consistent subset S, while the (black) curve denotes the decision boundary of the classifier PDDS,d,θ . The relative size of the PDD consistent subsets reported in Figure 2 is summarized in the following table. θ = 0.2 θ = 0.1

 = 1.00 η = 1.00 70 (0.7%) 227 (2.3%)

 = 0.75 η = 1.00 128 (1.3%) 439 (4.4%)

 = 0.75 η = 0.99 62 (0.6%) 337 (3.4%)

From the figure and the table above it is clear that the smaller the value of the parameter θ, the closer the class boundary to the data set shape, the greater the number of data set objects rejected by the PDD rule, and the greater the number of prototypes belonging to the consistent subset. Moreover, the smaller the value of the parameter , the greater the number of prototypes belonging to the consistent subset, the more accurate the form of the decision boundary, and the smaller the probability of rejecting objects belonging to the class represented by the data set. For example, in Figure 2(a) ( = 1) there is a “hole”, approximately centered in (−0.78, −0.78), in the lower tail of the data set (but also other smaller “holes” exist along the data set shape), while the same region is covered by the prototypes in Figure 2(b) ( = 0.75). Finally, the smaller the value of the parameter η, the smaller the

110

CPDD TNR

0.8 CNNDD TNR

0.4

CNNDD size

0.2 CPDD size

0 0

0.05

0.1 0.15 False Negative Rate

0.2

CNNDD TNR

0.8

0.6 CPDD TNR CNNDD size

0.4

0.2 CPDD size

0 0

0.05

(a) CPDD AUC=0.997, CNNDD AUC=0.996 (b) CPDD AUC=0.967

η=0.95

DR CNNDD

0.8 ρ=0.9

0.6

FP

DR ρ=0.9

0.4

η=0.95

CPDD

0.2 FP

0 −3 10

−2

0.2

Satellite image red soil, k=4 1 CNNDD TNR

0.8 CPDD TNR

0.6

0.4 CNNDD size

0.2 CPDD size

0 0

0.05

−1

0

10 10 Relative subset size, |S|/|D|

10

η=0.95

TNR CNNDD FNR ρ=0.9

0.6 ρ=0.9

0.4

0.2

η=0.95

TNR CPDD FNR

0 −2 10

−1

10 Relative subset size, |S|/|D|

(d)

0.2

Satellite image red soil, k=4

1

0.8

0.1 0.15 False Negative Rate

CNNDD (c) CPDD AUC=0.986, CNNDD AUC=0.992

AUC=0.970, Ionosphere good, k=4

1

False Negative Rate, True Negative Rate

False Negative Rate, True Negative Rate

Image segmentation path, k=4

0.1 0.15 False Negative rate

0

10

False Negative Rate, True Negative Rate

0.6

Ionosphere good, k=4 1

True Negative Rate, Relative subset size |S|/|D|

Image segmentation path, k=4 1

True Negative Rate, Relative subset size |S|/|D|

True Negative Rate, Relative subset size |S|/|D|

F. Angiulli / Prototype-Based Domain Description

1

0.8

η=0.95

TNR CNNDD FNR

0.6 ρ=0.9

0.4

0.2 ρ=0.9 TNR 0 −3 10

(e)

η=0.95

CPDD

FNR

−2

−1

10 10 Relative subset size, |S|/|D|

0

10

(f)

Figure 3. Comparison between the CPDD and the CNNDD rule.

number of prototypes belonging to the consistent subset, but the greater the probability of rejecting objects belonging to the class represented by the data set, since the most sparse regions of the feature space belonging to the class are left uncovered.

5

EXPERIMENTAL RESULTS

In this section, experiments involving the CPDD rule on three data sets from the UCI Machine Learning Repository, namely Image segmentation, Ionosphere, and Satellite image, are described.2 In particular, for the Image segmentation data set (19 attributes) the path class (330 objects) was considered the normal one, while the remaining 1,980 objects were considered anomalies, for the Ionosphere data set (34 attributes) the good class (225 objects) was considered the normal one, while the objects of the bad class were considered anomalies, and for the Satellite image (36 attributes) the red soil class (1,533 objects) was considered the normal one, while the remaining 3,902 objects were considered anomalies. Figure 3 reports comparison of the CPDD and CNNDD (for r = +∞) rules on the three considered data sets. The parameter k was set to 4 in all the experiments, while the parameter θ was varied from zero to a suitable large value, and, then, the size of the subset computed, the false negative rate, and the true negative rate, were measured. If not otherwise specified, the parameters  and η are set to 1. The Euclidean distance was employed as distance function d. The True Positive Rate (TPR, for short) is the fraction of normal 2

Also other data set were considered: the behavior of the method on these other data sets was analogous to what here described.

objects accepted by the classifier, while the False Positive Rate (FPR, for short) is the fraction of abnormal objects accepted by the classifier. Dually, the False Negative Rate (FNR, for short) is the fraction of normal objects rejected by the classifier, while the True Negative Rate (TNR, for short) is the fraction of abnormal objects rejected by the classifier. It holds that FNR=1-TPR and FPR=1-TNR. Figures 3(a)-(c) compare the ROC curves of the CPDD (solid lines) and CNNDD (dash-dotted lines) methods and also the relative size |S|/|D| of the corresponding consistent subsets S achieving the same value of FNR. The ROC curve is the plot of the FNR versus the TNR (or, correspondingly, TPR versus TNR), and the area under the ROC curve (AUC, for short) provides a summary to compare two classifiers. From these curves it is clear that the the CPDD consistent subset (dashed lines) is much smaller than the CNNDD subset (dotted lines) guaranteeing the same FNR. Moreover, the AUCs of the two methods are very similar. Figures 3(d)-(f) report the TNR (solid lines) ad FNR (dashed lines) of the CPDD method, and the TNR (dash-dotted lines) and FNR (dotted lines) of the CNNDD method as a function of the relative subset size |S|/|D|. For the CPDD method the pair of parameters  = 1, η = 0.95 (upper curve),  = 1, η = 1 (middle curve), and  = 0.9, η = 1 (lower curve), were considered, in order to study sensitivity to these parameters. As far as the middle curves of the CPDD ( = 1, η = 1) and the curves of the CNNDD is concerned, it can be noted that for the same value of FNR or TNR the subset of the CPDD is sensibly smaller than that of the CNNDD. As notable examples (highlighted by means of big points on the curves) compare (1)

111

0.8

SVM TNR CPDD TNR

0.6

SVM size, γ=1.0

0.4

η=1.00

0.2

η=0.95 CPDD size, k=4

0 0

0.2

0.4 0.6 False Negative Rate

0.8

1

(a) CPDD AUC=0.997, SVM AUC=0.982

Ionosphere good 1

CPDD TNR

SVM TNR

0.8 SVM size, γ=0.1

0.6 η=1.00

0.4

η=0.95

0.2

CPDD size, k=4

0 0

0.2

0.4 0.6 False Negative Rate

0.8

1

True Negative Rate, Relative subset size |S|/|D|

Image segmentation path

1

True Negative Rate, Relative subset size |S|/|D|

True Negative Rate, Relative subset size |S|/|D|

F. Angiulli / Prototype-Based Domain Description

(b) CPDD AUC=0.970, SVM AUC=0.918

Satellite image red soil 1 SVM TNR

SVM size, γ=0.0001

0.8 CPDD TNR

0.6

0.4

0.2

η=1.00

0 0

0.2

η=0.95

CPDD size, k=4

0.4 0.6 False Negative Rate

0.8

1

(c) CPDD AUC=0.986, SVM AUC=0.974

Figure 4. Comparison between the CPDD rule and the one-class SVM.

the CPDD subset of relative size 0.054, achieving FNR=0.024 and TNR=0.997, with the CNNDD subset of relative size 0.158, achieving FNR=0.027 and TNR=0.983, for the Image segmentation data set, (2) the CPDD subset of relative size 0.064, achieving FNR=0.031 and TNR=0.869, with the CNNDD subset of relative size 0.277, achieving FNR=0.045 and TNR=0.862, for the Ionosphere data set, and (3) the CPDD subset of relative size 0.042, achieving FNR=0.041 and TNR=0.952, with the CNNDD subset of relative size 0.106, achieving FNR=0.027 and TNR=0.943, for the Satellite image data set. As far as the upper curves of the CPDD is concerned ( = 1, η = 0.95), it can be noted that by decreasing the value of the parameter η, very high values of TNR are obtained in correspondence of very small subsets, but the associated FNR worsens with respect to the case η = 1. This can be explained since the smaller the parameter η, the greater the portion of the accepting region of the PDD rule which is left uncovered by the CPDD consistent subset. As far as the lower curves of the CPDD is concerned ( = 0.9, η = 1), on the contrary, it can be noted that by decreasing the value of the parameter , the FNR improves while the TNR gets worse. This can be explained since the smaller the parameter , the greater, and also the closer to each other, the number of prototypes composing the CPDD consistent subset. Hence, by properly setting the parameters  and η the user can tune the trade off between FNR and TNR and, simultaneously, between subset size and accuracy. The following table summarizes the AUCs of the CPDD for various combinations of the parameters  and η.  η Image segmentation Ionosphere Satellite image

Data set

1.00 1.00 0.997 0.970 0.986

1.00 0.95 0.989 0.956 0.981

0.90 1.00 0.996 0.972 0.989

0.90 0.95 0.990 0.967 0.987

Figure 4 compares the ROC curves of the CPDD method with that of the one-class SVM classifier [10, 5]. As for the one-class SVM, the radial basis function kernel was used, varying parameter γ between 10−4 and 102 , and then the curve associated with the best AUC has been selected. As for the CPDD rule, the parameter used were k = 4,  = 1, and η ∈ {0.95, 1.00}. Interestingly, the CPDD performed better than the one-class SVM both in terms of accuracy (AUCs of the two methods are reported in figure) and in terms of size of the model. Indeed, as for the size of the model is concerned, for small FNRs, the size of the CPDD subset is practically identical to the number of support vectors, while, for greater values of FNRs, the former number is much smaller than the

latter one. This can be explained by noticing that the CPDD subset does not contain the reference set outliers, which form the great majority of the reference set for large FNRs. Moreover, by setting the parameter η to 0.95, the size of the CPDD subset is further decreased, while the accuracy of the CPDD classifier remained good, as witnessed by the table above reported.

6

CONCLUSIONS AND FUTURE WORK

In this work the CPDD one-class classification algorithm has been presented and compared with the CNNDD and the one-class SVM classifiers, pointing out some advantages of the novel approach. A lot of additional questions are worth of being considered. Among them, studying the sensitivity of the method to the parameter k, comparison with the CNNDD rule under different metrics, comparison with other one-class classification methods, and using kernel functions to possibly improve size of the model and/or accuracy.

REFERENCES [1] F. Angiulli, ‘Condensed nearest neighbor data domain description’, IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(10), 1746–1758, (2007). [2] F. Angiulli, ‘Fast nearest neighbor condensation for large data sets classification’, IEEE Transactions on Knowledge and Data Engineering, 19(11), 1450–1464, (2007). [3] G. Ausiello, P. Crescenzi, G. Gambosi, V. Kann, A. MarchettiSpaccamela, and M. Protasi, Complexity and Approximation, SpringerVerlag, Berlin, 1999. [4] M. Bellare, S. Goldwasser, C. Lund, and A. Russeli, ‘Efficient probabilistically checkable proofs and applications to approximations’, in STOC, pp. 294–304, (1993). [5] C.-C. Chang and C.-J. Lin, LIBSVM: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/∼cjlin/libsvm. [6] V. Chv´ atal, ‘A greedy heuristic for the set-covering problem’, Mathematics of Operations Research, 4(3), 233–235, (1979). [7] S. Floyd and M. Warmuth, ‘Sample compression, learnability, and the vapnik-chervonenkis dimension’, Machine Learning, 21(3), 269–304, (1995). [8] M.R. Garey and D.S. Johnson, Computer and Intractability, W. H. Freeman and Company, New York, 1979. [9] P.E. Hart, ‘The condensed nearest neighbor rule’, IEEE Trans. on Information Theory, 14, 515–516, (1968). [10] B. Schlkopf, J.C. Platt, J. Shawe-Taylor, A.J. Smola, and R.C. Williamson, ‘Estimating the support of a high-dimensional distribution’, Neural Computation, 13(7), 1443–1471, (2001). [11] D.M.J. Tax, One-class classification, Ph.D. dissertation, Delft University of Technology, June 2001.

112

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-112

Online Rule Learning via Weighted Model Counting Fr´ed´eric Koriche1 Abstract. Online multiplicative weight-update learning algorithms, such as Winnow, have proven to behave remarkably for learning simple disjunctions with few relevant attributes. The aim of this paper is to extend the Winnow algorithm to more expressive concepts characterized by DNF formulas with few relevant rules. For such problems, the convergence of Winnow is still fast, since the number of mistakes increases only linearly with the number of attributes. Yet, the learner is confronted with an important computational barrier: during any prediction, it must evaluate the weighted sum of an exponential number of rules. To circumvent this issue, we convert the prediction problem into a Weighted Model Counting problem. The resulting algorithm, SharpNow, is an exact simulation of Winnow equipped with backtracking, caching, and decomposition techniques. Experiments on static and drifting problems demonstrate the performance of the algorithm in terms of accuracy and speed.

1

INTRODUCTION

A recurrent theme in Machine Learning is the development of online mistake-driven learning algorithms [4]. Such algorithms are “anytime learners” that can be interrupted at each instant to provide a prediction whose correctness is related to the number of mistakes that have been made so far. Basically, the underlying model takes place in a sequence of trials. At any time step, the learner is presented an observation and it is asked to predict its associated class. If the prediction is incorrect, we charge it one mistake. In a landmark paper, Littlestone [13] introduced the Winnow algorithm, which has rapidly become the blueprint of many efficient online learners. Winnow resembles the Perceptron algorithm in its simplicity, but uses multiplicative, rather than additive, weight updates on input features. Consequently, when the target concept is a k out of n variable disjunction, the number of mistakes grows as k log n instead of kn. The fact that the dependence on n is reduced to logarithmic, rather than linear, makes this algorithm applicable even if the number of features is enormous. This remarkable property opens the door to learning problems characterized by high-dimensional feature spaces. One of them concerns the well-known problem of rule learning which consists in identifying, from a collection of examples, a small set of rules that explains all the positive examples and none of the negative ones [10]. In the paradigm of concept learning, any rule theory can be viewed as a DNF formula, that is, a disjunction of conjunctive features. Based on this notion, Winnow can naturally be extended to rule theories by projecting the data into an higher-dimensional feature space in which any conjunctive feature is viewed as a basic attribute. The enhanced algorithm inherits of an increased expressiveness while preserving a strong learning power. Indeed, if the observed examples are vectors of n attributes taking values over a domain of size d then, providing 1

LIRMM, Universit´e Monptellier 2, France, [email protected]

that the number of all conjunctive features is bounded by (d + 1)n , the performance of Winnow degrades only linearly with the input dimension. In fact, for any target concept, the number of mistakes grows essentially as kn log(d + 1), where k is the minimum number of rules needed to represent the concept into a rule theory. In its primal form, Winnow maintains a vector that assigns a weight to each distinct feature. For rule learning, such an implementation is computationally prohibitive, since the number of possible rules grows exponentially with the input dimension. Specifically, during any prediction, the learner is confronted with the problem of computing the weighted sum of an exponential number of features. Kernel methods have emerged as a standard approach for solving counting problems that arise from high-dimensional feature spaces. The underlying idea is to start from the dual form of the learning algorithm and to use a kernel function that simulates the target feature space while working with original input data. Specifically, in the setting of Boolean DNF formulas, efficient kernel functions have been obtained for the Perceptron algorithm and its maximum margin variants [15]. Unfortunately, it seems impossible to find an analogous result for the Winnow algorithm. Indeed, as observed by Khardon et al. [11], the Kernel Winnow Prediction problem is #P-hard, even for the restricted class of monotone DNF formulas. Such a computational barrier does not imply that, in practice, the sole option for the learner is a brute force enumeration of its feature space. To this very point, in the AI literature, a great deal of attention has been devoted to a related problem, referred to as Weighted Model Counting [5, 9, 17]. The problem is to evaluate the sum of weights of all the assignments satisfying a CNF formula. The basic building block for most model counting algorithms is the Davis-Putnam (DP) procedure that performs a backtracking search in the space of candidate models [3]. Based on this procedure, recent programs such as Cachet [16] and SharpSat [19] can handle large instances by combining formula caching and decomposition into connected components. The power of these techniques raises the natural question of whether the Kernel Winnow Prediction problem can be solved in practice by translation to Weighted Model Counting. This paper provides initial evidence that the answer is affirmative: such a translation can be indeed effective for learning, with fast convergence and speed, “sparse” target concepts involving a small number of relevant rules. The resulting algorithm, called SharpNow, is an exact simulation of kernel Winnow, equipped with backtracking, caching, and decomposition techniques. For sparse target concepts, it can efficiently handle large spaces of conjunctive features with an accuracy superior to that of kernel Perceptron-like algorithms. The background about online rule learning can be found in section 2. The translation method and the SharpNow algorithm are presented in section 3. Experiments on both static and drifting problems are reported in section 4. Finally, section 5 concludes the paper with a discussion about related work and perspectives of further research.

113

F. Koriche / Online Rule Learning via Weighted Model Counting

2

ONLINE RULE LEARNING

In the online learning model, the algorithm is presented with a series of examples {xt , yt } labeled by a target concept. At each time step t, the algorithm first outputs a class prediction yˆt for the observation xt , and then updates its hypothesis based on the true class yt . We adopt the convention that yt ∈ {−1, +1}. The learners of particular interest in this study are the Perceptron [14] and Winnow [13] algorithms applied to rule theories.

2.1

Rule Theories

Let {x1 , · · · , xn } be a set of attributes taking values in a discrete domain of size d. For convenience, we assume a standard way of naming values, as a list of natural numbers. A concept is a function f from dn to {−1, +1}, and an example is a vector x in dn . We say that a series of examples {xt , yt } is labeled by a target concept f if f (xt ) = yt for each index t. An atom is an expression xi = j, also denoted xji , where xi is an attribute and j a value. A conjunctive feature, or rule, is a conjunction of atoms, and a rule theory is a set of rules. A rule r covers an example x if for all atoms xji in r, the value of xi in x is j. A rule theory R covers an example x if at least one rule in R covers x. For instance, the following expression is a theory with three rules ⎧ ⎨ sky = sunny ∧ humidity = normal sky = cloudy ∧ temp = mild ⎩ sky = rain ∧ wind = weak The example (sunny, normal , mild , weak ) is classified as positive by this theory, because it is covered by the first rule. It is well-known that any concept can be represented by an equivalent rule theory. The rule size of a concept f , denoted |f |, is the minimal number of rules needed to represent f as a rule theory. The feature space Rn,d of all rules generated from n attributes taking values over a domain of size d is represented by an indexed set {r1 , · · · , rN }, where N = (d + 1)n . Given an example x, the feature expansion of x onto Rn,d is a vector φ(x) in {0, 1}N where φi (x) = 1 if and only if ri covers x.

2.2

In particular, the kernel obtained by Sadohara [15] can be derived from d = 2. Based on the so-called kernel trick, the prediction rule 1 can be replaced with  yˆt = sign

yˆt = sign(wt · φ(xt ))

(1)

No change is made to wt if the prediction is correct. In case of mistake, the algorithm uses the additive rule wt+1 = wt + yt φ(xt )

(2)

For rule theories, implementing the Perceptron in its the primal form is computationally infeasible since we would need to maintain a weight vector of (d+1)n size. Yet, it is well-known that the dual form of the algorithm is a linear combination of inner products formed by the current observation xt and the previous examples {(xs , ys )} on which mistakes where made. In the setting of Rn,d , each inner product φ(xs ) · φ(xt ) can be simulated by the kernel function K(xs , xt ) = 2|{i:xs,i =xt,i }|

ys K(xs , xt )

(3)

s=1

Each trial of the kernel Perceptron algorithm can thus be executed in polynomial time. Unfortunately, the algorithm can provably require many updates even for very simple rule theories [11]. Theorem 1. There exists a target concept of polynomial rule size and a sequence of examples labeled by it which causes the kernel Perceptron algorithm to make 2Ω(n) mistakes. Importantly, this result still holds for most Perceptron-like algorithms, including the version parameterized with a learning rate and a nonzero threshold, and the recent maximum margin variants [12].

2.3

Winnow

The algorithm has a very similar structure. It takes as input two parameters: a learning rate η and a threshold θ, and maintains a vector wt which is initialized to w0 = 1. Upon receiving an example xt , the algorithm predicts according to the rule yˆt = sign(wt · φ(xt ) − θ)

(4)

Again no change is made to wt if the prediction is correct. In case of mistake, the hypothesis is updated with a multiplicative rule wt+1 = wt exp(ηyt φ(xt ))

(5)

According to these specifications, the following result can be deduced by a simple adaptation of Winnow’s amortized analysis [1]. η Theorem 2. Let θ = 2 sinh (d + 1)n . Then, for any target concept η f , the number m of mistakes made by the Winnow algorithm over any sequence of examples labeled by f satisfies

Perceptron

For sake of clarity, we examine the zero-threshold version of the Perceptron algorithm. Throughout its execution, it maintains a vector wt in RN which is initialized to w0 = 0. Upon receiving an example xt , the algorithm predicts using the rule

m

m≤

eη + 1 [1 + |f |n log(d + 1)] η

Thus, the Winnow algorithm has a polynomial mistake bound for learning polynomial-size rule theories. However, the key difficulty is to provide a computationally efficient simulation of the algorithm. Specifically, the Kernel Winnow Prediction problem is to infer the sign of wt · φ(xt ) − θ for the last example of a given sequence {(xt , yt )} of examples labeled by a target polynomial-size rule theory, after applying the prediction rule 4 and the update rule 5 on the weight vector wt for the previous examples. As shown in [11] this problem is #P-hard, which implies that, unless #P = P, there is no general construction that will run Winnow using kernel functions. In a nutshell, the most important message to be gleaned from online rule learning is that both additive and multiplicative update algorithms are, in theory, limited by either computational efficiency or convergence reasons. On the one hand, even if kernel Perceptron-like algorithms may be executed efficiently, they can provably require an exponential number of mistakes and, on the other hand, even if the kernel Winnow algorithm has a polynomial mistake bound, it seems impossible to simulate its execution in polynomial time.

114

3

F. Koriche / Online Rule Learning via Weighted Model Counting

WEIGHTED MODEL COUNTING

Algorithm 1: W MC(KB , A)

Despite the undoubtable importance of the aforementioned results, it remains to be seen whether, in practice, the execution of kernel Winnow can be simulated using efficient techniques that have been developed for solving real-world #P problems. The key motivation of this paper is to convert the Kernel Winnow Prediction problem into a Weighted Model Counting problem, for which general and efficient model counting techniques can be applied.

3.1

The Translation Method

Informally, any instance of the Weighted Model Counting problem consists in a set of weighted clauses; the task is to evaluate the sum of weights of assignments satisfying these clauses [17]. The intuitive idea behind the translation method is just to ascribe a weighted clause to each labeled example that has led to a mistake. To this end, we need to introduce additional definitions. Consider again a set {x1 , · · · , xn } of attributes taking values over a discrete domain of size d. In the following, any rule defined over this vocabulary is viewed as an assignment, that is a set of atoms or, equivalently, a map that assigns to each atom a value in {0, 1}. A literal is an atom xji or its negation ¬xji and a clause is a disjunction of literals. Given a rule r and a literal xji (resp. ¬xji ), we say that r satisfies xji (resp. ¬xji ), if xji ∈ r (resp. xji ∈ r). Given a rule r and a clause c, we say that r satisfies c if r satisfies at least one literal occurring in c. Based on these notions, the feature expansion of a clause c under Rn,d is a map φ(c) in [0, 1]N where φi (c) = 1 if and only if ri satisfies c. A weighted clause is an expression of the form (c, w) where c is a clause and w is a value in R. Intuitively, w reflects how strong a constraint it is: the higher the weight, the greater the difference in likelihood between a rule that satisfies the constraint and one that does not. In this setting, any “unweighted” clause c is treated as an abbreviation of (c, 0): it denotes a hard constraint that restricts the space of possible rules. A weighted knowledge base KB is a finite set of weighted clauses. The weight of KB , denoted KB, is KB  =

N



w1−φi (c)

(6)

i=1 (c,w)∈KB

As usual, we take the convention that 00 = 1 and 0z = 0 for any real number z > 0. Thus, the weight of the knowledge base KB is just the sum of weights of the assignments that are models of KB . We are now in position to present the translation method. The learning algorithm starts with the knowledge base KB 0 = ∅. On seeing an example xt , the algorithm predicts according to

(7) yˆt = sign KB t ∪ {¬xji : xt,i = j} − θ Recall that in the above expression, each unary clause ¬xji is treated as an abbreviation of (¬xji , 0). In case of mistake, KB t is simply expanded with a weighted clause that conveys, in a concise form, the information gathered from xt and yt . In formal terms 

{xji : xt,i = j}, eηyt (8) KB t+1 = KB t ∪ For instance, suppose that our learner makes a mistake on the positive example (sunny, normal , mild , weak ). Then its knowledge base is expanded with the weighted clause (c, eη ), where c is the disjunction of all atoms sky = cloudy, sky = rain, . . . that are false in the example. In the next prediction, the weight of any rule that violates c will be multiplied by eη .

Input: a weighted knowledge base KB and a set of atoms A Output: the sum of weights of all subsets of A according to KB if I N C ACHE(KB ) then return G ET F ROM C ACHE(KB ) if I S L EAF(KB ) then return 2|A| ∗ (c,w)∈KB w weight ← 1 for each connected component KB i in KB do let Ai be the set of atoms occurring in KB i choose an atom a in Ai KB i,1 ← {(c − {a}, w) : (c, w) ∈ KB i , a ∈ c} weight 1 ← W MC(KB i,1 , Ai − {a}) KB i,2 ← {(c − {¬a}, w) : (c, w) ∈ KB i , ¬a ∈ c} weight 2 ← W MC(KB i,2 , Ai − {a}) S ET T O C ACHE(KB i , weight 1 + weight 2 ) weight ← weight ∗ (weight 1 + weight 2 ) return weight

3.2

The SharpNow Algorithm

In the spirit of model counting algorithms such as Cachet [16] and SharpSat [19], we begin to develop a procedure for evaluating the weight of knowledge base that combines backtracking search with formula caching and decomposition into connected components. The procedure, called W MC, takes as input a weighted knowledge base KB and a set of atoms A. Basically, the procedure performs a depth-first search in the tree of partial assignments generated from A. Notably, a leaf of the tree is reached whenever every clause occurring in KB is empty. In this case, the resulting weight can be evaluated by simply taking the product of weights of these clauses. Following [2], the depth-first search procedure is enhanced by using decomposition into connected components. Namely, by identifying in linear time the connected components in the constraint graph of KB , the resulting weight can be determined by multiplying together the weight of each connected component. Finally, the W MC procedure is equipped with a caching technique that prevents it from recomputing the same component. Because the length of weighted clauses is in O(nd), we employ the hybrid coding scheme suggested in [19] that concisely encodes a set of clauses KB as a vector of indices. Notice that the technique of component caching is particularly relevant in the setting of online learning when the algorithm is susceptible to recompute many identical subtrees from one prediction to the next. Proposition 3. Let KB be a weighted knowledge base, xt an example, and At the set of all atoms that are true in xt . Let A be the set of atoms in At that occur in KB , and A = At − A. Then KB ∪ {¬xji : xt,i = j} = 2|A| ∗ W MC(KB , A) Proof. Based on the completeness of the DP backtracking search procedure for model counting [3], we know that  W MC(KB , A) = {w : (c, w) ∈ KB , r ∩ c = ∅} r⊆A

Let KB  = KB ∪ {¬xji : xt,i = j}. From definition 6, we can infer  KB   = {w : (c, w) ∈ KB , r ∩ c = ∅} r⊆At

= 2|A| W MC(KB , A)

115

F. Koriche / Online Rule Learning via Weighted Model Counting

With these notions in hand, we can now present the algorithm SharpNow. As specified by the translation method, the algorithm starts with an empty knowledge base. During any trial, SharpNow predicts according to rule 7 and updates its knowledge base in light of rule 8. The prediction rule is implemented using the W MC procedure as specified in Proposition 3. The following result claims that SharpNow and Winnow make exactly the same predictions on the same series of labeled examples. As an immediate corollary, the mistake bound of SharpNow is the same as the one derived for the original algorithm. This implies, among others, that the size of the knowledge base maintained by the learner is polynomial in the input dimension. Theorem 4. SharpNow is an exact simulation of kernel Winnow Proof. We consider that Winnow and SharpNow are run with the same parameters η and θ, and the same series of labeled examples {(xt , yt )}. A sufficient condition for establishing the result is to prove that the following equation holds: wt · φ(xt ) = KB t ∪ {¬xji : xt,i = j}

(9)

First of all, consider an assignment r which is not an element of Rn,d . Then, for any possible example xt , r violates at least one clause in the set {¬xji : xt,i = j}. It follows that r is not a model of KB t ∪ {¬xji : xt,i = j}. Thus, to prove 9, we only need to consider assignments that are elements of Rn,d . So, let ri be a rule in Rn,d , KB wt,i the weight of ri maintained  by wt , and wt,i the weight of ri KB according to KB t , i.e. wt,i = {w : (c, w) ∈ KB t , ri ∩ c = ∅}. KB We shall prove by induction on the number of trials that wt,i = wt,i . Consider the first trial. We have w0 = 1 and since KB 0 = ∅, we KB have w0,i = 1. Now consider an arbitrary trial and assume by inducKB tion hypothesis that wt−1,i = wt−1,i at the beginning of the trial. If KB no mistake has occurred during the trial, then wt,i = wt,i trivially holds. Thus, suppose that a mistake has occurred. If φ i (xt ) = 1,  then we know that ri violates the clause {xji : xt,i = j}. So KB yt η KB yt η wt,i = e wt−1,i = e wt−1,i = wt,i . As a similar strategy apKB plies when φi (xt ) = 0. By applying the fact that wt,i = wt,i to each rule in Rn,d , we therefore obtain the desired result.

4

EXPERIMENTS

To provide empirical support for SharpNow, we evaluated it on several learning problems where the target concept is characterized by a small set of rules. A comparison with other standard online mistakedriven algorithms relies on their ability to achieve fast convergence to the optimal hyperplane in the conjunctive feature space Rn,d . The experiments were conducted on a 3.00 GHz Intel Xeon 5160 with 4 GB RAM running Windows XP. All algorithms were written in C++. Notably, the SharpNow algorithm was run using a cache of η size 1 GB, and a learning rate η = 1.278 for which the term 2 sinh η in Theorem 2 is minimized.

4.1

Static Problems

We begin with experiments on several UCI datasets, with no (known) concept drift, aiming at evaluating the performance of SharpNow relative to the kernel Perceptron and kernel Passive-Aggressive (PA) algorithms. Basically, the PA algorithm [7] is a maximum margin variant of Perceptron that forces the learner to achieve a unit margin on the most recent example while remaining as close as possible to the previous hypothesis. Both algorithms were implemented using the kernel prediction rule 3, and the PA algorithm was run with the update rule (PA-I) and a slack variable C fixed to 100.

data set tic-tac-toe kr-vs-kp nothing one pair two pairs three of a kind straight flush full house four of a kind straight flush royal flush Table 1.

Perceptron error ms 0.058 0.02 0.042 0.09 0.032 0.89 0.116 2.01 0.026 0.78 0.004 0.12 0.167 0.87 0.012 0.06 0.070 0.14 0.019 0.03 0.010 0.03 0.003 0.01

PA error 0.073 0.073 0.020 0.113 0.042 0.007 0.158 0.011 0.069 0.029 0.009 0.003

ms 0.02 0.16 0.70 2.09 0.53 0.22 0.70 0.03 0.13 0.03 0.01 0.01

SharpNow error ms 0.009 0.21 0.028 713 0.021 16.5 0 2.44 0 5.17 0 4.41 0.107 119 0.010 0.03 0 5.84 0 2.34 0.007 0.01 0.003 0.01

Results for the tic-tac-toe, kr-vs-kp, and poker hand datasets

Experiments were conducted with the “tic-tac-toe” dataset (958 instances, 9 attributes, 27 atoms), the “kr-vs-kp” dataset (3196 instances, 36 attributes, 73 atoms), and the “poker-hand” dataset (1,025,010 instances, 10 attributes, 85 atoms). The last dataset was divided into 10 binary problems. Each problem consists in finding a particular class of poker hand, where all examples of this class are considered as positive, and all other examples are negative. Each card is described using two attributes (suit and rank); to compare pairs of cards, we introduced four additional binary attributes, same-suit, same-rank, before-rank and next-rank. The total number of atoms is thus 165. For all experiments, the accuracy results have been obtained using only one epoch of the training set. For the tic-tac-toe and kr-vs-kp datasets, we employed a standard 10-fold cross-validation. For the poker-hand dataset, we used the training set of 25,010 instances and a subset of 5,000 test instances in the pool of 1, 000, 000 test instances. The test set was filled with up to 2,500 positives and the rest as negatives, all examples been chosen at random in the pool. Results are reported in Table 1. In term of accuracy, the performance of SharpNow is remarkable. Notably, for many problems in the poker hand dataset, we observed that the algorithm converges using less than 5,000 trials while kernel Perceptron and the kernel Passive-Aggressive show some difficulty in approaching the target concept. The running time is measured in milliseconds per trial. As expected, SharpNow is the slowest because it must solve a Weighted Model Counting problem during each prediction. Yet, with a speed of several milliseconds per trial, SharpNow can be used in real-time for medium-size datasets involving a sparse target function.

4.2

Drifting Problems

A natural application for online learning algorithms is to track concepts that are allowed to change over time. In this setting, we analyze the performance of SharpNow relative to the kernel Perceptron and kernel Forgetron algorithms. The Forgetron [8] is a shifting variant of Perceptron that gradually forgets the oldest supports in the set of examples on which mistakes were made; to this end, it uses a decaying rule that diminishes the contribution of old supports and a fixed memory budget B that removes the oldest support. We conducted experiments with a variant of the Stagger Concepts, a standard benchmark for drifting problems. Each example is a scene involving objects o1 , · · · , ok with attributes color (oi ) ∈ {green, blue, red }, shape(oi ) ∈ {triangle, circle, square}, and size(oi ) ∈ {small , medium, large}. In the original problem [18], each scene involves only one object. To analyze the performance of

116

F. Koriche / Online Rule Learning via Weighted Model Counting

100

100 SharpNow Forgetron (B = 25) Perceptron

80

80

70

70

60

60

50

50

40

40

30

30

20

20

10

10

0

0 0

500

Figure 1.

Time Step (t)

1000

1500

Stagger Concepts with 5 objets

algorithms with more complex scenes including multiple objects, we made 20 series of experiments ranging from k = 1 to k = 20. Each experiment lasts for 1500 trials, and consists of three rule theories that lasts for 500 time step each: (1) {color (ok ) = red ∧ size(ok ) = small }, (2) {color (ok ) = green, shape(ok ) = circle}, and (3) {size(ok ) = medium, size(ok ) = large}. At each trial, the learner is trained on one example, and tested on 250 positive and 250 negative examples generated randomly according to the current concept. The accuracy results obtained with k = 5 and k = 20 are reported in Figures 1 and 2. We observe that SharpNow needs some time to readjust its weights but always converges toward the target concepts; by contrast the kernel Perceptron and kernel Forgetron algorithms show some difficulty in approaching the first and the last concepts. This phenomenon increases with k, revealing that SharpNow is quite robust to irrelevant features. The running time of SharpNow ranges 1 from less than 1000 seconds per trial with k ≤ 5 to 14 seconds per trial for k = 20 (180 atoms and 1028 conjunctive features).

5

SharpNow Forgetron (B = 100) Perceptron

90

Test Error %

Test Error %

90

CONCLUSIONS

We presented SharpNow, an online rule learning algorithm that aims at combining a multiplicative weight update strategy and a weighted model counting method. As an exact simulation of kernel Winnow, the mistake bound of SharpNow is linear in the input dimension. Preliminary experiments on static and drifting problems tend to confirm that SharpNow is particularly efficient for learning small rule theories in presence of many irrelevant conjunctive features. The problem of extending multiplicative weight-update algorithms to expressive concept classes has been a subject of ongoing research in Machine Learning. Yet, to the best of our knowledge, very few investigations have attempted to handle rule theories. A notable exception is the work by Chawla et. al [6] who suggested to explore Markov Chain Monte Carlo (MCMC) methods for approximating the Kernel Winnow Prediction problem. The main difference with our study is that MCMC methods cannot guarantee an exact simulation of kernel Winnow. Moreover, the running time of their resulting algorithm remains quite slow, taking days to learn a DNF formula over 20 variables. By contrast, the weighted model counting technique takes several milliseconds per trial for similar problems. Several avenues of research naturally emerge from this study. One of them concerns the analysis of SharpNow under noisy environments using, for example, a discount factor suggested in kernel functions. An orthogonal direction is to extend our method to multi-class environments by simulating multiplicative voting algorithms [4].

0

500

Figure 2.

Time Step (t)

1000

1500

Stagger Concepts with 20 objets

REFERENCES [1] P. Auer and M. K. Warmuth, ‘Tracking the best disjunction’, Machine Learning, 32(2), 127–150, (1998). [2] R. J. Bayardo and J. D. Pehoushek, ‘Counting models using connected components’, in 17th National Conference on Artificial Intelligence, pp. 157–162, Austin, TX, (2000). [3] E. Birnbaum and E. L. Lozinskii, ‘The good old Davis-Putnam procedure helps counting models’, Journal of Artificial Intelligence Research, 10, 457–477, (1999). [4] N. Cesa-Bianchi and G. Lugosi, Prediction, Learning, And Games, Cambridge University Press, Cambridge, UK, 2006. [5] M. Chavira and A. Darwiche, ‘On probabilistic inference by weighted model counting’, Artificial Intelligence, (2008). To appear. [6] D. Chawla, L. Li, and S. Scott, ‘On approximating weighted sums with exponentially many terms’, Journal of Computer and System Sciences, 69(2), 196–234, (2004). [7] K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer, ‘Online passive-aggressive algorithms’, Journal of Machine Learning Research, 7, 551–585, (2006). [8] O. Dekel, S. Shalev-Shwartz, and Y. Singer, ‘The Forgetron: A kernelbased Perceptron on a fixed budget’, in Advances in Neural Information Processing Systems 18, Vancouver, Canada, (2005). [9] C. Domshlak and J. Hoffmann, ‘Probabilistic planning via heuristic forward search and weighted model counting’, Journal of Artificial Intelligence Research, 30, 565–620, (2007). [10] J. F¨urnkranz, ‘Separate-and-conquer rule learning’, Artifical Intelligence Review, 13(1), 3–54, (1999). [11] R. Khardon, D. Roth, and R. A. Servedio, ‘Efficiency versus convergence of boolean kernels for on-line learning algorithms’, Journal of Artificial Intelligence Research, 24, 341–356, (2005). [12] R. Khardon and R. A. Servedio, ‘Maximum margin algorithms with boolean kernels’, J. of Machine Learning Res., 6, 1405–1429, (2005). [13] N. Littlestone, ‘Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm’, Machine Learning, 2(4), 285–318, (1988). [14] F. Rosenblatt, ‘The Perceptron: a probabilistic model for information storage and organization in the brain’, Psych. Rev., 65, 386–408, (1958). [15] K. Sadohara, ‘Learning of boolean functions using support vector machines.’, in 12th Int. Conference on Algorithmic Learning Theory, pp. 106–118, Washington, DC, (2001). [16] T. Sang, F. Bacchus, P. Beame, H. A. Kautz, and T. Pitassi, ‘Combining component caching and clause learning for effective model counting’, in 7th Int. Conference on Theory and Applications of Satisfiability Testing, Vancouver, BC, Canada, (2004). [17] T. Sang, P. Beame, and H. A. Kautz, ‘Performing Bayesian inference by weighted model counting’, in 20th National Conference on Artificial Intelligence, pp. 475–482, Pittsburgh, PA, (2005). [18] J. C. Schlimmer and R. H. Granger, ‘Beyond incremental processing: Tracking concept drift’, in 5th National Conference on Artificial Intelligence, pp. 502–507, Philadelphia, PA, (1986). [19] M. Thurley, ‘sharpSat - counting models with advanced component caching and implicit BCP’, in 9th Int. Conference on Theory and Applications of Satisfiability Testing, pp. 424–429, Seattle, WA, (2006).

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-117

117

Focused Ensemble Selection: A Diversity-Based Method for Greedy Ensemble Selection Ioannis Partalas, Grigorios Tsoumakas, Ioannis Vlahavas1 Abstract. Ensemble selection deals with the reduction of an ensemble of predictive models in order to improve its efficiency and predictive performance. A number of ensemble selection methods that are based on greedy search of the space of all possible ensemble subsets have recently been proposed. This paper contributes a novel method, based on a new diversity measure that takes into account the strength of the decision of the current ensemble. Experimental comparison of the proposed method, dubbed Focused Ensemble Selection (FES), against state-of-the-art greedy ensemble selection methods shows that it leads to small ensembles with high predictive performance.

1 Introduction Ensemble methods [6] has been a very popular research topic during the last decade. Their success arises from the fact that they offer an appealing solution to several interesting learning problems of the past and the present, such as: improving predictive performance over a single model, scaling inductive algorithms to large databases, learning from multiple physically distributed data sets and learning from concept-drifting data streams. Typically, ensemble methods comprise two phases: the production of multiple predictive models and their combination. Recent work [9, 8, 7, 15, 4, 10, 11, 2], has considered an additional intermediate phase that deals with the reduction of the ensemble size prior to combination. This phase is commonly named ensemble pruning, selective ensemble, ensemble thinning and ensemble selection, the last one of which is used within this paper. Ensemble selection is important for two reasons: efficiency and predictive performance. Having a very large number of models in an ensemble adds a lot of computational overhead. For example, decision tree models may have large memory requirements [9] and lazy learning methods have a considerable computational cost during execution. The minimization of run-time overhead is crucial in certain applications, such as stream mining. Equally important is the second reason, predictive performance. An ensemble may consist not only of high performance models, but also of models with lower predictive performance. Intuitively, combining good and bad models together will not have the expected result. Pruning the low-performing models while maintaining a good diversity of the ensemble is typically considered as a proper recipe for a successful ensemble. The problem of pruning an ensemble of classifiers has been proved to be NP-complete [14]. Exhaustive search for the best subset of classifiers isn’t tractable for ensembles that contain a large number of models. Greedy approaches, such as [2, 4, 9, 10, 11], are fast, as they 1

Department of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece, email: {partalas,greg,vlahavas}@csd.auth.gr

consider a very small part of the space of all combinations. These methods, start with an initial ensemble (empty or full) and search in the space of the different ensembles, by iteratively expanding or contracting the initial ensemble by a single model. The search is guided by either the predictive performance or the diversity of the alternative ensembles. This paper contributes a novel method for greedy ensemble selection, based on a new diversity measure that takes into account the strength of the decision of the current ensemble. Experimental comparison of the proposed method, dubbed Focused Ensemble Selection (FES), against state-of-the-art greedy ensemble selection methods shows that it leads to small ensembles with high predictive performance. The remainder of this paper is structured as follows: Section 2 presents background information on ensemble methods and Section 3 reviews previous work on ensemble selection. Section 4 introduces the proposed method. Section 5 presents the setup of the experimental study and Section 6 discusses the results. Finally, Section 7 concludes this work.

2 Ensemble Methods 2.1 Producing the Models An ensemble can be composed of either homogeneous or heterogeneous models. Homogeneous models derive from different executions of the same learning algorithm by using different values for the parameters of the learning algorithm, injecting randomness into the learning algorithm or through the manipulation of the training instances, the input attributes and the model outputs [6]. Two popular methods for producing homogeneous models are bagging [3] and boosting [13]. Heterogeneous models derive from running different learning algorithms on the same dataset. Such models have different views about the data, as they make different assumptions about them. For example, a neural network is robust to noise in contrast to a k-nearest neighbor classifier.

2.2 Combining the Models A lot of different ideas and methods have been proposed in the past for the combination of classification models. The main motivation underlying this research is the observation that there is no single classifier that performs significantly better in every classification problem [18]. The necessity for high classification performance in some critical domains (e.g. medical, financial, intrusion detection) have

118

I. Partalas et al. / Focused Ensemble Selection: A Diversity-Based Method for Greedy Ensemble Selection

urged researchers to explore methods that combine different classification algorithms in order to overcome the limitations of individual learning paradigms. Unweighted and Weighted Voting are two of the simplest methods for combining not only Heterogeneous but also Homogeneous models. In Voting, each model outputs a class value (or ranking, or probability distribution) and the class with the most votes (or the highest average ranking, or average probability) is the one proposed by the ensemble. In Weighted Voting, the classification models are not treated equally. Each model is associated with a coefficient (weight), usually proportional to its classification accuracy. Let x be an instance and mi , i = 1..k a set of models that output a probability distribution mi (x, cj ) for each class cj , j = 1..n. The output of the (weighted) voting method y(x) for instance x is given by the following mathematical expression: y(x) = arg max c

k X

wi mi (x, cj ),

i=1

where wi is the weight of model i. In the simple case of voting (unweighted), the weights are all equal to one, that is, wi = 1, i = 1..k.

3 Ensemble Selection 3.1 Greedy Approaches Margineantu and Dietterich [9] introduce heuristics to calculate the benefit of adding a classifier to an ensemble, using forward selection in a number of them. These heuristics are based on the diversity and the performance of the classifiers. The authors experiment with boosting ensembles and conclude that pruning can help an ensemble to increase its predictive performance. Fan et al. [7] prune an ensemble of classifiers using forward selection of the classification models, like in [9]. As a heuristic, they use the benefit that is obtained by evaluating the combination of the selected classifiers with the method of voting. Their results show that pruning increases the predictive performance and speeds up the run time of an ensemble of C4.5 decision trees trained on disjoint parts of a large data set. Caruana et al. [4] produce an ensemble of 1000 classifiers using different algorithms and sets of parameters for these algorithms. They subsequently prune the ensemble following an approach that is similar to [9]. This way they manage to achieve very good predictive performance compared to state-of-the-art ensemble methods. Banfield et al. [2], propose a method that selects a subensemble in a backward manner. The authors reward each classifier according to its decision with regard to the ensemble decision. The method removes the classifier with the lowest accumulated reward. Martinez-Munoz et al. [11, 10] present two algorithms for pruning an ensemble of classifiers. In [11] the authors define for each classifier a vector with dimensionality equal to the size of the training set, where each element i corresponds to the decision of the classifier for the instance i. The classifier is added to the ensemble according to its impact in the difference between the vector of the ensemble (average of individual vectors) with a predefined reference vector. This reference vector indicates the desired direction towards which the vector of the ensemble must align. In [10], the authors produce an initial ensemble of bagging models. Then using a forward selection procedure, they add to the ensemble the classifier that disagrees the most with the current ensemble. The process ends when a predefined size for the final pruned ensemble is reached.

3.2 Other Approaches Giacinto and Roli [8] employ Hierarchical Agglomerative Clustering (HAC) for ensemble selection. This way they implicitly used the complete link method for inter-cluster distance computation. Pruning is accomplished by selecting a single representative classifier from each cluster. The representative classifier is the one exhibiting the maximum average distance from all other clusters. Zhou and Tang [20] perform stochastic search in the space of model subsets using a standard genetic algorithm. Standard genetic operations such as mutations and crossovers are used and default values are used for the parameters of the genetic algorithm. The voted performance of the ensemble is used as a function for evaluating the fitness of individuals in the population. Tsoumakas et al. [15] prune an ensemble of heterogeneous classifiers using statistical procedures that determine whether the differences in predictive performance among the classifiers of the ensemble are significant. Only the classifiers with significantly better performance than the rest are retained and subsequently combined with the methods of (weighted) voting. Zhang et al. [19], formulate the ensemble pruning problem as a mathematical problem and apply semi-definite programming (SDP) techniques. Their algorithm requires the number of classifiers to retain as a parameter and runs in polynomial time. Partalas et al. [12], present an ensemble selection method under the framework of Reinforcement Learning, where the learning module finds an optimal policy for including or excluding a classifier from the ensemble.

4 Focused Ensemble Selection Let H = {ht , t = 1, 2, . . . , T } be the set of classifiers (or hypotheses) of an ensemble, where each classifier ht maps an instance x to a class label y, ht (x) = y. Greedy ensemble selection approaches start either with an empty set of classifiers (S = ∅) or the complete ensemble (S = H). For simplicity of presentation we focus on the former initial conditions only, yet our argumentation holds for both. At each step the current subset S is expanded by a model ht ∈ H \ S, based on either the predictive performance [9, S 7, 4] or the diversity [9, 11, 10, 2] of the expanded ensemble S {ht }. Methods that are based on diversity have been shown to be more effective than those that are based on accuracy. The S methods in [10, 2] measure the diversity of candidate ensembles S {ht } by comparing the decision of the current ensemble S with the decision of candidate classifiers ht ∈ H \ S on a set of evaluation examples (xi , yi ), i = 1, 2, . . . , N . Each example consists of a feature vector xi and a class label yi . We can distinguish 4 events concerning both of these decisions: etf

:

y = ht (xi ) ∧ y = S(xi )

ef t

:

y = ht (xi ) ∧ y = S(xi )

ett

:

y = ht (xi ) ∧ y = S(xi )

ef f

:

y = ht (xi ) ∧ y = S(xi )

where S(xi ) is the classification of instance xi by ensemble S. This classification is derived from the application of an ensemble combination method on S, which usually is voting. The diversity measure in [10] is based on etf only, while the one in [2] neglects ef t . We argue that all events should contribute to the calculation of an appropriate diversity measure. Event ef t for example, corresponds to the case where the candidate classifier errs, while

119

I. Partalas et al. / Focused Ensemble Selection: A Diversity-Based Method for Greedy Ensemble Selection

the ensemble is correct. Although, the ensemble is correct, we do not know how many votes lead to its correct decision. If the difference in votes between the correct and wrong decision is marginal, then this candidate classifierSmight lead to a misclassification of example xi by the ensemble S {ht }. The above example concerning event ef t , brings up another disadvantage of the methods in [10, 2]. The decisions of individual models within the current ensemble are not separately considered, as the current ensemble is treated as a whole. We hypothesize that better results can be obtained from a measure that takes into account the strength of the current ensemble’s decision. We argue that an example that is incorrectly (correctly) classified by most of the members of the current ensemble, should not affect strongly the ensemble selection method, as this is probably a very hard (easy) example. On the other hand, examples that are misclassified by about half of the ensemble’s members, are near to change status (correct/incorrect classification) and should strongly affect the method. In order to deal with the above issues, we propose a diversity measure that considers all events and takes into account the strength of the current ensemble’s decision. We define the following quantities: N Ti , which denotes the proportion of models in the current ensemble S that classify example (xi , yi ) correctly, and N Fi = 1 − N Ti , which denotes the number of models in S that classify it incorrectly. The proposed method, dubbed Focused Ensemble Selection (FES), starts with the full ensemble (S = H) and iteratively removes the classifier ht ∈ S that minimizes the following quantity: f es(ht ) =

N “ X i=1

N Ti ∗ I(etf ) − N Fi ∗ I(ef t) +

” +N Fi ∗ I(ett ) − N Ti ∗ I(ef f ) ,

where I(true) = 1 and I(f alse) = 0. Note that events etf and ett increase the metric, because the candidate classifier is correct, while events ef t and ef f decrease it, as the candidate classifier is incorrect. The strength of increase/decrease depends on the strength of the ensemble’s decision. If the current ensemble S is incorrect, then the reward/penalty is multiplied by the proportion of correct models in S. On the other hand, if S is correct, then the reward/penalty is multiplied by the proportion of incorrect models in S. This weighting scheme focuses the attention of the algorithm to examples that are near to change status, while it overlooks examples whose correct classification is either very easy or very hard. In event etf for example, the addition of a correct classifier when the ensemble is wrong contributes a gain of 1 multiplied by the proportion of classifiers in that ensemble that are correct. The rationale is that if the number of classifiers is small, then correct classification of this example is hard to achieve and thus the contribution is penalized, while if the number of classifiers is large, then the correct classification of this example is easier to achieve and thus the contribution is rewarded. An issue that is worth mentioning here concerns the dataset used for calculating the diversity (or predictive performance) measures in greedy ensemble selection methods. One approach is to use the training set for evaluation, as in [11]. This offers the benefit that plenty of data will be available for evaluation and training, but is susceptible to overfitting. Another approach is to withhold a part of the training set for evaluation, as in [4, 2] and the REPwB method in [9]. This is less prone to overfitting, but reduces the amount of data that are available for training and evaluation compared to the previous approach. FES supports both of these approaches.

Another important issue that concerns ensemble selection methods, is when to stop adding classifiers in the ensemble, or, in other words, how many models should the final ensemble include. One solution is to perform the search until all models have been added into (removed from) the ensemble and select the ensemble with the highest accuracy on the evaluation set. This approach has been used in [4]. Others prefer to select a predefined number of models, expressed as a percentage of the original ensemble [9, 7, 11, 2]. FES supports both of these approaches, but follows the former by default, because it is more flexible and automated, since it doesn’t require the specification of a percentage. Algorithm 1 presents the proposed method in pseudocode. Its time complexity is O(T 2 |S|N ), which can be optimized to O(T 2 N ) if the predictions of the current ensemble are updated incrementally each time a classifier is removed from it. Algorithm 1 The proposed method in pseudocode Require: Ensemble of classifiers H 1: S = H 2: B = ∅ 3: acc = 0 4: while S = ∅ do 5: h = arg minf es(ht ) h ∈S

6: S = S \ {h} 7: acctemp = Accuracy(S) 8: if acctemp > acc then 9: acc = acctemp 10: B=S 11: end if 12: end while 13: return B

5 Experimental Setup 5.1 Datasets We experimented on 12 data sets from the UCI Machine Learning repository [1]. Table 1 presents the details of these data sets (Folder in UCI server, number of instances, classes, continuous and discrete attributes, percentage of missing values). We avoided using datasets with less than 650 examples, so that an adequate amount of data is available for training, evaluation and testing. Table 1. Details of data sets: Folder in UCI server, number of instances, classes, continuous and discrete attributes, percentage of missing values id d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12

UCI Folder car cmc credit-g kr-vs-kp hypothyroid segment sick soybean tic-tac-toe vehicle vowel waveform-5000

Inst

Cls

Cnt

Dsc

MV(%)

1728 1473 1000 3196 3772 2310 3772 683 958 946 990 5000

4 3 2 2 4 7 2 19 2 4 11 3

0 2 7 0 7 19 7 0 0 18 3 21

6 7 13 36 23 0 23 35 9 0 10 0

0.00 0.00 0.00 0.00 5.40 0.00 5.40 0.00 0.00 0.00 0.00 0.00

120

I. Partalas et al. / Focused Ensemble Selection: A Diversity-Based Method for Greedy Ensemble Selection

5.2 Methodology The methodology of the experiments proceeds as follows: Initially, the whole dataset is split into three disjunctive parts, a training set, an evaluation set and a test set with 40%, 40% and 20% of the initial dataset respectively. In this paper, we focus on ensembles of heterogeneous models. We therefore run different learning algorithms with different parameters on the training set, in order to produce 200 models that constitute the initial ensemble. The WEKA machine learning library [17] was used as the source of learning algorithms. We trained 24 multilayer perceptrons (MLPs), 60 kNNs, 110 support vector machines (SVMs), 2 naive Bayes classifiers and 4 decision trees. The different parameters used to train the algorithms were the following (the rest of the parameters were left unchanged in their default values): • MLPs: we used 6 values for the nodes in the hidden layer {1, 2, 4, 8, 32, 128} and 4 values for the momentum term {0.0, 0.2, 0.5, 0.8}. • kNNs: we used 20 values for k distributed evenly between 1 and the plurality of the training instances. We also used 3 weighting methods: no-weighting, inverse-weighting and similarityweighting. • SVMs: we used 11 values for the complexity parameter {10−7 , 10−6 , 10−5 , 10−4 , 10−3 , 10−2 , 0.1, 1, 10, 100, 1000}, and 10 different kernels. We used 2 polynomial kernels (of degree 2 and 3) and 8 radial kernels (gamma ∈ {0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 2}). • Naive Bayes: we built one model with default parameters and one with kernel estimation. • Decision trees: we used 2 values for the confidence factor ({0.25, 0.5}), and 2 values for Laplace smoothing ({true, false}). We compare the performance of our approach, Focused Ensemble Selection (FES), against the following greedy ensemble selection methods: Forward Selection (FS) [4], Complementariness (COM) [10], Margin Distance Minimization (MDM) [11] and Concurrency Thining (CT) [2]. The evaluation set is used for the calculation of diversity and performance measure for all competing algorithms, because preliminary experiments have shown that it leads to significantly better results than using the training set in ensembles of heterogeneous models. Voting was used for model combination in FES, FS, COM and CT. Similarly to FES, all rival algorithms follow the approach of [4], which selects the ensemble with the highest accuracy on the evaluation set, instead of using an arbitrary percentage of selection. In addition, the following section discuses comparative results with alternative versions of the algorithms that select a fixed percentage (20%) of models. The resulting ensemble is evaluated on the test set, using voting for model combination. We also calculate the performance of the best single model (BSM) in the ensemble, and the performance of the complete ensemble of 200 models (ALL), using voting for model combination, based on the performance of the models on the evaluation dataset. The whole experiment is performed 10 times for each dataset and the results are averaged.

6 Results and Discussion Table 2 presents the classification accuracy of each algorithm on each dataset. The accuracy of the winning algorithm at each dataset is

highlighted with bold typeface. A first observation is that the proposed approach achieves the best performance in most of the datasets (6), followed by BSM (3), CT (2), MDM and FS (1) and finally COM and ALL (0). Table 2. Classification accuracy for each algorithm on each dataset.

id d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12

FES 98.3 52.7 74.4 99.0 99.3 96.9 98.1 91.5 98.7 81.1 90.3 86.0

FS 98.1 52.4 74.2 99.1 99.2 96.9 98.0 91.1 98.5 80.1 90.5 85.7

COM 98.2 51.2 73.2 99.0 99.2 96.9 98.0 91.0 98.6 80.8 89.8 85.7

CT 98.4 52.3 74.0 99.0 99.3 96.8 98.2 91.6 98.6 80.9 90.3 85.9

MDM 97.4 51.5 73.4 97.9 97.8 96.5 97.4 91.7 98.4 79.1 87.8 84.4

BSM 99.4 42.8 69.5 95.4 90.7 98.5 95.2 90.1 95.8 64.4 98.9 72.7

ALL 82.7 47.6 70.8 95.6 91.9 97.8 95.4 89.8 63.9 75.3 90.7 80.7

According to [5], the appropriate way to compare two or more algorithms on multiple datasets is based on their average rank across all datasets. On each dataset, the algorithm with the highest accuracy gets rank 1.0, the one with the second highest accuracy gets rank 2.0 and so on. In case two or more algorithms tie, they all receive the average of the ranks that correspond to them. Table 3 presents the rank of each algorithm on each dataset, along with the average ranks. The proposed approach has the best average rank (2.17), followed by CT (2.71), FS (3.29), COMP (4.0), MDM (4.92), BSM (5.33) and ALL (5.58). Although the difference of the average ranks between the 2nd best algorithm (CT) and FES is small, CT achieves the highest accuracy (and rank) in only two datasets. We therefore argue that FES should be preferred over CT and the rest of its rivals for ensemble selection. Table 3. Corresponding rank for each algorithm on each dataset.

id d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12

FES 3.0 1.0 1.0 3.0 1.5 4.0 2.0 3.0 1.0 1.0 4.5 1.0

FS 5.0 2.0 2.0 1.0 3.5 4.0 3.5 4.0 4.0 4.0 3.0 3.5

COM 4.0 5.0 5.0 3.0 3.5 4.0 3.5 5.0 2.5 3.0 6.0 3.5

CT 2.0 3.0 3.0 3.0 1.5 6.0 1.0 2.0 2.5 2.0 4.5 2.0

MDM 6.0 4.0 4.0 5.0 5.0 7.0 5.0 1.0 5.0 5.0 7.0 5.0

BSM 1.0 7.0 7.0 7.0 7.0 1.0 7.0 6.0 6.0 7.0 1.0 7.0

ALL 7.0 6.0 6.0 6.0 6.0 2.0 6.0 7.0 7.0 6.0 2.0 6.0

Av. Rank

2.17

3.29

4.0

2.71

4.92

5.33

5.58

We next turn to statistical procedures, in order to investigate whether the performance differences between FES and the rest of the algorithms are significant. According to [5], the appropriate statistical test for the comparison of two algorithms on multiple datasets is the Wilcoxon signed rank test [16]. Note that the majority of past approaches have used the paired t-test, which is inappropriate for this task. We performed 6 tests, one for each paired comparison of FES with each of the other algorithms, at a confidence level of 95%. The test found that FES is significantly better than all other algorithms, apart from CT.

I. Partalas et al. / Focused Ensemble Selection: A Diversity-Based Method for Greedy Ensemble Selection

Table 4 shows the average size of the final ensembles that are selected by the algorithms on each dataset. A general remark is that the number of selected models is small compared to the size of the original ensemble. Only 5.05% to 14.95% of the 200 classifiers are finally selected by the algorithms. Furthermore, the number of models selected based on the maximum accuracy in the evaluation set, is smaller than using a fixed size, such as 20% [10, 11] or 10% [2] of the models, leading to further reduction of the computational cost of the final ensemble. Table 4. Average size of selected ensembles for each algorithm.

id d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12

FES 11.6 18.3 15.7 13.4 11.7 20.5 7.2 23.3 27.9 8.6 8.5 20.8

FS 6.1 15.9 14.4 11.1 5.8 17.2 3.9 13.7 9.4 13.3 8.3 40.6

COMP 5.1 13.4 20.9 11.8 5.0 17.8 3.5 11.0 11.1 10.3 6.8 15.8

CT 6.5 16.7 12.7 11.2 7.2 15.3 3.7 10.9 11.2 10.6 4.4 15.1

MDM 20.9 26.1 33.7 27.9 6.7 29.6 9.3 40.0 31.9 21.9 31.8 79.1

Av. Size

15.6

13.3

11.0

10.5

29.9

In order to investigate whether the performance of greedy ensemble selection algorithms is significantly better when the size of the final ensemble is selected dynamically, rather than using a predefined percentage of models (20%), we performed Wilcoxon tests on the predictive performance of the two alternative versions of each algorithm on all datasets. With 95% confidence the test showed no statistical differences, but the results were in favor of the dynamic approach. Figure 1 presents the mean number of each type of models that are selected by FSD across all datasets for the type of models that are selected. FSD selects on average 7.5 SVMs, 5.2 MLPs, 1.3 kNNs, 1.2 DTs and 0.4 NB models. This shows that SVMs and MLPs, which are traditionally highly accurate classifiers, dominate the final ensembles. On the other hand we notice that the final ensembles include on average 30% of the trained DTs, 22% of the trained MLPs and NB models, 7% of SVMs and 2% of kNNs. This shows that our production procedure led to quite diverse DTs, MLPs and NB models, while on the other hand most of the produced SVMs and kNNs were probably very similar. These results can be taken into account, in order to produce a more diverse initial ensemble. 12

Mean Number of Models

10

8

MLP

Figure 1.

k−NN

SVM

NB

DT

Aggregates for FSD concerning the type of models that are selected.

121

7 Conclusions This paper contributed a new method for greedy ensemble selection, named Focused Ensemble Selection (FES). The main idea of the method is to overlook examples that are either very easy or very hard, and focus on those that are near to change status (correct/incorrect classification). We performed experiments comparing FES with state-of-the-art methods from the related bibliography. Although FES was not found significantly better than all competitors that were considered in this paper, it still was found consistently better based on both the average rank and the number of datasets, where it achieved the highest accuracy. We consider that the main novel idea of this paper (taking into consideration the strength of the ensemble’s classification) is a positive contribution that could be valuable to other researchers working in ensemble selection and ensemble methods in general.

REFERENCES [1] A. Asuncion and D.J. Newman. UCI machine learning repository, 2007. http://www.ics.uci.edu/∼mlearn/MLRepository.html. [2] Robert E. Banfield, Lawrence O. Hall, Kevin W. Bowyer, and W. Philip Kegelmeyer, ‘Ensemble diversity measures and their application to thinning.’, Information Fusion, 6(1), 49–62, (2005). [3] L. Breiman, ‘Bagging Predictors’, Machine Learning, 24(2), 123–40, (1996). [4] R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes, ‘Ensemble selection from libraries of models’, in Proceedings of the 21st International Conference on Machine Learning, (2004). [5] Janez Demsar, ‘Statistical comparisons of classifiers over multiple data sets’, Journal of Machine Learning Research, 7, 1–30, (2006). [6] T. G. Dietterich, ‘Ensemble Methods in Machine Learning’, in Proceedings of the 1st International Workshop in Multiple Classifier Systems, pp. 1–15, (2000). [7] Wei Fan, Fang Chu, Haixun Wang, and Philip S. Yu, ‘Pruning and dynamic scheduling of cost-sensitive ensembles’, in Eighteenth national conference on Artificial intelligence, pp. 146–151. AAAI, (2002). [8] Giorgio Giacinto and Fabio Roli, ‘An approach to the automatic design of multiple classifier systems’, Pattern Recognition Letters, 22(1), 25– 33, (2001). [9] D. Margineantu and T. Dietterich, ‘Pruning adaptive boosting’, in Proceedings of the 14th International Conference on Machine Learning, pp. 211–218, (1997). [10] G. Martinez-Munoz and A. Suarez, ‘Aggregation ordering in bagging’, in International Conference on Artificial Intelligence and Applications (IASTED), pp. 258–263. Acta Press, (2004). [11] G. Martinez-Munoz and A. Suarez, ‘Pruning in ordered bagging ensembles’, in 23rd International Conference in Machine Learning (ICML2006), pp. 609–616. ACM Press, (2006). [12] I. Partalas, G. Tsoumakas, I. Katakis, and I. Vlahavas, ‘Ensemble pruning using reinforcement learning’, in 4th Hellenic Conference on Artificial Intelligence (SETN 2006), pp. 301–310, (May 18–20 2006). [13] Robert E. Schapire, ‘The strength of weak learnability’, Machine Learning, 5, 197–227, (1990). [14] Christino Tamon and Jie Xiang, ‘On the boosting pruning problem’, in 11th European Conference on Machine Learning (ECML 2000), pp. 404–412. Springer-Verlag, (2000). [15] G. Tsoumakas, L. Angelis, and I. Vlahavas, ‘Selective fusion of heterogeneous classifiers’, Intelligent Data Analysis, 9(6), 511–525, (2005). [16] F. Wilcoxon, ‘Individual comparisons by ranking methods’, Biometrics, 1, 80–83, (1945). [17] I.H. Witten and E. Frank, Data Mining: Practical machine learning tools and techniques, Morgan Kaufmann, 2005. [18] D. Wolpert, The mathematics of generalization, Addison-Wesley, 1995. [19] Yi Zhang, Samuel Burer, and W. Nick Street, ‘Ensemble pruning via semi-definite programming’, Journal of Machine Learning Research, 7, 1315–1338, (2006). [20] Zhi-Hua Zhou and Wei Tang, ‘Selective ensemble of decision trees’, in 9th International Conference on Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing, RSFDGrC 2003, pp. 476–483, Chongqing, China, (May 2003).

122

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-122

MTForest: Ensemble Decision Trees based on Multi-Task Learning Qing Wang and Liang Zhang and Mingmin Chi and Jiankui Guo 1 Abstract. Many ensemble methods, such as Bagging, Boosting, Random Forest, etc, have been proposed and widely used in real world applications. Some of them are better than others on noisefree data while some of them are better than others on noisy data. But in reality, ensemble methods that can consistently gain good performance in situations with or without noise are more desirable. In this paper, we propose a new method namely MTForest, to ensemble decision tree learning algorihms by enumerating each input attribute as extra task to introduce different additional inductive bias to generate diverse yet accurate component decision tree learning algorithms in the ensemble. The experimental results show that in situations without classification noise, MTForest is comparable to Boosting and Random Forest and significantly better than Bagging, while in situations with classification noise, MTForest is significantly better than Boosting and Random Forest and is slightly better than Bagging. So MTForest is a good choice for ensemble decision tree learning algorithms in situations with or without noise. We conduct the experiments on the basis of 36 widely used UCI data sets that cover a wide range of domains and data characteristics and run all the algorithms within the Weka platform.

1

Introduction

Decision-tree is one of the most successful and widely used learning algorithms, due to its various attractive features: simplicity, comprehensibility, no parameters, and being able to handle mixed-type data. The most widely used decision tree learning algorithm is C4.5 [1] which recently had been ranked 1st in the ”top10 algorithms in data mining” [16]. Ensemble methods train a collection of learners and then combine their predictions to make final decision. Since the generalization ability of an ensemble could be significantly better than that of a single learner, so studying the methods for constructing good ensembles has become one of the most active research areas in supervised learning [8]. And a lot of ensemble methods to improve the generalization ability of decision tree learning algorithms have been proposed and widely used in real word applications. Typically, an ensemble is built in two steps, that is, generating multiple component learners and then combining their predictions. According to the styles of training the component learners, current ensemble learning algorithms can be roughly categorized into two classes, that is, algorithms where component learners must be trained sequentially and algorithms where component learners could be trained in parallel [9]. The representative of the first category is Boosting [4], which sequentially generates a series of component 1

Department of Computer and Information Technology, Fudan University, Shanghai, China. Email:{wangqing,lzhang,mmchi,gjk}@fudan.edu.cn

learners and iteratively increases the weights on the instances most recently be misclassified by the former component learner. The representative of the second category is Bagging [2] which independently generates many samples from the original training set via bootstrap sampling and then trains the component learners from each of these samples. Other representatives of this category include Random Forest [3], Randomized C4.5 [5], Random Subspace [6], etc. Many ensemble methods for decision trees have been proposed and widely used in real world applications. Some of them are better than others on noise-free data while some of them are better than others on noisy data, such as Boosting and Random Forest are better than Bagging in situations without noise, while Bagging is more robust to noise and is better than Boosting and Random Forest in situations with noise [7]. But in reality, due to time and cost reason, ensemble methods that can consistently gain good performance in situations with or without noise is more desirable. In this paper, we propose a new way to ensemble decision tree based on multi-task learning which generates diverse but accurate component decision tree learners in the ensemble through using different input attribute as extra task to introduce different inductive bias to the decision tree learning process. The resulting forest can achieve better performance on both noise-free and noisy data and have the following desirable characteristics: 1. Its accuracy is as good as Random Forest and Boosting and can achieve significantly improvement over Bagging in situations without noise. 2. Its accuracy is slightly better than Bagging and significantly better than Random Forest and Boosting in situations with an amount of noise. 3. It is simple and easy to parallelize. The rest of this paper is organized as follows. In Section 2, we introduce the related works for ensemble decision tree learning algorithms. In Section 3, we introduce our ensemble method for decision tree learning. In Section 4, we describe the experimental setup and results in detail. Finally, we make a conclusion and outline our main directions for further research.

2 Related works Bagging [2] is one of the older, simpler, and better known methods for creating an ensemble of classifiers which independently generates many samples from the original training set via bootstrap sampling and then trains a component learner from each of these samples. The Bagging algorithm has achieved great success in building ensembles of decision trees, neural networks and other unstable learning algorithms. Boosting [4] sequentially generate component classifiers by

Q. Wang et al. / MTForest: Ensemble Decision Trees Based on Multi-Task Learning

iteratively increases the weights on the instances most recently be misclassified and have gain great success on both stable and unstable learning algorithms. Both Bagging and Boosting are methods that generate a diverse ensemble of classifiers by manipulating the training data. Ho’s random subspace technique [6] selects random subsets of the available features to be used in training the individual decision trees in an ensemble. Ho’s approach randomly selects one half of the available features for each decision tree and creates ensembles of size 100. Ho summarized the results as follows: The subspace method is better in some cases, about the same or worse in other cases when compared to the other two forest building techniques Bagging and Boosting [6]. One other conclusion was that the subspace method is best when the data set has a large number of features and samples, and that it is not good when the data set has very few features coupled with a very small number of samples or a large number of classes [6]. Dietterich introduced an approach termed randomized C4.5 [5] to ensemble C4.5 learning algorithm. In this approach, at each node in the decision tree, the 20 best splits are determined and one of them is randomly selected for use at that node other than select the best split. For continuous attributes, it is possible that multiple tests from the same attribute will be in the top 20. Through experiments with 33 data sets from the UCI repository, it was found that randomized C4.5 can gain substantial improvement over bagging but is not comparable to Boosting on noise-free data, while randomized C4.5 is more robust than Boosting on noisy data. Breiman’s Random Forest [3] technique incorporates elements of random subspaces and bagging and is specific to using decision trees as the base classifier. At each node in the tree, a subset of the available features is randomly selected and the best split available within this subset is selected for split. Also, bagging is used to create the training set of data items for each individual tree. The number of features randomly chosen (from n total) at each node is a parameter of this approach. Through experiments with 16 data sets from the UCI repository and 4 synthetic data sets, it was found that Random Forest is comparable to Boosting and sometimes better on noise-free data and is more robust than Boosting on noisy data. Empirical study [5, 7] on these ensemble methods for decision tree learning have shown that Boosting and Random Forest are the best ensemble methods for decision tree in situation without noise, while Bagging is best ensemble methods in situations with noise. So in this paper, we use Bagging, Boosting and Random Forest as benchmark ensemble methods to compare with our method.

3

Ensemble decision trees based on multi-task learning

Multi-Task Learning (MTL) [11] trains multiple tasks simultaneously while using a shared representation and has been the focus of much interest in the machine learning community over the last decade. It has been empirically [11, 15] as well as theoretically [13, 15] shown to often significantly improve performance relative to learning each task independently. When the training signals are for tasks other than the main task, from the point of view of the main task, the other tasks are serving as a bias [11]. This multi-task bias causes the learner to prefer hypotheses that explain more than one task, i.e. it must be biased to prefer hypotheses that have utility across multiple tasks. Because in multi-task learning extra task is serve as additional inductive bias, we can use different extra task to bias each component learner in the ensemble to generate different component learn-

123

ers [11, 14]. The multi-task learning theory guarantees that the component learner will be with high accuracy if the extra task is related to the main task and the component learner will be with high diversity if the each extra task represents different bias. But in most learning environments, we are only given the training data which is composed of a vector of input attributes {A1 , A2 , ..., An } and the class variable C and we do not have any other extra related tasks information. In [12], it has shown that some of the attributes that attribute selection process discards can beneficially be used as extra outputs for inductive bias transfer. So in our method, we treat each input attribute as extra task to bias each component decision tree in the ensemble. It obvious that we can generate a good ensemble in which component learners could be with high accuracy as well as high diversity given that each attribute (task) highly correlated with the class attribute and not highly correlated with each other. So it does better if we using a feature selection step to choose these attributes subset as extra tasks, but to make the algorithm simple and easy to implement, in this paper, we simply use each attribute in the input as an extra task to bias each component decision tree learning algorithm in the ensemble.

3.1 The MTForest algorithm Our ensemble method is described in Algorithm 1. In MTForest, we generate different component decision trees by use different input attribute as extra task together with the main classification task. We call this two-task decision tree algorithm below. The two-task decision tree learning process is similar to standard C4.5 decision tree learning algorithm except that the Information Gain and Gain Ratio criteria of each split Si is calculated by combine the main classification task and the extra task, showing below. MTIG(Si ) = MainTaskIG(Si ) + weight ∗ ExtraTaskIG(Si ) (1) MTGR(Si ) = MainTaskGR(Si )+weight∗ExtraTaskGR(Si ) (2) The parameter weight is served as an trade-off parameter between the classification accuracy and the diversity of each component two-task decision trees. In our experiments, we set the value of weight as 2. To further enhance the diversity among the component decision trees in the ensemble, we grow each two-task decision tree to maximum size and do not prune and incorporate our algorithm with Bagging, i.e. constructing each two-task decision tree on a new training set using bootstrap sampling from original training set. Our ensemble method enumerates each attribute in the input as an extra task to bias each component decision tree learning algorithm in the ensemble, so its ensemble size is equal to the number of attribute in the input which is different from most of other ensemble methods such as Bagging, Boosting, Random Forest that need to specify the ensemble size. Also the building process of each two-task decision tree learning algorithm do not depend on each other, so MTForest can easily be parallelized. For numeric attributes, in our implementation, we process in the following way. When a numeric attribute be choose as extra task, we first discretize this attribute by k-bin discretization where k =10, then in selecting the splitting attribute, this numeric attributes (task) are treated the same as non-numeric class attributes.

4 Experimental methodology and results In this section, we describe the experimental methodology, the data sets, and the obtained results.

124

Q. Wang et al. / MTForest: Ensemble Decision Trees Based on Multi-Task Learning

Algorithm 1 The MTForest Algorithm Input: Training instances set D, k, weight Output: A collection of two-task decision tree classifiers S S={}; for each attribute Ai in the input If Ai is numeric discretize attribute Ai by k-bin discretization; Di = bootstrap sampling from D with the same size; Using Ai as extra task together with the main classification task C to create an unpruned two-task decision tree Ti using Eq.1 and Eq.2 on the training instances set Di ; S=S Ti ; return S

4.1

Methodology

We conduct experiments under the framework of Weka[18]. For the purpose of our study, we use the 36 well-recognized data sets from the UCI repositories [17] which represent a wide range of domains and data characteristics. There is a brief description of these data sets in Table 1. We adopted the following three steps to preprocess data sets. 1. First, missing values in each data set are filled in using the unsupervised filter ReplaceMissingValues in Weka; 2. Second, numeric attributes are discretized using the unsupervised filter Discretize in Weka; 3. It is well known that, if the number of values of an attribute is almost equal to the number of instances in a data set, this attribute does not provide any information to the class. So,we use the unsupervised filter Remove in Weka to delete attribute does not provide any information to the class. Two occurred within the 36 data sets, namely Hospital Number in data set colic.ORIG and Animal in data set zoo. In our experiments, we compare MTForest to Bagging C4.5, Boosting C4.5 and Random Forest in terms of classification accuracy in both noise-free and noisy situations. We use the implementation of C4.5 (weka.classifiers.trees.J48), Random Forest (weka.classifiers.trees.RandomForest), Bagging (weka.classifiers. meta.Bagging) and Boosting (weka.classifiers.meta.AdaBoostM1) in Weka, and implement our algorithms under the framework of Weka. For Bagging and Boosting, we set the ensemble size as 50; while for Random Forest we set the ensemble size as 100. We done this for two reasons, first because it is large enough to ensure convergence of the ensemble effect with most of our data sets, second it is the same ensemble sizes used in [3]. To Random Forest, an important parameter is the number of features randomly selected at each node; in our experiments we use the default value because it can achieve best results in most case [7]. For our methods, the ensemble size is the number of input attributes which is often far smaller than 50 except on two data set(audiology, sonar) and we set the value of parameter weight as 2. In all experiments, the classification accuracy of each algorithm on a data set was obtained via 10 runs of ten-fold cross validation. Runs with the various algorithms were carried out on the same training sets and evaluated on the same test sets. To compare two ensemble algorithms across all domains, we employ the statistics used in [5], namely the win/draw/loss record. The win/draw/loss record presents three values, the number of data sets for which algorithm A obtained better, equal, or worse performance than algorithm B with respect to classification accuracy. We report the statistically signifi-

Table 1.

Description of the data sets used in the experiments.

Datasets anneal anneal.ORIG audiology autos balance-scale breast-cancer breast-w car colic colic.ORIG credit-a credit-g diabetes glass heart-c heart-h heart-statlog hepatitis hypothyroid ionosphere iris kr-vs-kp labor letter lymph mushroom primary-tumor segment sick sonar soybean tic-tac-toe vehicle vote yeast zoo

Size 898 898 226 205 625 286 699 1728 368 368 690 1000 768 214 303 294 270 155 3772 351 150 3196 57 20000 148 8124 339 2310 3772 208 683 958 846 435 1484 101

Attribute 39 39 70 26 5 10 10 7 23 28 16 21 9 10 14 14 14 20 30 35 5 37 17 17 19 23 18 20 30 61 36 10 19 17 10 18

Classes 6 6 24 7 3 2 2 4 2 2 2 2 2 7 5 5 2 2 4 2 3 2 2 26 4 2 21 7 2 2 19 2 4 2 10 7

Missing Y Y Y Y N Y Y N Y Y Y N N N Y Y N Y Y N N N Y N N Y Y N Y N Y N N Y N N

Numeric Y Y N Y Y N N N Y Y Y Y Y Y Y Y Y Y Y Y Y N Y Y Y N N Y Y Y N N Y N Y Y

cant win/draw/loss record; where a win or loss is only counted if the difference in values is determined to be significant at the 95% level by a paired t-test.

4.2 Results Table 2 shows the comparison results of two-tailed t-test with a 95% confidence level between each pair of algorithms, in which each entry w/t/l means that the algorithm at the corresponding row wins in w data sets, ties in t data sets, and loses in l data sets, compared to the algorithm at the corresponding column. Table 4 shows the detailed experimental results of the mean classification accuracy and standard deviation of each algorithm on each data set, and the average values are summarized at the bottom of the table. From Table 2 and 4 we can see that MTForest can achieve substantial improvement over C4.5 on most data set (13 wins and 2 losses) which suggest that MTForest is potentially a good ensemble technique for decision tree. MTForest can also gain significantly improvement over Bagging (7 wins and 2 losses) and is comparable to two state-of-the-art ensemble technique for decision trees, Boosting (8 wins and 8 losses) and RandomForest (3 wins and 4 losses). An interesting phenomenon on iris data set is that, MTForest is the only ensemble method which can gain improvement over the C4.5 while other three ensemble methods used to compare all decrease the accuracy over C4.5. An important issue of an ensemble method is the question of how well it performs in situations when there is a large amount of classification noise, i.e., training examples with incorrect class labels. Since

Q. Wang et al. / MTForest: Ensemble Decision Trees Based on Multi-Task Learning

Table 2. Summary of experimental results on noise-free data with two-tailed t-test with 95% confidence level. Each cell contains the number of wins, ties and losses between the algorithm in that row and the algorithm in that column. w/t/l Bagging Boosting Random Forest MTForest

C4.5 11/24/1 16/17/3 15/19/2 13/21/2

Bagging

Boosting

Random Forest

12/17/7 9/22/5 7/27/2

6/25/5 8/20/8

3/29/4

Table 3. Summary of experimental results on noisy data with two-tailed t-test with 95% confidence level. Each cell contains the number of wins, ties and losses between the algorithm in that row and the algorithm in that column. w/t/l Bagging Boosting Random Forest MTForest

C4.5 15/21/0 10/14/12 13/17/6 18/14/4

Bagging

Boosting

Random Forest

3/13/20 6/19/11 5/27/4

15/21/0 20/15/1

9/23/4

some noise in the outputs is often present, robustness with respect to noise is a desirable property. Following Breiman [3], the following experiment was done which changed about one in twenty class labels (i.e., injecting 5% noise). For each data set in the experiment, we randomly split off 10% of the data set as a test set, and runs are made on the remaining training set. The noisy version of the training set is gotten by changing, at random, 5% of the class labels into an alternate class label chosen uniformly from the other labels. We repeat this process 100 times to compute the classification accuracy of each algorithm in this noisy situation. Table 3 shows the comparison results of two-tailed t-test with a 95% confidence level between each pair of algorithms in this noisy situations, in which each entry w/t/l has the same meanings as in Table 2. Table 5 shows the detailed experimental results of the mean classification accuracy and standard deviation of each algorithm on each data set in this noisy situation, and the average values are summarized at the bottom of the table. From Table 3 and 5 we can see that MTForest can achieve substantial improvement over C4.5 on most data set (18 wins and 4 losses) which suggest that MTForest is potentially a good ensemble technique for decision tree in this noisy situations. And MTForest can significantly outperform Boosting (20 wins and 1 losses) and Random Forest (9 wins and 1 losses) and is slightly better than Bagging in this noisy situation (5 wins and 4 losses). From Table 5, we can also see that MTForest has the best average value (83.30) over all the data sets used.

5

Conclusion and Future Work

We address the problem of ensemble decision trees learning algorithms that can consistently gain good performance in situations with or without noise. Previous study has shown that Bagging can always improve the classification performance of decision tree learning algorithms on both noise-free and noisy data, but its performance on noise-free data is not comparable to Boosting and Random Forest. In this paper, we propose a new ensemble method for decision tree learning algorithms by enumerating each input attribute as extra task together with the main classification task to generate different component decision trees in the ensemble. The experimental results show that the performance of our algorithm can comparable to Boosting and Random Forest on noise-free data and as good as Bagging on noisy data. Dietterich [8] indicated that roughly there are four ensemble

125

schemes, that is, perturbing the training set, perturbing the input attributes, perturbing the output representation, and injecting randomness to the learning algorithm. The success of MTForest suggests that we can also inject different additional inductive bias to the learning algorithm to create an ensemble of classifiers. It will be interesting to explore whether or not we can using selective ensemble technique [10] to select a subset of the two-task decision trees created to improve the performance; exploiting task relatedness to assign different weight to each component decision tree classifiers in the ensemble; extending multi-task ensemble technique to ensemble stable classifier (such as Naive Bayes, KNN) where bagging can not work well. These have been left to be investigated in the future.

ACKNOWLEDGEMENTS This research is partially supported by the National Key Basic Research Program (973) of China under grant No.2005CB321905. We thank the anonymous reviewers for their great helpful comments.

REFERENCES [1] J.Quinlan, C4.5:Programs for Machine Learning, Morgan Kaufmann, (1993). [2] L.Breiman, Bagging Predictors, Machine Learning, 24, pp.123-140, (1996). [3] L. Breiman, Random Forests, Machine Learning, 45, pp.5-32, (2001). [4] R.E. Schapire, A Brief Introduction to Boosting, In Proc.16th International Joint Conference on Artificial Intelligence, pp.1401-06, (1996). [5] T.G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Machine Learning, 40, 139-157, (2000). [6] T.Ho, The Random Subspace Method for Constructing Decision Forests, IEEE Trans. Pattern Analysis and Machine Intelligence, 20, 832-844, (1998). [7] R.E. Banfield, L.O. Hall, K. W. Bowyer, and W.P. Kegelmeyer, A Comparison of Decision Tree Ensemble Creation Techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 173-180, (2007). [8] T.G. Dietterich. Ensemble learning. In: The Handbook of Brain Theory and Neural Net-works, 2nd edition, M.A. Arbib, Ed. Cambridge, MA: MIT Press, (2002). [9] Z.H. Zhou and Y. Yu, Adapt Bagging to Nearest Neighbor Classifiers, Journal of Computer Science and Technology, 20, 48-54, (2005). [10] Z.H. Zhou, J.X. Wu, W. Tang. Ensembling neural networks: Many could be better than all. Artificial Intelligence, 137, pp.239-263 (2002). [11] R.Caruana,Multi-Task Learning, Machine Learning, 28, pp.41-75, (1997). [12] R.Caruana, Virginia R. Benefitting from the Variables that Variable Selection Discards. Journal of Machine Learning Research, 3, pp.124564, (2003). [13] J. Baxter, A model for inductive bias learning, Journal of Artificial Intelligence Research, 12, pp.149-198, (2000). [14] Qiang Ye and P.W. Munro. Improving a Neural Network Classifier Ensemble with Multi-task Learning, In Proc International Joint Conference on Neural Networks, (2006). [15] R.K. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal Machine Learning Research, 6, pp.1817-1853, (2005) [16] Xindong Wu, Vipin Kumar, Ross, Joydeep Ghosh, Qiang Yang, Hiroshi Motoda, Geoffrey Mclachlan, Angus Ng, Bing Liu, Philip Yu, Zhi-Hua Zhou, Michael Steinbach, David Hand, Dan Steinberg, Top 10 algorithms in data mining, Knowledge and Information Systems, 14, pp. 1-37, (2008). [17] Blake. C., Merz. C. J. UCI repository of machine learning databases. In Department of ICS, University of California, Irvine.http://www.ics.uci.edu/ mlearn/MLRepository.html. [18] Witten, I. H., Frank, Data Mining:Practical Machine Learning Tools and Technology with Java Implementation, Morgan Kaufmann, (2000).

126

Q. Wang et al. / MTForest: Ensemble Decision Trees Based on Multi-Task Learning

Table 4. The detailed experimental results of the classification accuracy and standard deviation on data without additionally introduced noise.

Table 5. The detailed experimental results of the classification accuracy and standard deviation on data with 5% randomly introduced noise.

Data Set C4.5 Bagging Boosting Random Forest MTForest Datasets C4.5 Bagging Boosting Random Forest MTForest anneal 98.61±0.98 98.70±0.92 95.71±2.25 98.01±1.32 98.51±1.18 anneal 98.65±0.97 98.68±0.92 99.55±0.68 99.35±0.79 99.23±0.80 anneal.ORIG 90.36±2.51 91.86±2.48 92.32±2.16 91.67±2.37 92.65±2.41 anneal.ORIG 90.28±2.74 91.48±2.69 90.27±2.73 90.06±2.52 91.54±2.58 audiology 76.61±8.13 80.37±7.39 80.63±8.41 78.03±7.66 81.21±7.73 audiology 77.22±7.69 80.97±7.50 84.82±7.13 79.97±6.85 82.13±6.98 autos 76.01±10.12 83.42±8.32 81.36±8.55 82.30±8.57 80.35±8.54 autos 81.54±8.32 85.47±6.81 87.69±7.55 84.97±7.68 83.11±8.03 balance-scale 64.14±4.16 75.09±4.94 71.89±4.32 78.47±3.85 79.85±3.92 balance-scale 65.95±4.85 74.35±5.50 68.83±4.87 77.16±4.31 78.92±4.16 breast-cancer 75.26±5.04 73.76±5.85 66.04±8.21 70.07±7.36 69.49±6.96 breast-cancer 74.08±5.64 73.23±6.42 65.52±8.05 68.13±7.64 68.44±7.48 breast-w 92.86±3.49 94.79±2.94 93.81±3.09 95.72±2.53 94.99±2.68 breast-w 94.01±3.28 95.44±2.71 96.70±2.08 96.34±2.44 95.67±2.49 car 91.51±1.95 92.83±1.86 93.69±1.82 94.03±1.62 92.85±2.00 car 92.22±2.01 93.59±1.80 96.72±1.50 94.70±1.66 92.37±1.95 colic 84.15±5.89 84.62±6.00 79.18±7.23 83.58±5.98 84.65±5.65 colic 84.31±6.02 84.83±5.81 82.56±5.85 84.37±5.47 85.65±5.36 colic.ORIG 66.23±1.40 66.31±1.23 66.28±1.23 72.12±4.72 67.90±3.31 colic.ORIG 71.76±5.63 68.08±3.53 71.67±5.45 72.58±5.00 68.09±3.43 credit-a 84.85±4.14 85.77±4.14 79.97±4.18 84.00±4.52 84.51±4.14 credit-a 85.06±4.12 85.71±3.91 82.99±4.15 85.01±3.81 85.83±4.00 credit-g 72.16±3.41 73.79±3.96 72.18±3.46 74.81±3.49 73.64±2.97 credit-g 72.61±3.49 74.04±4.03 73.64±3.15 75.53±3.21 74.03±2.86 diabetes 74.01±4.87 74.37±4.66 70.62±5.13 72.64±4.77 71.74±4.31 diabetes 73.89±4.70 73.92±4.53 72.07±4.67 73.15±3.96 72.57±4.69 glass 55.60±8.89 59.29±8.81 56.50±8.64 59.63±8.61 60.08±8.84 glass 58.14±8.48 59.90±9.33 56.51±9.50 60.55±8.94 61.56±8.98 heart-c 78.18±6.65 80.07±6.28 77.20±6.97 78.78±7.06 78.10±6.65 heart-c 79.14±6.44 79.98±6.66 78.15±7.29 78.78±7.12 79.48±6.94 heart-h 80.23±7.69 80.53±7.19 78.01±6.90 79.87±6.20 78.46±7.03 heart-h 80.10±7.11 80.97±6.92 79.34±7.14 80.10±6.03 79.24±6.90 heart-statlog 77.59±7.13 79.22±7.00 76.67±7.23 78.37±7.00 77.70±7.50 heart-statlog 79.78±7.71 79.74±6.89 78.26±7.34 79.25±6.45 79.59±7.01 hepatitis 81.24±9.50 81.77±8.04 82.02±8.57 82.00±7.34 82.60±8.08 hepatitis 81.12±8.42 81.83±7.64 83.53±8.77 82.14±6.51 82.39±8.34 hypothyroid 93.23±0.44 93.25±0.44 91.95±0.85 91.69±0.90 92.91±0.67 hypothyroid 93.24±0.44 93.26±0.44 92.26±0.94 92.58±0.78 93.14±0.65 ionosphere 87.61±4.92 89.52±4.51 90.71±5.04 90.89±4.51 91.88±4.20 ionosphere 87.47±5.17 89.40±4.69 92.22±4.53 90.86±4.69 91.45±4.40 iris 95.27±4.77 95.33±5.23 91.93±6.31 93.40±6.80 94.87±5.09 iris 95.99±4.64 95.67±5.05 94.53±6.24 95.27±5.04 96.27±4.28 kr-vs-kp 99.25±0.46 99.30±0.40 94.78±1.32 98.28±0.69 99.21±0.45 kr-vs-kp 99.44±0.37 99.46±0.37 99.60±0.31 99.27±0.44 99.46±0.36 labor 83.80±14.6285.60±13.2689.27±12.88 88.89±12.77 89.30±12.37 labor 84.97±14.2484.99±14.0686.30±14.78 89.76±12.12 89.47±12.53 letter 80.56±0.83 83.74±0.87 85.77±0.89 87.49±0.76 89.81±0.62 letter 81.31±0.78 84.24±0.83 89.89±0.78 89.78±0.68 90.81±0.61 lymph 77.84±9.62 78.79±8.82 81.76±9.55 82.82±9.26 79.72±10.26 lymph 78.21±9.74 79.70±9.61 83.26±8.85 83.04±9.49 80.31±9.60 mushroom 99.99±0.01 99.99±0.01 96.31±0.66 99.95±0.07 99.87±0.12 mushroom 100±0.00 100±0.00 100±0.00 100±0.00 100±0.00 primary-tumor 41.01±6.59 45.19±6.16 42.63±6.61 41.29±6.05 43.66±6.77 primary-tumor 41.89±6.88 44.14±5.81 41.39±6.81 40.97±6.33 42.71±6.13 segment 92.92±1.71 93.90±1.56 92.65±1.80 94.36±1.65 94.83±1.46 segment 93.42±1.67 94.03±1.47 95.24±1.37 96.07±1.23 95.32±1.27 sick 98.04±0.78 98.11±0.77 96.75±0.98 97.22±0.77 97.99±0.77 sick 98.16±0.68 98.25±0.68 98.15±0.73 98.22±0.65 98.28±0.74 sonar 71.32±9.79 74.09±9.30 75.16±9.05 77.31±8.75 77.09±9.01 sonar 71.09±8.40 74.08±8.95 79.21±9.02 78.58±9.06 78.59±9.14 soybean 92.19±2.97 94.04±2.76 89.88±3.26 92.24±2.67 93.45±3.05 soybean 92.63±2.72 94.05±2.61 94.04±2.61 93.80±2.72 93.86±2.82 tic-tac-toe 83.81±3.55 92.79±2.47 95.53±2.10 94.80±2.23 93.20±2.57 tic-tac-toe 85.57±3.21 94.73±2.04 98.92±1.03 97.05±1.76 95.88±1.89 vehicle 68.56±4.41 71.59±3.88 70.54±3.81 71.49±3.76 72.32±3.32 vehicle 70.74±3.62 72.10±3.82 72.55±4.02 72.63±3.56 73.05±3.76 vote 95.60±3.10 96.11±2.81 93.18±3.80 95.15±3.18 96.15±2.86 vote 96.27±2.79 96.27±2.67 94.80±3.05 96.18±2.85 96.25±2.58 yeast 51.72±3.45 53.39±3.85 51.87±3.78 51.18±3.42 52.44±3.60 yeast 52.56±3.44 54.35±3.89 52.93±3.74 52.43±3.53 53.25±3.52 zoo 92.39±6.71 93.30±6.97 81.85±16.47 93.16±6.69 95.17±6.03 zoo 92.61±7.33 93.20±7.37 97.34±5.75 94.65±6.03 94.48±6.64 mean 81.28±4.90 83.11±4.67 81.10±5.24 83.07±4.75 83.30±4.65 mean 82.06±4.77 83.52±4.64 83.84±4.76 84.12±4.45 84.07±4.54

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-127

127

Many-Valued Concept Lattices for Conceptual Clustering and Information Retrieval Nizar Messai and Marie-Dominique Devignes and Amedeo Napoli and Malika Smail-Tabbone1

1

LORIA - INRIA Nancy Grand Est, 615, rue du Jardin Botanique 54600 Villers-l`es-Nancy, FRANCE, email: {messai, devignes, napoli, malika}@loria.fr

2

Background and related work

2.1

Formal Concept Analysis

In this section we recall basic definitions related to FCA. More detailed definitions and results can be found in [6]. FCA starts from a given input data represented as a formal context to provide the set of all formal concepts which form a concept lattice. A formal context is denoted by K = (G, M, I) where G is a set of objects, M is a set of attributes, and I is a binary relation between G and M (I ⊆ G × M ). (g, m) ∈ I denotes the fact that object g ∈ G is in relation through I with attribute m ∈ M (also read as g has m). Table 1 shows an example of formal context. The objects are planets of the Solar System Table 1. An example of formal context

×

×

× ×

100
×

DS<100

×

× ×

Satellite

×

× ×

DS>200

Mercury Venus Earth Mars

Distance to Sun (106 km)

M>5

PP Attr. P Obj. PP P

Mass (1021 t) 4<M<5

Diameter (103 km)

M<1

Formal concept analysis (FCA) [6] is a data analysis method allowing to derive implicit relationships from a set of objects described by their attributes. In the basic setting, data to be analyzed must be represented as a formal context which has the form of a binary table with rows corresponding to objects and columns corresponding to attributes. A table entry does or does not contain a cross depending on whether an object does or does not have an attribute. Based on attribute sharing between objects, the data are structured into units called formal concepts. These concepts are partially ordered and form a special hierarchy of concepts called a concept lattice. A concept lattice is an equivalent representation of data in a formal context which emphasizes the relationships between objects, attributes, and formal concepts, and provides a suitable support for navigation into the data set it represents. This particular kind of conceptual clustering was the main motivation behind the successful use of FCA in a wide range of application fields [5]. Whereas the basic FCA setting requires a particular representation of data (i.e. a formal context), real-world data sets are often complex and heterogeneous. Their representation in terms of a binary table does not produce a formal context. It rather produces a many-valued (MV) context where table entries (i) are empty when there is no relation between the corresponding objects and attributes, (ii) contain arbitrary values taken by the attributes for the objects, or (iii) truth degrees depending on the link between an object and an attribute. For using FCA, any operation on data with such representation must be preceded by a transformation of MV contexts into binary contexts using an appropriate conceptual scaling [6]. In this paper we present an approach for analyzing data sets represented as MV contexts. First, we define an MV Galois connection based on similarity between attribute values to compute MV concepts and MV concept lattices. Then, we investigate the usability of

D>20

Introduction

10
1

the approach for various data analysis tasks such as classification and information retrieval. The paper is organized as follows. Section 2 recalls basic FCA related notions, introduces MV contexts, and gives a survey of existing approaches dealing with MV contexts. Sections 3 and 4 propose a detailed formalization of an MV Galois connection and its derived structures (MV concepts and MV concept lattices). Section 5 details the application of MV concept lattices to information retrieval. Finally, section 6 concludes the present work and discusses research perspectives. For completing the content of this paper, detailed proofs of the propositions and other examples may be found in a detailed annex at http://www.loria.fr/∼messai/files/papers/Messai ECAI08 Annex.pdf.

D<10

Abstract. In this paper we present an extension of the Galois connection to deal with many-valued formal contexts. We define a manyvalued Galois connection with respect to similarity between attribute values in a many-valued context. Then, we define many-valued formal concepts and many-valued concept lattices. Depending on a similarity threshold, many-valued concept lattices may have different levels of precision. This feature makes them very useful for multilevel conceptual clustering. Many-valued concept lattices are also used in a new lattice-based information retrieval approach for efficiently answering complex queries.

×

× ×

and the attributes are characteristics of these planets. The formal concepts are computed based on the relation I as maximal sets of objects having in common maximal sets of attributes. Formally, a concept is represented by a pair (A, B) such that A ⊆ G,

128

N. Messai et al. / Many-Valued Concept Lattices for Conceptual Clustering and Information Retrieval

B ⊆ M , A = B, and B  = A where A = {m ∈ M | (g, m) ∈ I ∀g ∈ A} (the set of attributes common to all objects in A) and B  = {g ∈ G | (g, m) ∈ I ∀m ∈ B} (the set of objects which have all attributes in B). A and B are respectively called the extent and the intent of the concept (A, B). The derivation operators  : P(G) → P(M ) and  : P(M ) → P(G) form a Galois connection between the powerset lattices P(G) and P(M ) [6]. The set of all concepts in a formal context is denoted by B(G, M, I). A concept (A1, B1) is a sub-concept of (A2, B2) when A1 ⊆ A2 (or equivalently B2 ⊆ B1). In this case, (A2, B2) is a super-concept of (A1, B1) and we write (A1, B1) ≤ (A2, B2). The set of concepts in B(G, M, I) ordered using the partial order “≤” forms the concept lattice of the context (G, M, I) denoted by B(G, M, I). The concept lattice of the context given in Table 1 is represented by the so-called line diagram (or Hasse diagram) shown in Figure 1.

Figure 1. B(G, M, I): the concept lattice corresponding to the context (G, M, I) given in Table 1.

2.2

Many-valued contexts

The basic FCA setting provides appropriate tools for processing data sets easily representable as formal contexts. However, in real-world data sets, objects are described by attributes which can take on several values. In this case, the attributes are said to be many-valued, in contrast to one-valued attributes considered in the previous section. Examples of MV attributes are color, weight, height, etc. The tabular representation of such data sets results in a table where rows are the objects, columns are the attributes, and table entries are attribute values. This representation is called a many-valued context [6]. Formally, an MV context is denoted by (G, M, W, I) where G is a set of objects, M is a set of attributes, W is a set of attribute values, and I is a ternary relation between G, M and W (i.e., I ⊆ G × M × W ). (g, m, w) ∈ I denotes the fact that “the attribute m takes the value w for the object g”. This fact is also denoted by m(g) = w (this notation will be preferred in the present paper). Table 2 shows an example of MV context. It represents the original data set represented as a formal context in Table 1. The objects are planets and the attributes are physical real-valued measures characterizing the planets. In the rest of the paper we will use the initials of the attribute names to designate the attributes e.g. Distance to Sun will be written DS. The basic FCA setting detailed so far cannot be immediately applied to MV contexts. A survey of approaches taking into account MV contexts is given in the following section.

2.3

Processing many-valued contexts

In the literature, there are mainly two families of approaches dealing with MV contexts. The first one consists in transforming an MV

Table 2. An example of MV context.

Diameter

PP Attr. P (103 km) Obj. PP P Mercury Venus Earth Mars

48.8 12.1 12.7 6.7

Mass

Distance to

(1021 t)

Sun ( 106 km)

0.33 4.87 5.97 0.64

57.9 108 150 228

Satellite

1 2

context into an ordinary formal context [6]. The transformation process is called conceptual scaling and the obtained formal context is called a scale. Conceptual scaling consists in replacing each MV attribute by a set of one-valued attributes. The formal context given in Table 1 is a possible scale of the MV context given in Table 2. The MV attribute “Diameter” is replaced by the attributes “D < 10”, “10 < D < 20”, and “D > 20”. It is possible to define different scales for the same MV context. The choice of a scale depends on attribute interpretations and makes conceptual scaling a user dependant task which can hardly be automatized in the case of large data sets. The second family of approaches deals with a particular form of MV contexts, namely fuzzy contexts i.e. MV contexts where the attribute values are truth values of the statement “the object g has the attribute m”. The proposed approaches are usually called “fuzzy formal concept analysis (FFCA)” [1, 2, 11, 7, 12] although they show different ways of dealing with truth values in fuzzy contexts. In [7], the so-called “variable threshold concept lattice” is obtained after transforming an MV context into a one-valued context as follows. First, a threshold δ is chosen. Then, only the values higher than δ are replaced by a cross. In [12], two thresholds are chosen to form the so-called “window” and only the values in this window are replaced by a cross. These two approaches can be seen as a particular kind of conceptual scaling as they result in an ordinary formal context which is then processed using FCA. In [11], the authors use fuzzy logic to deal with fuzzy contexts. In particular, intersection and union operators on fuzzy sets are used to compute fuzzy intents of fuzzy formal concepts. In [1], an approach based on residuated lattices is defined. In this approach, both attributes and objects are seen as fuzzy sets contrary to the previous approach where only attribute sets are considered as fuzzy sets. Other approaches similar to -and covered by- the last one are discussed with details in [2]. In the following, we propose an original approach to process MV contexts. This approach is based on a new Galois connection that takes into account similarities between attribute values in MV contexts. The approach deals with MV contexts where attribute values are totally ordered. This form includes numerical contexts (i.e. the attribute values are numbers) and arbitrary formal contexts where attribute values can be totally ordered such as tiny, small, medium, big, and huge for an attribute font size in an MV context dealing with textual documents. In this way, the present work has connection with Symbolic Data Analysis (SDA) where symbolic description of complex symbolic objects are built according to a Galois Connection [3]. Rephrased in FCA terms, some features of SDA could be reused in the present work for dealing with MV contexts containing categorical ordered attribute values.

N. Messai et al. / Many-Valued Concept Lattices for Conceptual Clustering and Information Retrieval

3

Many-valued Galois connection

3.2

To make the formalization easier we will use an equivalent representation of the MV context. This representation is also an MV context in which the attribute values are shifted to the interval [0,1]. The equivalent representation of the MV context given in Table 2 is given in Table 3. The attribute values in the equivalent representation are obtained by dividing the values in the same column by the maximum value in this column. For example the values of the attribute “Diameter” are divided by 48.8. As the formalization will be done on the Table 3. The equivalent representation of the MV context given in Table 2.

Obj.\Attr. Mercury Venus Earth Mars

D (*48.8) 1 0.3 0.3 0.1

M (*5.97) 0.1 0.8 1 0.1

DS (*228) 0.3 0.5 0.7 1

S (*2)

0.5 1

equivalent representation, we will use (G, M, W, I) to denote such representation instead of the original MV context.

3.1

Attribute sharing between objects

In the basic FCA setting, an attribute m is shared by a set A of objects if and only if each object g in A has the attribute m. This condition is strong and cannot be straightforwardly adapted to MV attributes since the same MV attribute can take on different values for different objects. This condition is relaxed in the case of MV contexts and an attribute is said to be shared by two objects (or more) whenever it has similar values for both objects. Definition 1 Given an MV context (G, M, W, I) and a threshold θ ∈ [0, 1]. 1. Two attribute values wi and wj of an attribute m are similar if and only if |wi −wj | ≤ θ. 2. Two objects gi and gj in G share an attribute m in M if and only if m(gi ) = wi and m(gj ) = wj are similar i.e. |wi − wj | ≤ θ. More precisely, gi and gj share m[wi ,wj ] (assuming that wi ≤ wj ). The interval [wi , wj ] is called similarity interval of m for gi and gj . 3. More generally, a set A ⊆ G of objects shares an attribute m whenever any two objects in A share m. The similarity interval of m for A is [ming∈A (m(g)), maxg∈A (m(g))] and the attribute m shared by objects in A is noted by m[ming∈A (m(g)),maxg∈A (m(g))] . For illustration, consider the MV context given in Table 3 and a threshold θ = 0.2. The objects Mercury and Venus share DS[0.3,0.5] . The previous statement is interpreted as follows. The planets Mercury and Venus have similar distances to the sun, and more precisely, their distances from the sun vary between two values which are 0.3*228 106 km and 0.5*228 106 km. The threshold θ defines the maximal difference allowed between two attribute values. The choice of θ depends on the processed data sets and on the results to be extracted from these data sets. Such choice is similar to the choice of frequency threshold for itemsets in data mining. In the following, Definition 1 will be used to define two derivation operators which form a Galois connection between object sets and attribute sets in an MV context.

129

Partial orders, derivation operators, and many-valued Galois connection

Intuitively, the derivation operators to be defined associate to a set of objects the set of their common attributes with the appropriate similarity intervals and, dually, associate to a set of attributes with appropriate similarity intervals the set of all objects having these attributes. In the following, Iθ denotes all possible intervals [α, β] such that β − α ≤ θ and [α, β] ⊆ [0, 1]. Definition 2 Given an MV context (G, M, W, I) and a threshold θ ∈ [0, 1]: 1. For a set A ⊆ G of objects: Aθ = {m[α,β] ∈M ×Iθ such that m(g)=Ø and ∀gi , gj ∈A, |m(gi ) − m(gj )| ≤ θ, α = ming∈A (m(g)), and β = maxg∈A (m(g))} (the set of attributes common to the objects in A). 2. Dually, for a set B ⊆ M ×Iθ of attributes with similarity intervals: Bθ = {g ∈ G such that ∀m[α,β] ∈ B, m(g) ∈ [α, β]} (the set of objects sharing attributes in B). Consider the MV context given in Table 3 and a threshold θ = 0.2, {Mercury,Venus}θ = {DS[0.3,0.5] } and {DS[0.3,0.5] }θ = {Mercury,Venus}. The definition of a Galois connection between the object sets in G and the attribute sets in M × Iθ requires the definition of appropriate partial order relations on both sets. Formally, all the possible sets of objects are represented by the powerset P(G) and all the possible sets of attributes with similarity intervals given a threshold θ are represented by the powerset P(M ×Iθ ). Considering the inclusion relation “⊆” on P(G), (P(G),⊆) is a partially ordered set. The partial order relation to be defined on P(M ×Iθ ) must combine the inclusion relation “⊆” between subsets of M with the inclusion between similarity intervals of attribute values and the threshold θ. Definition 3 Given two sets B1 and B2 in P(M ×Iθ ). B1 ⊆θ B2 if and only if ∀ m[α1 ,β1 ] ∈ B1 , ∃ m[α2 ,β2 ] ∈ B2 such that [α2 , β2 ] ⊆ [α1 , β1 ]. In the MV context given in Table 3, examples of elements in P(M ×Iθ ) ordered with respect to “⊆θ ” (θ = 0.2) are {D[0.3,0.3] , M[0.8,1] , DS[0.5,0.7] } ⊆θ {D[0.3,0.3] , M[0.8,0.8] , DS[0.5,0.5] } and {D[0.1,0.3] } ⊆θ {D[0.3,0.3] , M[0.8,1] , DS[0.5,0.7] }. Proposition 1 (P(M ×Iθ ), ⊆θ ) is a partially ordered set (i.e. ⊆θ is reflexive, antisymmetric and transitive). Theorem 1 (Galois connection) The derivation operators introduced in definition 2 form a Galois connection (called many-valued Galois connection) between (P(G), ⊆) and (P(M ×Iθ ), ⊆θ ). It follows from Theorem 1 that the derivation operators define two closure systems [6] on G and M ×Iθ . The intersection between two subsets, B1 and B2 , of M ×Iθ is defined as the set of m[α,β] such that {m[α,β] } ⊆θ B1 and {m[α,β] } ⊆θ B2 . The MV Galois connection defined above is used in the following to define and compute MV concepts and MV concept lattice given an MV context.

4

Many-valued concepts and many-valued concept lattices

Definition 4 A many-valued concept is a pair (A, B) where A ⊆ G and B ⊆ M ×Iθ such that Aθ = B and Bθ = A. A and B are

130

N. Messai et al. / Many-Valued Concept Lattices for Conceptual Clustering and Information Retrieval

respectively the extent and the intent of (A, B). Consider again the MV context in Table 3 and a threshold θ = 0.2. Then, ({Mars,Venus,Earth},{D[0.1,0.3] }) and ({Venus,Earth},{D[0.3,0.3] ,M[0.8,1] ,DS[0.5,0.7] }) are MV concepts. Definition 5 Consider an MV context (G, M, W, I) and a threshold θ. 1. If (A1 , B1 ) and (A2 , B2 ) are MV concepts, (A1 , B1 ) is a subconcept of (A2 , B2 ) when A1 ⊆ A2 (which is equivalent to B2 ⊆θ B1 ). In this case (A2 , B2 ) is a superconcept of (A1 , B1 ) and we write (A1 , B1 ) ≤θ (A2 , B2 ). The relation ≤θ is the hierarchical order of MV concepts. 2. The set of all MV concepts of (G, M, W, I) ordered in this way is denoted by Bθ (G, M, W, I) and called the many-valued concept lattice of (G, M, W, I).

scaling called plain scaling [6]. In this case, an attribute m is shared by two objects g1 , g2 only when m(g1 ) = m(g2 ). In the case of a threshold θ = 1, an attribute m is shared by two objects g1 and g2 whenever m(g1 )=Ø and m(g2 )=Ø. This case considers only the presence or absence of a relation between an object and an attribute when computing concepts which corresponds to the basic FCA setting. The MV concept lattice obtained for θ = 1 is the same as the ordinary concept lattice of the formal context obtained by replacing all attribute values by crosses (×). The only difference is that intervals of all possible values of attributes are given in the intents of MV concepts. Figure 3 shows the MV concept lattices of the MV context given in Table 3 for θ = 0 (left) and for θ = 1 (right).

The MV concept lattice of the context represented in Table 3 for a threshold θ = 0.2 is shown in Figure 2. This representation is slightly different from the reduced labelling [6]. The concept corresponding to any node of the diagram is obtained as follows. Similarly to the reduced labelling, the extent is obtained by considering the objects located below the considered node and those that can be reached by descending line paths from this node. However, the intent is given all at once above the considered node. This MV concept lattice is

Figure 3. B0 (G, M, W, I) (left) and B1 (G, M, W, I) (right): the MV concept lattices of (G, M, W, I) given in Table 3 for θ = 0 and θ = 1 respectively.

Figure 2. B0.2 (G, M, W, I): the MV concept lattice of (G, M, W, I) given in Table 3 for θ = 0.2.

Varying the threshold θ results in restricting or relaxing the conditions for attribute sharing between objects. Increasing θ corresponds to enlarging similarity intervals of attribute values which results in forming MV concepts whose intents are coarser partitions of attributes. Conversely, decreasing θ corresponds to reducing similarity intervals of attribute values which results in forming MV concepts whose intents are finer partitions of attributes. Consequently, the possibility of varying the threshold θ makes MV concept lattices a good candidate for multi-level clustering of data in MV contexts. This feature is particularly useful for large contexts. Indeed, one general lattice can be built first (by choosing higher θ) to provide a rough view on the whole context. Then, according to the precision need, more specific lattices can be built (by choosing smaller θ).

computed adapting the classical approach. An MV concept lattice provides an exhaustive and precise representation of data in an MV context. Indeed, attribute values and similarity intervals in the intents of MV concepts represent all possible combinations of MV attributes shared by objects given a similarity threshold θ. For illustration consider the attribute DS transformed into “DS < 100”, “100 < DS < 200”, and “DS > 200” in the scale given in Table 1. None of these attributes is shared by Mercury and Venus. However, the difference between real-world values taken on by DS for both planets is less than the difference allowed between planets having DS = 110 and DS = 190 (which share attribute “100 < DS < 200” and hence the attribute DS). This limit is overcome in the case of MV concept lattice by automatic computation of similarity intervals. Considering the example of DS, Mercury and Venus share DS[0.3,0.5] and B0.2 (G, M, W, I) contains the MV concept ({Mercury,Venus},{DS[0.3,0.5] }). The previous remark concerns also attribute M for objects Venus and Earth, and attribute D for objects Venus, Mars, and Earth. The MV concept lattice Bθ (G, M, W, I) depends on the choice of the threshold θ. A threshold θ = 0 is equivalent to the conceptual

5

Using many-valued concept lattices for IR

Mainly motivated by navigation capabilities in conceptual hierarchies such as concept lattices, various research works have focused on the use of FCA for IR tasks. Many FCA-based IR systems have been proposed for Web document retrieval [4, 10] and for domainspecific retrieval [8] as well. Although these approaches have proved good performances, they are still limited by the problem of representing complex object/attribute relationships (especially in domainspecific retrieval) into one-valued contexts which always results in a loss of information. To overcome this problem, some approaches try to improve IR by using domain ontologies and thesauri [10, 4, 8, 9]. Another way of applying FCA in presence of complex data without transformation into a one-valued context consists in using an MV Galois connection. Data are represented as an MV context and then as an MV concept lattice which is used as retrieval support. Definition 6 (1) A query is a set Y of weighted attributes where weights can be single values or intervals. In an MV concept lattice, a query is represented by a query concept (Yθ , Y ) where Yθ is the set of objects that share all attributes in Y .

N. Messai et al. / Many-Valued Concept Lattices for Conceptual Clustering and Information Retrieval

(2) An object g is relevant for a query Y whenever gθ ∩ Y = Ø (i.e. g shares at least one of the attributes in Y ). Let us consider the example of MV context (G, M, W, I) given in Table 3 and its corresponding MV concept lattice B0.2 (G, M, W, I) shown in Figure 2. It is possible to deal with queries such as “which are the planets having a diameter of about 12 thousand km and a distance to sun of about 150 million km”. The formal representation of this query is {D0.3 , DS0.7 }. The values 0.3 and 0.7 weighting the query attributes are obtained in the same way as the values in Table 3. It is also possible to deal with queries such as “which are the planets having a diameter between 6 thousand km and 13 thousand km and a distance to sun between 100 million km and 150 million km”. The formal representation of this query is {D[0.1,0.3] , DS[0.5,0.7] }. It is also possible to to deal with queries such as “which planets have a satellite”. The formal representation of this query is {S}. A retrieval session starts by inserting a query into the MV concept lattice. According to the definition of objects relevance, the relevant objects are to be found in the extents of the query concept and its super-concepts. Objects in the extent of the query concept, Yθ , are the most relevant since they have all the query attributes in Y . Let us consider (A, B) a query super-concept i.e. (Yθ , Y ) ≤θ (A, B). The objects in A, are less relevant than those in Yθ because they have only a part of the query attributes (B ⊆θ Y ). Accordingly, the query answer is incrementally computed while navigating in the MV concept lattice starting from the query concept and following the upward links to consider the query super-concepts. For illustration consider the query “which are the planets having a diameter of about 12 thousand km and a mass of about 5 1021 t and a distance to sun of about 150 million km” which is formally represented by {D0.3 , M0.8 , DS0.7 }. The steps of the retrieval session for this query considering B0.2 (G, M, W, I) are shown in Figure 4. The obtained answer for this query is the following.

Figure 4. Retrieval session for the query {D0.3 , M0.8 , DS0.7 }. The dotted ellipses show the concepts whose extents contain the relevant objects. The numbers near the ellipses represent the steps of the retrieval.

1 – Venus: (D0.3 , M0.8 , DS0.7 ). 2 – Earth: (D0.3 , M1 , DS0.5 ). 3 – Mars: (D0.1 ). It can be noticed in this answer that the object Mercury does not appear although it is in the extent of a query super-concept, namely ({Venus,Mercury}, {DS[0.3,0.5] }). This absence is justified by the fact that Mercury does not fulfill the relevance criterion (cf. Definition 6-(2)). As it is shown here, the IR process benefits from the extension of FCA capabilities with MV Galois connection and MV concept lattices. In this way, it is possible to deal with real-world applications

131

involving search on complex data, e.g. document search on the Web, biological data or Web site search. One advantage of the formalism introduced here is firstly its simplicity and secondly the fact that the standard algorithms for designing concept lattices can be reused straightforwardly, with a minimal adaptation. These are guarantees that the present MV FCA extension is generic, powerful, and can be used to improve and extend existing applications by allowing to handle complex data.

6

Conclusion and future work

In this paper, we introduced an extension of FCA to deal with complex data represented as MV contexts. We defined an MV Galois connection based on similarity between attribute values. The basic idea is that two objects share an attribute whenever the values taken by this attribute for these objects are similar (i.e. their difference is less than a threshold). This Galois connection is the basis of the computation of MV concepts and MV concept lattices. Depending on the similarity threshold, MV concept lattices can have different levels of precision. This makes them a good tool for multi-level clustering which is particularly useful for large contexts. Many-valued concept lattices are also suitable for lattice-based IR. We defined a retrieval method based on navigation into MV concept lattices to efficiently deal with complex queries. In the future, the MV Galois connection will be extended to deal with more general MV contexts, i.e. symbolic MV attributes. Similarity between attribute values can be taken from domain ontologies and thesaurus. Resulting lattices can be used in different application domains including Clustering, Information Retrieval, and Complex Data Mining. At the moment, an evaluation study is carried out on real-world biological data.

REFERENCES [1] R. Belohlavek, ‘Lattices generated by binary fuzzy relations’, Tatra Mountains Mathematical Publications, 16, 11–19, (1999). [2] R. Belohlavek and V. Vychodil, ‘What is a fuzzy concept lattice?’, in CLA 2005, Proccedings. Olomouc, Czech Republic, pp. 34–45. [3] H.H. Bock and E. Diday, eds. Analysis of Symbolic Data: Exploratory Methods for Extracting Statistical Information from Complex Data, volume 15 of Studies in Classification, Data Analysis, and Knowledge Organization. Springer, February 2000. [4] C. Carpineto and G. Romano, Concept Data Analysis: Theory and Applications, John Wiley & Sons, 2004. [5] B. Ganter, G. Stumme, and R. Wille, eds. Formal Concept Analysis, Foundations and Applications, volume 3626 of LNCS. Springer, 2005. [6] B. Ganter and R. Wille, Formal Concept Analysis, Springer, mathematical foundations edn., 1999. [7] J. Ma, W.-X. Zhang, and S. Cai, ‘Variable threshold concept lattice and dependence space’, in FSKD 2006, Proccedings, volume 4223 of LNCS, pp. 109–118. Springer. [8] N. Messai, M.-D. Devignes, A. Napoli, and M. Smail-Tabbone, ‘Querying a bioinformatic data sources registry with concept lattices’, in ICCS 2005, Proceedings, volume 3596 of LNAI, pp. 323–336, Kassel, Germany, (July 2005). Springer. [9] N. Messai, M.-D. Devignes, A. Napoli, and M. Smail-Tabbone, ‘Extending attribute dependencies for lattice-based querying and navigation’, in ICCS 2008, Proceedings, volume 5113 of LNAI, pp. 189–202, Toulouse, France, (July 2008). Springer. [10] U. Priss, ‘Lattice-based Information Retrieval’, Knowledge Organization, 27(3), 132–142, (2000). [11] S. Ben Yahia and A. Jaoua, ‘Discovering knowledge from fuzzy concept lattice’, Data mining and computational intelligence, 167–190, (2001). [12] W. Zhou, Z. Liu, Y. Zhao, and Z. Xie, ‘Clustering-based reduction algorithm on the structure of fuzzy concept lattices’, in ICFCA 2007, Supplementary Volume, pp. 131–145.

132

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-132

Online optimization for variable selection in data streams Christoforos Anagnostopoulos1,2 and Dimitris Tasoulis2 and David J. Hand2,3 and Niall M. Adams3 Abstract. Variable selection for regression is a classical statistical problem, motivated by concerns that too many covariates invite overfitting. Existing approaches notably include a class of convex optimisation techniques, such as the Lasso algorithm. Such techniques are invariably reliant on assumptions that are unrealistic in streaming contexts, namely that the data is available off-line and the correlation structure is static. In this paper, we relax both these constraints, proposing for the first time an online implementation of the Lasso algorithm with exponential forgetting. We also optimise the model dimension and the speed of forgetting in an online manner, resulting in a fully automatic scheme. In simulations our scheme improves on recursive least squares in dynamic environments, while also featuring model discovery and changepoint detection capabilities.

(RLS-AF) also implies a certain degree of tracking capability in the general time-varying case [5]. In this paper, we use RLS-AF to maintain an up-to-date estimate of the correlation structure, which we then use to incrementally update our estimate of the Lasso solution and tune the penalty weight, using AIC. We lay out the details of this algorithm in Section 2. The extensive simulation study that we describe in Section 3 establishes that the resulting scheme scales very well with the dimension of the stream, is fully automatic and reacts interpretably to changes in the stream dynamics, maintaining excellent predictive performance.

1

Assume a static linear regression model with normal errors, where yi is a univariate response and Xi a 1 × p vector of regressors:

Introduction

Regression models with many covariates can potentially capture more information about the response, but are also prone to overfit. Variable selection techniques have been used as a systematic way of negotiating this trade-off [11, 14] in the static, offline case. The need for such methods to carry over in the streaming data context has been recognised in the literature [6] but not yet systematically addressed. Variable selection can lead to improvements in both parsimony and predictive performance [11]. Early methods would proceed in a greedy manner, iteratively adding maximally predictive covariates into the model, but suffered from local optima and lack of robustness[4]. More recent developments, such as the Lasso [15], have largely overcome these shortcomings via the use of convex, sparsity-inducing penalties. The weight of the penalty controls the model dimension and its optimal value can be estimated using general purpose model scoring techniques, such as the Akaike Information Criterion (AIC) [1], or more complex, tailored methods [17]. Several efficient algorithms have been proposed for the optimisation of Lasso-type problems (e.g., [15, 10, 8]). However, an online implementation had been lacking, so that only greedy heuristics have so far been employed in the streaming data case (e.g., [16]). Another primary concern in the streaming context is adaptivity to drifting stream dynamics. An effective way to enforce adaptivity is to make past data decreasingly relevant to the estimation procedure via the use of forgetting factors [13]. In the case of linear regression, these forgetting procedures may be employed online and complemented by a gradient descent procedure that converges to the optimal rate of forgetting [13] for fixed drift parameters. This convergence property of recursive least squares with adaptive forgetting 1 2 3

Author for correspondence: [email protected] The Institute for Mathematical Sciences, Imperial College London, SW7 2PG, London Department of Mathematics, Imperial College London, South Kensington Campus, London SW7 2AZ, UK

2

Description of the Algorithm

2.1

Recursive Least Squares

yi = Xi β T + ,

 ∼ N (0, σ 2 )

Given a dataset (yi , Xi )n i=1 of observations from this model, denoted by (y, X) in vector form, the ordinary least squares (OLS) estimate of β is that which minimises the residual sum of squares (RSS): β OLS = argminβ

n X (yi − Xi β T )2 = P Q−1

(1)

i=1

where P = Xy and Q = X T X are sample covariance estimates. The proof of this result is standard (e.g., [7]) and rests on the assumption that X T X is non-singular, which in practice holds for n >> p. Assume now that the data arrive in a time-ordered fashion, so that at time t, we have an estimate βtOLS = Pt Q−1 t . At time t + 1, we can linearly update the sample covariances, e.g., Qt+1 = Qt + XtT Xt , but updating βt+1 via (1) requires us to invert Qt+1 which is impractical (O(p3 )) in online contexts for large p. We hence employ the −1 Sherman-Morrison formula [12] to express Q−1 t+1 in terms of Qt , −1 T under the assumption that Xt+1 Qt Xt+1 = −1: T −1 = Q−1 − Q−1 t t+1 = (Qt + Xt Xt )

−1 T Q−1 t Xt+1 Xt+1 Qt −1 T 1 + Xt+1 Qt Xt+1

(2)

This recursive characterisation involves only ‘affordable’ matrix multiplications, i.e., computable in O(p2 ), rather than O(p3 ).

2.2

Adaptive Forgetting

In case the true regression coefficients are time-varying, we can track them by iteratively scaling down the importance of past datapoints by a factor of λ ≤ 1, where λ = 1 yields the static, OLS solution: Qt+1 =

t+1 X i=1

T λt+1−i XiT Xi = λQt + Xt+1 Xt+1

(3)

C. Anagnostopoulos et al. / Online Optimization for Variable Selection in Data Streams

Since the update of the sample covariance is still of the same form, βtOLS = Pt Q−1 may still be updated recursively via (2). t Setting the value of λ is analogous to choosing an optimal window size and can be done using standard techniques, such as crossvalidation. However, the optimal value of λ may often be timevarying: for instance, if the drift features abrupt changes followed by intervals of no change, the value of λ should adapt accordingly, taking small values only at times of abrupt change. In [13], this need is addressed via an adaptive scheme that at each timepoint moves λ in the direction that minimises RSS the most at the current timepoint: λt+1 = λt + cλ sign(∂RSS/∂λt )

(4)

for some small constant cλ . The formula for the gradient is derived in [13, p.734] and is also O(p2 ) so that, overall, RLS-AF is fully online.

2.3

The Lasso Estimator

The Lasso algorithm for variable selection [15] penalises the RSS by the sum of the absolute values of the learnt regression coefficients, i.e., the L1-norm of the regression vector: ) ( n ! p X X L1 T 2 (yi − Xi β ) |βj | (5) β ← argmin +γ β

i=1

j=1

It is a well-known fact that L1-norm penalties naturally favor sparse solutions [10], while retaining the convexity of the objective function. As a result, β L1 can be computed by a line search algorithm: for each j = 1, . . . , p, we minimise the penalized RSS with respect to βjOLS and iterate until convergence. The minimisation step is closed form: ( ˛ ˛ ˛ ˛ sign(βjL1 − Sj )(˛βjL1 − Sj ˛ − γ), if ˛βjL1 − Sj ˛ ≥ γ L1 βj ← 0, otherwise (6) where Sj is the gradient of the penalised RSS with respect to βjL1 , holding all other coefficients constant, and is given by: Sj = ∂RSS/∂βjL1 = −0.5

n X

Xij (yi − Xi (β L1 )T )

(7)

i=1

This algorithm was first proposed as the shooting algorithm in [10]. Notably, the RHS of (7) expands to a formula that involves only the sample covariances, P and Q. We are hence free to employ instead the forgetful versions, Pt , Qt obtained from RLS-AF. This allows us to hybridise the two algorithms, so that at each tick we perform one iteration of the shooting algorithm, initialised at the previous tick’s estimate of β L1 , retaining an O(p2 ) complexity per tick. Note also that this algorithm assumes that the sample covariances have been standardised, so that rather than Pt and Qt , we have to provide it with their standardised versions, taking care to properly reweight β L1 . This has no effect on computational complexity.

2.4

Learning the Dimensionality Online

There remains the issue of setting γ, the shrinkage parameter featured in (5), which effectively controls the dimension of the learnt model. To do so we employ a standard model scoring criterion, the Akaike Information Criterion [1]. This penalises the RSS of a regression estimate βˆ by the number q of its non-zero coefficients: ˆ = n log RSS + 2q − n AIC(β) “ ” ˆ βˆT + 2q − n = n log R − 2P βˆT + βQ

133

where we have rewritten the RSS in terms of the sample covariances P and Q, as well as the sample variance R = y T y. Normally, n is equal to the sample size, but since we will be employing our forgetful versions Pt , Qt , Rt the effective sample size at time t depends on previous forgetting factors: n(t) = λt + λt λt−1 + · · · =

t Y t X

λm = λt (1 + n(t − 1)) (8)

i=1 m=i

In our case, the candidate regression estimates which we wish to compare are Lasso solutions, β L1 (γ), for various values of γ. We hence wish to determine the value of γ that produces the Lasso solution with minimum AIC. Ideally, we should perform an exhaustive search over γ, but at present we have no way of optimising this efficiently enough for an online scheme. Moreover, the relationship between γ and q is not closed form [8], so that gradient descent methods do not readily apply. Instead, as a first approximation, we perform a simple numerical minimisation at each tick, computing AIC(β L1 (γ)) for γ equal to each of γt − cγ , γt , γt + cγ , where cγ is a small constant and γt our current estimate. We then use the value with the smallest AIC as our setting of γt+1 . Hybridising RLS-AF with the shooting algorithm and the numerical minimisation of AIC yields our algorithm, RLASSO-AF, for online variable selection. Its complexity is dominated by RLS-AF and it is fully automatic, except for its dependence on initialised values and gradient descent steps (cλ and cγ ). ´˘ ` algorithm ONLINE LASSO-AF y, X, cγ , cλ , OLS L1 initialise β0 = 0, β0 = 0, γ0 = 0, λ0 = 1, P0 = 1, Q0 = I at time t do # compute RLS estimates (note that we denote Q−1 by Gt ) t yˆt+1 ← Xt+1 (βtL1 )T # current prediction n ← λt (n + 1) # effective sample size G Xt+1 # auxillary step for notational ease kt+1 = λ +X t G XT t

t+1

t+1

t

Gt+1 ← λ1t Gt − λ1t kt+1 Xt+1 Gt # this is Q−1 t+1 T Pt+1 ← λt Pt + Xt+1 yt+1 2 Rt+1 ← λt Rt + yt+1 # sample variance of y, required in AIC t+1 ← yt+1 − Xt+1 (βtOLS )T # RLS residual OLS βt+1 ← Pt+1 Q−1 t+1 # RLS regression ˘ coefficients estimate for γ in {γt − cγ , γt , γt + cγ } do # compute candidate˘Lasso estimate βˆ for j = 1, . . . , p do S ˛= −0.5(P˛t+1 (j) − Qt+1 (:, j)(βtL1 )T ) ˛ ˛ˆ − S ˛ ≥ γt then if ˛β(j) ˛ “˛ ” ˆ ˆ − S) ˛˛β(j) ˆ − S ˛˛ − γ β(j) ← sign(β(j) else

¯ ˆ β(j) ←0 # compute AIC of candidate Lasso estimate q ←number of“non-zero coefficients in βˆ ” ¯ ˆ t+1 βˆT + 2q − n AIC(γ) = n log Rt+1 − 2Pt+1 βˆT + βQ γt+1 ← the value in {γt − cγ , γt , γt + cγ } with smallest AIC L1 ← the respective candidate estimate βt+1 # compute ∂RSS/∂λ = ∂Tt+1 t+1 /∂λ in three recursive steps T T St+1 ← λ1t (I − kt+1 Xt+1 )St (I − Xt+1 kt+1 )+ 1 T 1 + λt kt+1 kt+1 − λt Gt+1 + T t+1 ψt+1 ← (I − kt+1 Xt+1 )ψt + St+1 Xt+1 T T ∂RSS ← ψ X  t+1 t+1 t+1 ∂λ ` ∂RSS ´ ¯ λt+1 ← λt + cλ sign ∂λ

134

3 3.1

C. Anagnostopoulos et al. / Online Optimization for Variable Selection in Data Streams

Experimental Results Description of the Simulation Engine

We test our method against simulated datasets (yt , Xt )n t=1 , letting (yt , Xt ) ∼ N (0, Σt ) where Σ1 , . . . , Σt , . . . is a sequence of dynamically changing (p + 1) × (p + 1) covariance matrices. The joint Gaussianity assumption is convenient in that it guarantees the required linear relationship between the response and the covariates, while also producing realistic variability in the covariates. We assume zero mean vectors for simplicity. To generate the dynamic sequence of Σt s, we employed the simple and easy to control method described in [2]. We divide our timescale into regular intervals by introducing ‘changepoints’, t1 , . . . , ti , . . . . The distance between changepoints determines the ‘speed of the drift’. We then randomly generate symmetric, positive definite matrices Q1 , Q2 , . . . to lie at each changepoint, setting Σti = Qi . In between, for t ∈ (ti , ti+1 ), we set: ti+1 − t Σt = μQi + (1 − μ)Qi+1 , with μ = ti+1 − ti

(9)

This turns the difficult problem of generating smoothly changing covariance matrices to the easy problem of generating arbitrary covariance matrices. Moreover, it allows us to control the true sparsity of the regression relationship: for each changepoint, we randomly generate the p × p covariance of the covariates and, separately, a p × 1 vector of regression coefficients of a specified sparsity. We can then use standard formulas to obtain a joint covariance Qi . We also considered abrupt change, setting μ = 1 in (9). Note that all the underlying covariance matrices were chosen to be standardised, so as to retain our sense of scale despite the changes in the correlation structure. We have repeated our experiments with non-standardised covariances and have found that our qualitative findings persist, although lack of standardisation introduces appreciably more volatility in the error sequence across all settings.

3.2

Figure 1. Plot of time and sampled averaged log(RSS) for RLS-AF, RLASSO-AF learning γ online and RLASSO-AF learning γ optimally via an offline search. The underlying streams have length 1000 and true sparsity of 0.5. In the case of online RLASSO-AF, the plot is shrunk for visibility.

shortcomings can be expected to dominate when there is little or no covariance drift, where least squares estimation enjoys ever increasing sample sizes and eventually ceases to overfit, making its performance hard to beat. In contrast, in the dynamic case where the least squares fit is imperfect, γ need not be as fine-tuned for our scheme to reap the benefits of variable selection. In Figure 2, we demonstrate via a surface plot how the relative improvement in performance of RLASSO-AF over RLS-AF depends on the dimension and true sparsity ratio of the underlying stream. Results are averaged over several streams of length 1000, featuring changepoints every 100 points, for each of a range of settings for the stream dimension and sparsity. The plot suggests that RLASSO-AF outperforms RLS-AF more and more as p grows, when the covariance structure is changing, regardless of whether the true model is sparse or not. In fact, relative performance does not seem to be affected strongly by the sparsity of the true regression model: our adaptive learning of γ seems not to be sophisticated enough to sufficiently exploit sparsity in the true model, suggesting room for improvement.

Results

We start by discussing performance in the static case to get some first insights into the problem. We measure performance by the mean squared residual error, (yt − Xt β)2 , averaged over time as well as over several streams of varying dimensionality. The results are reported in Figure 1, where we compare RLS-AF with RLASSO-AF, as well as its offline version where γ is optimally set via exhaustive search. We immediately observe that online RLASSO-AF is outperformed by RLS-AF for all values of p. However, this effect can be exclusively attributed to the difficulties of learning the shrinkage parameter online: when γ is optimally selected, our estimator outperforms RLS-AF by a margin that grows wider with p. These results are consistent with theory. In the static case, our scheme is guaranteed for any fixed setting of γ to converge to the respective Lasso estimator, which is known to be superior to the OLS estimate when γ is suitably set, in particular when the true model is sparse and p large [15]: we can outperform RLS-AF if we optimally select γ. However, making this selection online is not expected to be easy. First, closed-form model scoring criteria, such as the AIC, do not always perform well in practice as they rest on several unrealistic assumptions [4], so that their suitability cannot be guaranteed. Second, our adaptive scheme only approximately minimises AIC. These

Figure 2. Surface plot of relative increase in performance of RLASSO-AF versus RLS-AF, averaged over several streams of dimension ranging from 5 to 100 covariates and true sparsity ranging from 0% to 90%.

C. Anagnostopoulos et al. / Online Optimization for Variable Selection in Data Streams

We now move to describe the adaptive characteristics of our algorithm. This behavior is best illustrated against a single stream, but persists across a variety of settings, as explained later. Our chosen testbed is a stream of length n = 2000, featuring an abrupt change every 200 timepoints. The true regression relationships are 50% sparse: half the covariates are randomly selected at each changepoint to be set to exact zeros. The total number of covariates is 50. Our results against this testbed are summarised in Figure 3. First, we confirm the findings of [13] in immediately noticing that the forgetting factor adapts brilliantly in most cases (Figure 3, Plot III): shortly after a changepoint occurs, λ begins to drop fast, allowing the algorithm to quickly forget data prior to the changepoint and reestimate the sample covariances. An easy computation (that of equation (8)) shows that, when λ is at its lowest, the effective sample size drops to a mere 10 datapoints, an impressively short memory given that we are estimating a 50 × 50 covariance matrix. After a short interval, λ increases again making available a longer memory, in recognition of the fact that any irrelevant data have by now been forgotten and the process is once again constant. Second, we confirm that the Lasso predictions are more precise and less volatile than the RLS-AF predictions at and around the changepoints (shown as smaller peaks in Figure 3, IV), yielding an overall 155% boost in performance. Since this stream features abrupt changes, the covariance is static in between changepoints so that RLS-AF catches up with RLASSO-AF. Indeed, switching to smooth changes removes these static periods yielding further advantage to RLASSO-AF and raising the relative performance to 161% (plot not included for lack of space). Beyond predictive performance, we also gain appreciably in model discovery capabilities: in Figure 4 we can clearly observe that whenever a covariate suddenly becomes inactive, our algorithm will prune it fairly consistently shortly afterwards.

Figure 3. A plot of the learnt shrinkage parameter (Plot I), the learnt model dimension (Figure 3, Plot II), the learnt forgetting factor (Figure 3, Plot III) and the residual errors of both RLS-AF and RLASSO-AF against time. The underlying simulation utilises a stream featuring abrupt changes every 200 timepoints and 50% sparse true regression models.

135

Figure 4. Plot of five randomly selected true regression coefficients stacked against their respective learnt Lasso estimates, against time. Black indicates an exact 0, white indicating values away from 0. The underlying simulation utilises a stream featuring abrupt changes every 200 timepoints and 50% sparse true regression models.

Finally, we turn our attention to the adaptive behavior of the shrinkage parameter. As we indicated earlier, our adaptive routine, despite its shortcomings, manages to learn γ well enough for fastpaced environments, where there is enough to gain by variable selection so that fine-tuning is not necessary. This is already in some sense important, as it makes our algorithm practically applicable. However, the adaptive behavior is very interesting in its own right, as it reveals a complicated dependency of AIC to the shrinkage parameter, the residual error and the forgetting factor. To fully address this interplay lies beyond the scope of the current work and is our main research aim for the future. We take, however, the opportunity to introduce some key issues. Perhaps surprisingly, our adaptive algorithm does not spot a single reasonable value for γ. Instead, as depicted in Plot I of Figure 3, it seems to react to change in a consistent, repeatable manner, peaking shortly after changepoints. This can be attributed to the strong dependence of AIC to the residual error, as is suggested by plotting the cross-correlation between γ and the residual error (Figure 5), averaged over several samples from the same stream characteristics. This is evidence that, at times of crisis, our algorithm tends to prune more violently, in an attempt to contain the error. Still, as shown in Figure 3, Plot II, the actual number of variables pruned is on average higher during periods of constancy than during periods of change. A closer look at Figure 3, Plots II and III, reveals that this effect is mediated by the reaction of the forgetting factor: the number of variables pruned follows λ much more closely than it does the residual error. Indeed, a cross-correlation plot averaged over several streams confirms this (Figure 6). This strong correlation can be explained by the dependence of AIC on both the effective sample size (a function of λ) and, of course, the quality of the Lasso estimates, which also increases with λ. Overall, a complicated picture is painted involving lagged feedback mechanisms. At times of stability, AIC tends to favor sparser models, since the increasing quality of the Lasso estimates offers ‘more for less’, in particular since the true, underlying regression model is sparse. At times of change, however, the reaction of the algorithm is more complex. The sudden increase in residual error causes γ to temporarily shoot up, shrinking and pruning away coefficients so as to contain the error. Shortly afterwards the forgetting factor drops in response and γ follows. This dependence of γ on λ is

136

C. Anagnostopoulos et al. / Online Optimization for Variable Selection in Data Streams

in fact the key relationship of interest and deserves further study. In particular, it could perhaps be exploited to improve our algorithm by way of a joint optimisation step for γ and λ.

Figure 5. Plot of the sample cross-correlation between the number of variables pruned at each timepoint and the value of the forgetting factor.

ing line-search gradient descent. Recent work on pathwise coordinate optimisation [9] has established that several alternative selection schemes are amenable to such methods, which we intend to try out in future versions of our algorithm. Finally, our adaptive algorithm reveals interesting insights into the interdependence between shrinkage, pruning, forgetting and model scoring. This suggests room for improvement by way of a joint optimisation step for these parameters that exploits their interplay. We also wish to apply these insights to a yet harder variant of our problem, whereby only selected covariates are ever actually observed, introduced in [3]. Our algorithm is a first step in establishing a framework for online variable selection, so that we may investigate in a principled manner the interplay between forgetting, choice of focus and change. Our results open up an exciting research direction that raises novel questions about the nature of learning in dynamic environments.

ACKNOWLEDGEMENTS This work was undertaken as part of the ALADDIN (Autonomous Learning Agents for Decentralised Data and Information Systems) project and is jointly funded by a BAE Systems and the EPSRC (Engineering and Physical Research Council) strategic partnership, under EPSRC grant EP/C548 051/1. The work of David Hand was partially supported by a Royal Society Wolfson Research Merit Award.

REFERENCES

Figure 6. Plot of the sample cross-correlation between the shrinkage parameter and the value of the forgetting factor. The spread of the correlation is mostly due to the effect of averaging over several streams.

To conclude, we remark that our algorithm is somewhat sensitive to the size of the gradient steps, cλ and cγ , although to a much smaller extent than to the underlying quantities optimised, λ and γ.

4

Conclusion

Variable selection can lead to improvements in predictive performance in the static case, but has so far not been sufficiently exploited in the streaming context, for lack of online implementations of standard selection algorithms. We propose an online optimisation scheme that implements the Lasso algorithm, converging to the optimal solution in the static case. We hybridise it with recursive least squares adaptive forgetting to allow it to track changing covariance structures. We also estimate the optimal dimensionality via an algorithmic step that at each timepoint takes a step in the direction of minimum AIC, to produce a fully automatic O(p2 ) algorithm for online variable selection in streaming data. We test our algorithm against simulated data across a variety of settings. We observe that our adaptive choice of model dimension suffers in environments of slow change, where the least squares solution tends to be very accurate, but can offer great improvements in performance and promising model discovery capabilities in fast-paced environments. As future work we intend to test our algorithm against real-world datasets, also making use of alternative model scoring criteria, such as the BIC, Cp or corrected AIC. We also remark that the same algorithmic framework may be employed to construct online implementations of any variable selection scheme that may be solved us-

[1] H. Akaike, ‘Information theory and an extension of the maximum likelihood principle’, Intern. Symposium on Information Theory, 2 nd, Tsahkadsor, Armenian SSR, 267–281, (1973). [2] C. Anagnostopoulos and N.M. Adams, ‘Simulating dynamic covariance structures for testing the adaptive behaviour of variable selection algorithms’, Proc. of the 10th Intern. Conf. on Computer Modelling and Simulation, UKSIM/EUROSIM 2008, (2008). [3] C. Anagnostopoulos, N.M. Adams, and D.J. Hand, ‘Deciding what to observe next: adaptive variable selection for regression in multivariate data streams’, Proc. of ACM Symposium on Applied Computing, 2, (2008). [4] L. Breiman, ‘Heuristics of instability and stabilization in model selection.’, The Annals of Statistics, 24(6), 2350–2383, (1996). [5] M. Campi, ‘Estimation and control’, Journal of Mathematical Systems, 4, 13–25, (1994). [6] G. Dong, J. Han, L.V.S. Lakshmanan, J. Pei, H. Wang, and P.S. Yu, ‘Online mining of changes from data streams: Research problems and preliminary results’, Proc. of the 2003 ACM SIGMOD Workshop on Management and Processing of Data Streams, (2003). [7] N.R. Draper and H. Smith, Applied Linear Regression, Wiley, New York, 1966. [8] B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, ‘Least angle regression’, Annals of Statistics, 32(2), 407–499, (2004). [9] J. Friedman, T. Hastie, H. Hofling, and R. Tibshirani, ‘Pathwise coordinate optimization’, The Annals of Applied Statistics, 1(2), 302–333, (2007). [10] W.J. Fu, ‘Penalized regressions: the Bridge versus the Lasso’, Journal of Computational and Graphical Statistics, 7(3), 397–416, (1998). [11] E.I. George, ‘The variable selection problem’, Journal of the American Statistical Association, 95(452), 1304–1308, (2000). [12] G.H. Golub and C.F. Van Loan, Matrix Computations, Johns Hopkins University Press, 1996. [13] S. Haykin, Adaptive Filter Theory, Prentice-Hall, Inc. Upper Saddle River, NJ, USA, 1996. [14] A.J. Miller, Subset Selection in Regression, CRC Press, 2002. [15] R. Tibshirani, ‘Regression shrinkage and selection via the Lasso’, Journal of the Royal Statistical Society, Series B, 58(1), 267–288, (1996). [16] B. Yi, N. Sidiropoulos, T. Johnson, H.V. Jagadish, C. Faloutsos, and A. Biliris, ‘Online data mining for co-evolving time sequences’, Proc. of the 16th Intern. Conf. on Data Engineering, 13–22, (2000). [17] H. Zou, T. Hastie, and R. Tibshirani, ‘On the “Degrees of Freedom” of the Lasso’, Ann. Statist, 35(5), 2173–2192, (2007).

ECAI 2008 M. Ghallab et al. (Eds.) IOS Press, 2008 © 2008 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-58603-891-5-137

137

Sub Node Extraction with Tree Based Wrappers Stefan Raeymaekers and Maurice Bruynooghe1 Abstract. String based as well as tree based methods have been used to learn wrappers for extraction from semi-structured documents (e.g., HTML documents). Previous work has shown that tree based approaches perform better while needing less examples than string based approaches. A disadvantage is that they can only extract complete text nodes, whereas string based approaches can extract within text nodes. This paper proposes a hybrid approach that combines the advantages of both systems and compares it experimentally with a string based approach on some sub node extraction tasks.

1 Introduction Some information is only available in HTML documents. Wrappers for extracting such information are very useful and learning them from examples is an active research field as it is tedious to build them manually and they require maintenance each time the templates generating the webpages are updated. In string based approaches [1, 3, 5, 6, 4, 7, 10, 11, 15], the document is viewed as a sequence of tokens and markup tags from which a subsequence gets extracted. This is done by learning to recognize the start and end boundaries of the target substring. These boundaries are typically between two tokens.2 The markup tags define an implicit tree structure on the document. In the string representation, the relations in this tree are hidden, as in the flattened tree, parent or sibling nodes of a given node are separated by the tokens and tags that make up the subtrees under its (preceding) siblings. This renders the induction task more difficult. Tree based approaches [9, 12, 2] view the document as a tree, preserving the tree relations; [13] experimentally compares string and tree based approaches and concludes that the latter perform better: they need less examples and the induction time is often orders of magnitudes lower. However, they can only extract complete nodes while boundaries can be inside nodes, i.e., sub node extraction can be required. This paper investigates the viability of two approaches for sub node extraction, starting from a state of the art tree based system [12]. In one approach, trees are extended, i.e., the text nodes become internal nodes, having as children the tokens of the original text value. The other is a two phase approach:a hybrid system uses the tree based system [12] for node extraction, while in a second step the extracted nodes are further processed with STALKER, a state of the art string based system [11], to perform the sub node extraction. These approaches are experimentally compared with standalone STALKER. Whereas STALKER uses a hierarchical approach that splits the learning task in simpler tasks that are learned separately, the version used for the sub node extraction ommits the task 1

2

K.U.Leuven, Dept. of Computer Science, Celestijnenlaan 200A, 3001 Leuven, Belgium, email : {stefan.raeymaekers,maurice.bruynooghe}@cs.kuleuven.be Using tokens instead of characters as basic units speeds up the learning.

splitting. Indeed, the sub node extraction task is typically much simpler that the initial extraction task. The rest of the paper 3 . is organized as follows. In Section 2, we give an overview of the STALKER system and in Section 3 of the used tree based system. In Section 4 we discuss the main contribution of this paper, the extensions to enable sub node extraction. Section 5 describes the experimental setup and discusses the results. We conclude in Section 6. Below we introduce an example for later use. Example 1 A web site running a restaurant guide, allows visitors to search for restaurants based on parts of their name. It returns a list of restaurants that is constructed from a fixed template. In Figure 1, a possible outcome is shown for a search on ’china’. From this web page we can extract the following fields: the name of the restaurant(N), its type(T), the city(C) where it is located, and a phone number(P). For each restaurant we could also extract the url(L) from the link (leading to more detailed address information). And from the top sentence, the search term(S) that generated the page can be found. Note that the occurrence of the search term in the name is rendered in italic, while the land code of the phone number is in bold. a)

Restaurant Guide: search results for china

NewChinaTown (chinese) BrusselsTel: +32(0)2 345 67 89

RoyalChina(chinese) LeuvenTel: +32(0)16 61 61 61

ChinaGarden (chinese) AmsterdamTel: +31(0)20-4321234