Part III
Semistructured Data Management
Flexible Query Languages for Relational Databases: An Overview 1
2
3
Antonio Rosado , Rita A. Ribeiro , Slawomir Zadrozny , Janusz Kacprzyk
3
1
Universidade do Algarve, Campus de Gambelas, 8000 Faro, Portugal
[email protected] 2 UNINOVA, Campus of New University of Lisbon, Caparica 2929-516 Portugal
[email protected] 3 Systems Research Institute, Polish Academy of Sciences, 01-447 Waszawa, Poland {kacprzyk,zadrozny}@ibspan.waw.pl
1 Introduction Managers rely more and more on the use of databases to obtain insights and updated information on activities of their institutions and companies. More and more people, from experts to non-experts, are depending on information from databases, to fulfill everyday tasks, notably those related to decision making. Basically, the content of a database describes selected aspects of the real world relevant for a given company, institution, etc. Often, our knowledge about the entities represented in a database as well as our preferences as to what should be retrieved from a database are imperfect or imprecise. This raises a question of a proper modeling of imperfect information in the context of database management systems (DBMSs). The focus of this paper is on flexible query languages (FQL) for databases that are based on fuzzy sets theory. Since there are many contributions in this field, we propose two taxonomies to help and guide database designers and users. These taxonomies address the FQL in crisp relational databases and in fuzzy relational databases, respectively. Approaches mentioned in these taxonomies are not exhaustive in terms of literature, which is huge, but they are quite representative. We believe that these two taxonomies provide a better understanding of the field and can help select the best approaches to solve specific problems.
A. Rosado et al.: Flexible Query Languages for Relational Databases: An Overview, Studfuzz 203, 3–53 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
4
Antonio Rosado et al.
Dubois and Prade [16] enumerate the two reasons for using fuzzy sets theory [51] to make querying more flexible. First, fuzzy sets provides a better representation of the user’s preferences. For example, in a query asking for some apartment “not too expensive and not too far from downtown”, the user may feel much more comfortable using linguistic terms instead of precisely specified numerical constraints. Moreover, these linguistic terms express exactly what the preferences of a user are; for example when an interval to which the price has to belong is imprecisely specified. The linguistic terms clearly suggest that there is a smooth transition between acceptable and unacceptable prices. Thus, we can have a price definitely matching or definitely not matching the user’s request, but also matching to a certain degree. Another important aspect of using fuzzy sets theory is a direct consequence of the previous. Namely, as soon as we have a matching degree, answers can be ranked according to the users’ requirements [16]. According to many authors such as Bosc and Pivert [5],[8], Kacprzyk and Zadrozny [21], Takahashi [46], Medina et al. [32], etc. there are two main lines of research in the use of fuzzy set theory in the DBMS context. The first one assumes a conventional database and, essentially, develops a fuzzy querying interface using fuzzy sets, possibility theory, fuzzy logic, etc. Among authors who have contributed to this research are Bosc and Pivert [3],[5],[7], Dubois and Prade [16], Tahani [44], Takahashi [45],[46], Kacprzyk, Zadrozny and Ziolkowski [27], Ribeiro and Moreira [39]. In Section 4, we will describe their works and propose a taxonomy for these approaches. The second line of research uses fuzzy or possibilistic elements for developing a fuzzy database model that accounts for imprecision and vagueness in data. Here, of course, querying constitutes also an important element of a model. Some relevant concepts are presented, e.g., in [1], [8], [9], [12], [13], [18], [20], [32], [34], [35], [42], [43]. These approaches are described in Section 5. There are also other issues in the use of fuzzy sets theory in relational databases such as efficiency of fuzzy queries execution; fuzzy functional dependencies/constraints, fuzzy logical databases, but they are beyond the scope here. In Section 2, we review the fundamental concepts of the relational data model which includes its main querying formalisms: the relational algebra, the relational calculus and the SQL language. Next, in Section 3, we review the main concepts of fuzzy sets theory that will be used in this paper. Sections 2 and 3 provide the theoretical base to make the paper self-contained. We examine in detail the main approaches proposed in the literature that are concerned with the first line of research mentioned above, i.e. flexible query languages for the crisp relational data model. We propose a taxonomy to organize these approaches, thus resulting in an overall picture of the main research done. The last part, Section 5, is devoted to the second line of research mentioned above, i.e. the flexible query languages for fuzzy relational databases. Again, a taxonomy for different approaches is proposed. Finally, in Section 6, we present some conclusions about this work.
Flexible Query Languages for Relational Databases: an Overview
5
2 Brief introduction to the relational data model A relational database is a collection of relations, defined according to the relational database model [14]. A relation may be understood as the relation schema or the relation instance. The relation schema has the following form: R(A1:D1, …, An:Dn)
(1)
where R is the name of the relation, Ai (1 ≤ i ≤ n) is the i-th attribute (which also may be called column or field) and Di (1 ≤ i ≤ n) is the domain corresponding to the attribute Ai. Each Di defines a set of values being possible values for an attribute. Often, we refer to a schema by just indicating the set of attribute names R(X), X = {Ai,…An}. A relation instance of given relation schema is a set of tuples, each composed of values of the attributes belonging to the relation schema: {< d1, …, dn> | (d1 ∈ D1), …,(dn ∈ Dn)}
(2)
where di (1 ≤ i ≤ n) is the value of the tuple corresponding to attribute Ai; this value must belong to the set Di. Usually the term relation instance is abbreviated to relation whenever there is no confusion with other aspects of the relation. A relation is here denoted R, its n attributes are denoted A1, …, An and D1, …, Dn are their domains. The m tuples of a relation are denoted t1, …, tm and dij represents the value of the j-th attribute in tuple ti. Relations are often referred to as tables, with columns and rows corresponding to attributes and tuples of the relation. The typical operations on data in a database include insertion, deletion, updating and retrieval. The latter is the most important for our considerations. Usually not all data is retrieved but only the data matching certain criteria is required. From the operational point of view there are two approaches to devising a query language, i.e., a language allowing to express the range of required data. The first one consists in the use of a restricted version of the predicate/relational calculus of mathematical logic to specify the requirements the retrieved data should meet. In this approach the actual way of data retrieval is completely left to the database management system employed. In the second approach, a query is a sequence of operations that should be executed on the database to obtain the required data. These operations correspond to the underlying data model, i.e., correspond to the operations in relational algebra. From the theoretical point of view both approaches are equivalent in the sense of their expressive power. In Section 2, we present a brief overview of both formalisms. In relational database management systems implementations usually a hybrid approach is adopted, notably exemplified by the SQL language. Whichever querying approach is assumed, the most important part of a query is a set of conditions (criteria) of which rows will be selected to be included in
6
Antonio Rosado et al.
an answer to the query. Thus, it is interesting to study the retrieval process from the perspective where a query is meant to define a prototype of data to be retrieved. Then, during the retrieval process for every row a matching degree of its content and the prototype is calculated. In the classical crisp approach this matching degree is binary: a row matches the prototype or not. In real situations, the description of the prototype may be imprecise and this leads to a partial matching degree. This line of reasoning, adopted by many authors, provides an interesting basis for the analysis of flexible (fuzzy) querying languages, which we will explore next. To clarify the understanding of some proposals in the literature, we will use, whenever necessary, the following relational scheme example: Employees(#emp, name, #dep, age, job, salary, commission, town)
(3)
Departments(#dep, budget, size, city)
(4)
with the instances of relations Employees and Departments given in Fig. 1: This example will also make possible to explain, in a simple way, the main concepts of relational algebra and relational calculus. 2.1 Relational algebra Relational algebra defines a set of operations to manipulate data in relations [37] The list of basic operations includes: • the usual set theoretic operations including the union, difference and Cartesian product. The Cartesian product is slightly modified to fit the database context and produces a relation whose scheme is a union of the schemes of the argument relations. • the selection, σP(R), gives the tuples of relation R that satisfy a Boolean expression P, which is defined over the scheme of R. • the projection, πY(R), gives a relation obtained when all attributes from the set X-Y are removed, where R(X) is the scheme of the relation R. Thus, the scheme of the resulting relation comprises only a subset Y of the set of attributes of R and some tuples of the original relation are also removed (those with identical values of the attributes belonging to Y).
Flexible Query Languages for Relational Databases: an Overview #dep 1 7 9 8 3
#emp 22 29 31 32 58 64 71 74 85 95
budget $130000 $90000 $110000 $50000 $40000
name Arthur John Mary Peter Barbara Mary Michael Jude Horatio Ken
size 30 20 11 7 11
#dep 1 3 7 1 7 7 8 9 9 3
7
city S. Francisco New York Washington S. Francisco New York
age 30 35 40 39 39 27 30 35 50 55
job programmer accounter sales manager systems analyst marketing manager product manager research assistant secretary technical assistant controller
sal commissiontown $1500 $500 New York $1800 $400 Boston $2300 $200 San Francisco $2000 $300 New York $2500 $500 Los Angeles $2000 $400 New York $1500 $100 S. Diego $1600 $300 New York $2400 $400 New York $2800 $500 Boston
Fig. 1. Example of a crisp relational database Other operations can also be used but they could be expressed in terms of the five operations mentioned above. For example, the popular join operation, joinAθB(R,S), is a combination of the Cartesian product and the selection operation. Another operation, often discussed within the subject of “fuzzification”, is the division R ÷ S. If X and Y are the schemes of R and S, respectively, with Y ⊂ X, then R ÷ S gives the maximal (in the sense of “⊂”) relation T, having the scheme X-Y, such that T × S ⊂ R, i.e.,: R ÷ S = {t : ∀u ∈ S, (t,u) ∈ R}
(5)
The division operation may also be expressed as a combination of the projection, Cartesian product and difference operations. We complete this section by providing an example of a query expressed in the relational algebra. The query [7]: Q1 - Find the employees younger than 35 who work in a department whose budget is higher than $100000 may be expressed in relational algebra as:
π #emp ( join Emp.#dep = Dep.#dep (σ age<35 ( Emp), σ budget >100000 ( Dep)))
(6)
8
Antonio Rosado et al.
2.2 Relational calculus Relational calculus comprises domain relational calculus (DRC) and tuple relational calculus (TRC) [37]. DRC and TRC are declarative query languages based on the first-order predicate logic. The main idea is to describe what is sought rather than to define how to get it, in a similar fashion as in the relational algebra. Next, we briefly review the DRC language because it is the base for several flexible query languages (FQLs) described in Sections 4 and 5, respectively. A DRC query is an expression based on the first order predicate calculus language. The general form of such a query is as follows: {(x1,…,xn) | φ(x1,…,xn) }
(7)
where φ(x1,…,xn) is a formula of the language. An answer to such a query is a set (possibly empty) of tuples (a1,…,an) such that when substituting ai’s for xi’s in φ a true formula is obtained. The building blocks of formulae are atomic formulae. In the DRC two classes of atomic formulae are usually distinguished: R(x1, x2, …, xn)
(8)
x1 θ x 2 where xi (1 ≤ i ≤ n) is either a domain variable or a constant; R is an n-ary predicate ; θ is a comparison operator from the set {<, >, =, ≤, ≥, ≠}. Finally, a formula is: • • • •
an atomic formula, ¬ψ, ψ1 ∧ ψ2, ψ1∨ ψ2, ψ1 ⇒ ψ2 ∃ x (ψ (x)) ∀ x (ψ (x))
(9)
where ψ, ψ1, ψ2 are formulas and ψ(x) is a formula containing a domain variable x. An example of a DRC query equivalent to the example of the relational algebra query of Section 2.1 (see Eq. (6)) is the following:
EMP( x1 , x2 , x3 , x4 ,...) ∧ x4 < 35 ∧ (10) ( x1 ) | ∃x2 , x3 , x4 , x5 , x6 DEP( x5 , x6 ,...) ∧ x6 > 100000 ∧ x5 = x3 In the tuple relational calculus (TRC) we use the syntax similar to Eqs. (8)(9) but replacing the domain variables with tuple variables. For example, (10) may be expressed in TRC as:
9
Flexible Query Languages for Relational Databases: an Overview {t | t ∈ Employees ∧ t.age < 35 ∧ ∃ s ∈ Departments (s.#dep = t.#dep ∧ s.budget > 100000)}
(11)
2.3 SQL SQL is a de facto industry standard command language for the relational database management systems. SQL has commands to deal with all aspects concerning the creation, maintenance and use of a database such as the creation of tables, insertion of rows, querying the database, security issues, etc. In this section, we are only interested in what SQL provides for querying a database. The syntax of a basic query in SQL is [37]: SELECT select_list FROM from_list WHERE conditions
(12)
Such a query retrieves required data from some tables and builds a new table. The select_list specifies the expressions (often, just column names) whose values are to populate columns of the new table. The columns used in these expressions have to be listed in the from_list. The expressions of the SELECT clause are calculated only for the rows, from the FROM clause, that meet the conditions specified in the WHERE clause. The condition is a Boolean combination of atomic conditions created using the logical connectives AND, OR, NOT. Each atomic condition has the following syntax: “expression op expression”, where “op” is one of the comparison operators (<, <=, =, <>, >=, >) and “expression” is a column name, a constant or an expression (numeric expression or string expression). A basic query (12) corresponds to an expression in relational algebra involving the operations of projection, selection and Cartesian product. For example, the following query: Q2 - Find the names and age of all employees, is expressed in SQL as: SELECT name, age FROM Employees
(13)
and the result is a table with two columns (name,age), and as many rows as in the table Employees. A way to extend the basic form of a query is to use nested queries (subqueries). Usually, the nested queries are used along with the set operators IN, NOT IN, EXISTS, NOT EXISTS, op ANY, op ALL as exemplified by the query:
10
Antonio Rosado et al. Q3 - Find the names of employees who work in New York may be expressed as:
SELECT name FROM Employees WHERE #dep IN (SELECT #dep FROM Departments WHERE city = ‘New York’)
(14)
The subquery retrieves the set of departments that are located in New York. The main query retrieves the names of employees such that their department is in this set. The set operator IN allows to test whether a value belongs to a set or not.
3 A brief introduction to fuzzy sets theory Fuzzy sets theory [51] is an attempt to model an inherent vagueness of natural language. Almost any concept expressed in natural language, like young people, implies that elements of the universe of discourse (the particular people) are young to a certain degree. Note that the concept of young is context dependent. Such a graduality is modeled by a membership function µA that for each person x assigns a value µA(x) from the interval [0,1], representing the degree to which person x belongs to set A. In our example, µYOUNG(x) represents the degree to which x is considered young. The cardinality |A| of a fuzzy set A, defined on a finite universe set X, is given by the sum of the membership values of all elements of X in A, and is sometimes called scalar cardinality to distinguish from other types of cardinality (see e.g. [31])
A=
∑ µ A (x )
(15)
x∈X
The relative cardinality ||A|| of a fuzzy set A, in a finite universe set X, is defined by:
A =
A X
(16)
where |X| is the cardinality of the universal set X. The counterparts of the classic operations of the complement, union and intersection for fuzzy sets A and B, are defined as follows:
µ¬A(x) = 1 - µA(x)
(17)
Flexible Query Languages for Relational Databases: an Overview
11
µA∪B(x) = max [µA(x), µB(x)]
(18)
µA∩B(x) = min [µA(x), µB(x)]
(19)
The classic correspondence of set theoretical operations and logical connectives is preserved. Thus, (17)-(19) provide also interpretation for the connectives of negation, disjunction and conjunction. Several fuzzy implication operators have also been proposed in the literature [17]. The most commonly used are: Kleene-Dienes: I(x,y) = max(1 – x, y)
(20)
Lukasiewicz: I(x,y) = min(1,(1-x+ y))
(21)
1 if x ≤ y Gödel: I ( x, y ) = y otherwise
(22)
Goguen: I(x,y) = min(y/x,1)
(23)
Fuzzy sets theory was conceived primarily as a formalism to represent the meaning of natural language expressions. In the following subsections we will briefly review some concepts relevant for this topic. 3.1 Linguistic variable Basically, a linguistic variable is a variable assuming linguistic values instead of numerical values. Formally, a linguistic variable [53] is a quintuple (H, T, U, G, M) where H is the name of the variable, T is the set of linguistic names (called terms) that can be assigned to the variable; U is the universe of values that are used to define the meaning M of each linguistic value in T; and G is a grammar that is used to specify the values allowed in T. The meaning M(X) of a term X ∈ T, is specified as a fuzzy subset in U. The terms may be atomic terms such as “young” or composite terms, which result, for example, from applying modifiers (see next subsection) and logical connectives to atomic terms. For example [53], a linguistic variable called age (H = age) may have the term set T = {old, very old, not old, more or less young, quite young, not very old and not very young, …}, which, for simplicity reasons, is defined here in an informal way, without defining the grammar G. In this example, young and old are the atomic terms. The universe of discourse might be U = [0,100] and the meaning of the term young, M(young), could be given by a fuzzy set, such that:
12
Antonio Rosado et al.
u ∈ [0,25 ] 1, −1 ,u∈U µ M ( young ) (u ) = u − 25 2 , u ∈ ]25,100 ] 1+ 5
(24)
3.2 Modifiers A linguistic modifier can be modeled by using an operator that acts on the fuzzy set corresponding to the linguistic term to which the modifier is applied. For example [41] the linguistic modifier very in the linguistic expression “very young” intensifies the meaning expressed by the fuzzy term young. Hence, the effect of very is to decrease the membership of the values belonging to the fuzzy set YOUNG. The concentration operator can produce this effect:
µCON(A)(x) = µ2A(x)
(25)
Conversely, the dilation operator can be used for defining modifiers, as for example slightly and it is modeled as:
µ DIL( A) (x ) = µ 1A/ 2 (x )
(26)
Other usual modifier is not that is modeled by the complement operator. There are many more operators used to model linguistic modifiers, cf., e.g., [28]. 3.3 Fuzzy (linguistic) quantifiers Classical logic recognizes two quantifiers expressing that all objects posses certain property (general quantifier) or that at least one object possesses certain property (existential quantifier), respectively. However, natural languages offer many more forms of quantifiers. For example, quite often one says that most of the objects possess certain property. Basically, there are two types o fuzzy (linguistic) quantifiers [50], [54]: absolute - such as “approximately 3” and “several” and proportional such as “most” and “a few”. There are also two general types of propositions referring to linguistic quantifiers: 1. Q X’s are A’s (type I) 2. Q B’s are A’s (type II)
Flexible Query Languages for Relational Databases: an Overview
13
where Q is a linguistic quantifier, and A and B are fuzzy sets modeling certain fuzzy properties of the objects of the universe X. In what follows we briefly discuss how linguistic quantifiers may be formalized. Zadeh’s calculus of linguistically quantified propositions Zadeh [54] proposed an interpretation for fuzzy quantified statements of both types I and II based on the concepts of cardinality (Eq. (15)) and relative cardinality (Eq. (16)) of a fuzzy set. A fuzzy proposition Q X’s are A’s has the truth degree T that is computed using the following equations [54]: T = Qabsolute(|A|) = Qabsolute(∑i µA(xi))
(27)
where Q is respectively an absolute and a relative quantifier. For fuzzy propositions Q B’s are A’s, where both A and B are fuzzy sets, we have [54]: T = Qabsolute (|A∩B|) = Qabsolute (∑i µA(xi) ∧ µB(xi))
A∩ B T = Qrelative B
µ (x ) ∧ µ B (xi ) = Qrelative ∑i A i ∑i µ B (xi )
(28) (29)
OWA operators and Yager’s calculus of linguistically quantified propositions Yager [50] proposed the use of Ordered Weighted Averaging (OWA) operators for the evaluation of linguistically quantified propositions to overcomee some problems of the Zadeh’s proposal (see in [7] an example comparing Zadeh’s approach with Yager’s approach). An OWA operator of dimension n is a mapping f that performs an aggregation of its n arguments a1, …, an [50], such that:
f (a1 ,..., an ) =
n
∑bjwj
(30)
j =1
where ai ∈ [0,1], bj is the j-th largest from among ai, and wj (wj ∈ [0,1]) are weights such that ∑i wi = 1. The classical “AND” and “OR” may be expressed as special OWA operators: for wn = 1 and wj = 0 (∀j < n) we obtain f1(a1, …, an) = bn = mini ai for w1 = 1 and wj = 0 (∀j >1) we obtain f2(a1, …, an) = b1 = maxi ai
(31)
14
Antonio Rosado et al.
Moreover, any OWA operator lies somewhere between the “OR” operator and the “AND” operator [50] in the sense that: mini ai ≤ f(a1, …, an) ≤ maxi ai
(32)
It is also possible to define an OWA operator which may be interpreted as a linguistic quantifier. Yager [50] proposed a scheme of defining an OWA operator corresponding to a linguistic quantifier in the sense of Zadeh. Namely, starting with a linguistic quantifier Q that is monotone and regular (Q(0) = 0; Q(1) = 1) we set the weights of corresponding OWA operator as follows: wi = Qrelative(i/n) – Qrelative((i-1)/n) ∀ i=1,…,n
(33)
Then, the degree of truth T of a quantified proposition “Q X’s are A” is computed using this OWA operator [50]: T = f (µA(x1), …, µA(xn))
(34)
3.4 Possibility distributions Consider a vague proposition “X is young”. This is an imprecise proposition because it does not assign a particular value for X (more precisely, for X’s attribute age). Instead, it associates, with each possible value of X its possibility degree, a number in the interval [0,1] [29]. We can say that proposition p = “X is young” induces a possibility distribution π (the notation πX is often used to indicate what variable is considered) on the domain of the attribute age: X is young → π = YOUNG
(35)
or, equivalently: ∀u ∈ U
π (u) = µYOUNG(u)
(36)
that is, the possibility that a certain u∈U is an actual value of X is equal to the u’s membership degree to the fuzzy set YOUNG, which models the linguistic term young. Knowing the possibility distribution πX we may be also interested in determining what is the possibility that X’s value belongs to a set A⊆U. This leads to the concept of the possibility measure, i.e., a function Π such that:
Flexible Query Languages for Relational Databases: an Overview Π: 2 → [0,1] U
15
(37)
From the postulated properties of possibility measures it is assumed that (in fact, usually we start with the concept and properties of the possibility measure and only then the notion of the possibility distribution is introduced):
Π ( A) = sup π (u ) u∈A
(38)
The possibility measure alone does not tell us enough about the location of the actual value of X: outside or inside A. Thus, it is usually argued that it should be accompanied by the possibility measure of the complement of A. More precisely, the necessity measure, Ν, is defined as, expressing the “impossibility” of the set A:
( )
N ( A) = 1 − Π A = inf π (u ) u∈A
(39)
The formulae of Eqs. (38)-(39) are extended to the case where A is a fuzzy set in the following way:
Poss( X is A) = Π ( A) = sup min (π (u ), µ A (u )) u∈U
(40)
and:
Nec( X is A) = N ( A) = inf max(1 − π (u ), µ A (u )) u∈U
(41)
Now, if we know that the possibility distribution of the X’s value is π then the degree to which the actual value of X belongs to A (often denoted as “X is A”) belongs to the interval [Ν(A), Π(A)]. Eqs. (40)-(41) for the possibility and necessity measures are directly employed when the matching degree is computed in the context of querying possibilistic fuzzy databases - see Section 5. Actually, the interpretation of more advanced queries calls for more sophisticated formulae. Namely, let us assume that we have two variables X and Y, defined on the same universe U, and we know the possibility distributions of their values, πX and πY, respectively. The question is: what is the possibility that the actual values of these variables are equal. In order to answer this question consistently we proceed as follows. First, we observe that πX and πY jointly represent a possibility distribution πXY on U x U:
πXY(u,w) = min (πX(u), πY(w))
(42)
Antonio Rosado et al.
16
Second, the possibility (necessity) measure associated with πXY will be denoted ΠXY (ΝXY). Thirdly, the answers to our question are the values of the possibility and necessity measures, for the set of pairs of identical elements from U, i.e., Poss( X = Y ) = Π XY ({(u , u ) : u ∈ U }) = sup min(π X (u ), π Y (u ))
(43)
Nec( X = Y ) = N XY ({(u, u ) : u ∈ U }) = inf max(1 − π X (u ),1 − π Y (u ))
(44)
u∈U
u∈U
In Section 5, we discuss the Eqs. (40)-(41) for a more general case, when, roughly speaking, the “is” and “=” operators are replaced with other crisp or fuzzy relational operators. Now, we would like to distinguish another concept, useful in the context of a fuzzy database querying: the possibility distributions’ similarity. If we know the possibility distributions for two variables, X and Y, we can be interested in finding out how similar both distributions are. Similarity is meant here in a very broad sense. Obviously, Eq. (43) provides an assessment of this similarity, but other measures are also applicable. In Section 5 we discuss this concept in more detail. 3.5 Fuzzy relations A crisp relation R(x1, x2, ..., xn) defined on crisp sets X1, X2, …, Xn is a subset of the Cartesian product X1 × X2 × … × Xn. Similarly, a fuzzy relation R is a fuzzy set defined on the Cartesian product X1 × X2 × … × Xn. The membership degrees µR(x1, x2, …, xn) represent the strength of the relation between the elements of the tuples (x1, x2, …, xn). In the relational database terminology we will say that R is defined over schema (X1, X2, …, Xn). The composition of two binary crisp relations P(X,Y) and Q(Y,Z), denoted P(X,Y) ° Q(Y,Z), is a crisp binary relation R(X,Z) such that: R(X,Z) = {(x,z) ∈ X × Z | ∃ y ∈ Y (x,y) ∈ P ∧ (y,z) ∈ Q}
(45)
The composition of two fuzzy binary relations P(X,Y) and Q(Y,Z) may be defined in several ways. The most commonly used definitions are the following [29]:
µP
Q
(x, z ) = max min[µ P (x, y ), µQ ( y, z )] y∈Y
[
]
µ P•Q (x, z ) = max µ P (x, y )× µ Q ( y, z ) y∈Y
(46) (47)
respectively for max-min composition and max-product composition. A fuzzy relation R defined on X × X, verifying the following properties: [29]
Flexible Query Languages for Relational Databases: an Overview
17
1. reflexivity: ∀x ∈ X, µR(x,x) = 1
(48)
2. symmetry: ∀x,y ∈ X, µR(x,y) = µR(y,x)
(49)
3. max-min transitivity: ∀x,z ∈ X, µR(x,z) ≥ • max min[µR(x,y), µR(y,z)] y∈Y
(50)
is called a similarity relation R(X,X). If R(X,X) is only reflexive and symmetric then R is called a proximity relation [29]. An algebra for fuzzy relations Relational algebra (see Section 2) may be extended in order to provide the same type of operations for fuzzy relations. Next, we present the definitions for the operations of such an extended algebra [3]. The set operations union, intersection, difference and Cartesian product of two fuzzy relations R and S are direct applications of operations on fuzzy sets, i.e.: µ R ∪ S (x) = max (µR(x), µS(x))
(51)
µ R ∩ S (x) = min (µR(x), µS(x))
(52)
µ R - S (x) = µ R ∩ S (x) = min (µR(x), 1- µS(x))
(53)
µ R x S (xy) = min (µR(x), µS(y))
(54)
_
where x and y are tuples of relations R and S. Moreover, the operations selection, projection and join are defined as:
µσ P (R ) (x ) = min (µ R (x ), µ P (x ))
(55)
µπ Z (R ) (z ) = max µ R (zv )
(56)
v
where P is a fuzzy condition, z and v are tuples defined over schema Z and V, respectively, such that Z ∪ V = X and Z ∩ V = ∅; A and B are subsets of X and Y respectively, and θ is a fuzzy comparator, i.e. it is a fuzzy relation, defined on sets A and B, such as approximately equal. Let us briefly explain the three formulas above. First, for selecting the tuples x in a fuzzy relation R that satisfy a fuzzy condition P, we compute the matching degree of each tuple against P, that is, µP(x). The membership degree of each x in the resulting relation must take into account two values: µP(x) and its original
18
Antonio Rosado et al.
membership degree µR(x). Projecting a fuzzy relation R through one of its subschema Z, that is, Z ⊂ X, requires forming the sub-tuples z corresponding to Z, and considering for their membership degrees the highest membership degree of the tuples x that include z as a sub-tuple. Finally, joining two fuzzy relations R and S using a fuzzy condition A θ B requires concatenation of pairs of tuples x and y. The membership degree of each new tuple xy is the minimum of its matching degree related to A θ B and the original membership values of tuples x and y. The relational algebra operation of division (see Eq. (5)) can also be extended using the fuzzy sets (fuzzy relations) instead of the crisp sets (crisp relations). Dubois and Prade [16] proposed the following extension: µR÷S(t) = minu µS(u) → µR(t,u)
(57)
where the symbol → denotes a fuzzy logic implication. 3.6 PRUF Above we summarized how the semantics of propositions may be expressed in terms of possibility distributions (more generally, possibility theory). So far, we only discussed the simple propositions exemplified by Eq. (35), however, natural languages are syntactically much richer. Zadeh [53] classified fuzzy propositions of natural languages in five types: simple fuzzy propositions (type I), modified fuzzy propositions (type II), composed fuzzy propositions (type III), quantified fuzzy propositions (type IV), and qualified fuzzy propositions (type V). He proposed a semantic interpretation for each type in terms of the possibility distribution in a way analogous to Eq. (36). In Fig. 2 below we collected examples of propositions belonging to each type and the corresponding induced possibility distribution formulation. Type I II III IV V
Example Maria is young Bob is very tall Maria is young and Bob is very tall Most students are young It is quite true (probable or impossible) that Maria is young
Induced possibility distribution πage = YOUNG πheight = VERY_TALL π(age, height) = YOUNG × VERY_TALL πcount(age) = MOST, YOUNG πage = QUITE_TRUE, YOUNG
Fig. 2. Classification of fuzzy propositions [53]
Flexible Query Languages for Relational Databases: an Overview
19
We may observe that: QUITE_TRUE, YOUNG denotes the function µQUITE_TRUE(µYOUNG(x)); YOUNG × VERY_TALL is a Cartesian product of fuzzy set YOUNG and the modified fuzzy set VERY_TALL; and MOST is a quantifier.
4 Taxonomy of flexible query languages (FQLs) for crisp relational databases In this section we propose a taxonomy for the main proposals found in the literature related to FQLs for crisp relational databases. The purpose of proposing this taxonomy is to offer a structured view about the topic and to highlight main differences and similarities between various approaches. Further, we believe that this taxonomy can offer some guidance and clarification about the most relevant proposals in this research area. 4.1 Taxonomy of FQLs for crisp databases In Fig. 3 we present the main approaches in this topic grouped in four categories. Group 1, denoted basic fuzzy predicates, includes the first approach that used fuzzy predicates in queries. Group 2, denoted flexible aggregation operators, presents the proposals that study flexible aggregation of partial matching degrees via linguistic quantifiers and importance weights. Group 3, denoted SQL extensions, presents proposals introducing elements of fuzzy the querying paradigm in SQL. Finally, Group 4, denoted PRUF and flexible querying, describes proposals that focus on the interpretation of natural language expressions for the purposes of querying. FQL Taxonomy
Proposals of FQLs for crisp relational databases G1. Basic fuzzy predicates P1. Tahani [44] G2. Flexible aggregation P2. Kacprzyk, Zadrozny and Ziolkowski operators [26],[27] P3. Bosc and Pivert [6] P4. Dubois and Prade [16] G3. SQL extensions P5. (SQLf) Bosc and Pivert [3], [5], [7] P6. (FQUERY) Kacprzyk and Zadrozny [21] G4. PRUF and flexible P7. Takahashi [45], [46] querying Fig. 4. Main approaches to Flexible Query Languages (FQLs) for crisp relational databases
20
Antonio Rosado et al.
Since the SQL’s SELECT command is the standard for the querying of crisp relational databases, we will also use it for illustrative purposes and to clearly distinguish the taxonomy groups. The first two groups (G1-G2) of proposals, basic fuzzy predicates and flexible aggregation operators, have a theoretical character and essentially extend the WHERE clause’s condition of the SELECT command, by incorporating linguistic (fuzzy) terms and using flexible aggregation operators (connectives). The third group (G3) of proposals, SQL extensions, comprises more practical approaches that embed the fuzzy predicates in the syntax of the standard SQL. For example, one of the proposals (FQUERY for Access [21]) implements fuzzy predicates in the WHERE clause and another proposal (SQLf [7]) specifies a new language that extends SQL by incorporating fuzzy predicates not only in the WHERE clause but wherever it makes sense. The fourth group (G4) of proposals, natural language querying languages, instead of modifying the SQL’s SELECT command proposes a set of natural language query types that includes an adequate subset of fuzzy propositions. The proposals included in G4 are based on the PRUF language [53], and the mismatch between the natural language queries and information stored in the crisp relational database is bridged by translation rules that perform the respective conversion. Next, we will present a summary of the proposals by the authors listed under each group of the taxonomy. We will use Px to refer to each author’s proposal, where x is the sequential number of the author in the taxonomy. The description by thematic group will highlight the resemblances of approaches in terms of the topic of study and will simplify the readers’ understanding of such a vast literature. In spite of the fact that many other authors had also contributed to the advancement of flexible querying in crisp databases, in this work we selected the most representative for each topic. G1 Basic fuzzy predicates P1. Tahani [44] was the first author to propose the use of fuzzy sets theory to improve the flexibility of crisp database queries. He proposed a formal approach and architecture to deal with simple fuzzy queries for crisp relational databases. The queries are based on the SEQUEL language. The idea may be best illustrated by the query: Q4 – Find the names and department numbers of employees who are young and have a high salary OR those who are young and have low commission Using our database scheme defined in Eqs. (3)-(4) and Tahani's proposal such a query may be expressed as:
Flexible Query Languages for Relational Databases: an Overview SELECT name, # dep FROM Employees WHERE age=YOUNG AND (sal=HIGH OR commission=LOW)
21 (58)
Thus, Tahani proposed to use in the query condition vague terms typical for natural language. Syntactically, they are represented as fuzzy predicates. Their semantics is provided by appropriate fuzzy sets. Then, the main question is how the matching degree for each particular row is computed. For that purpose Tahani defines the matching function γ. For a tuple ti and a simple query P of type A = v, where A is an attribute (e.g., age) and v is a vague (fuzzy) term (e.g., young), the value of the function γ is:
γ (P,ti) = µv(u)
(59)
where u is the value of the attribute A in tuple ti. For example:
γ(AGE = young, <22, Arthur, 1, 30, programmer, 1500, 50,
(60)
New York>) = 0.5 if µyoung(30)=0.5. The matching function γ for complex queries involving logical connectives like age=YOUNG AND (sal=HIGH OR commission=LOW) is:
γ(P1 AND P2, ti) = min (γ(P1, ti), γ(P2, ti))
(61)
γ (P1 OR P2, ti) = max (γ(P1, ti), γ(P2, ti))
(62)
γ(¬ P, ti) = 1-γ(P, ti)
(63)
where P, P1, P2 are queries. Thus, the logical connectives in the queries are interpreted as the original Zadeh’s fuzzy connectives. G2 Flexible aggregation operators P2. Kacprzyk, Zadrozny and Ziolkowski [26], [27] were the first to propose the aggregation of partial queries (predicates, conditions) to be guided by a linguistic quantifier (see Section 3). Thus, they proposed to extend the querying language so as the selection operator’s condition may be expressed as: P = Q out of {P1, …, Pn}
(64)
22
Antonio Rosado et al.
where Q is a linguistic (fuzzy) quantifier and Pi are partial conditions to be aggregated. Thus, the overall matching degree may be computed using any of the approaches discussed in Section 3. In [27] and [26] the original Zadeh’s approach has been adopted, but later in [22] the authors used the OWA operators as the model for the linguistic quantifier. Both type I and type II linguistically quantified propositions (see Section 3) were studied in this context by the authors. In the latter case query (64) may be extended to: P = Q important out of {P1, …, Pn},
(65)
where the importance is represented by a fuzzy set of Pi’s in the sense that the value of the membership function of given Pi is equal to its importance weight. Thus, in (65) importance corresponds to B in “Q B’s are A’s” of Section 3. P3. Another scheme for the aggregation of fuzzy conditions of varying importance has been studied by Bosc and Pivert [6]. They proposed a fuzzy operator for the hierarchical aggregation of fuzzy conditions, which extends the concept of hierarchical aggregation given by Lacroix and Lavency for crisp conditions [30]. Lacroix and Lavency proposed to extend the concept of classical crisp queries in the following way. A query has two parts: a selection part, S, and a set of crisp conditions, PRF, called preferences. The semantic of this query is the following: select the tuples satisfying S and rank them according to PRF. More precisely, if there are no tuples satisfying condition S then the answer to the query is empty. Otherwise, the answer comprises the tuples that verify S and at the same time best satisfy PRF. In the latter case, various assumptions on the interrelation of the conditions belonging to the PRF may be made. Two cases are considered: (a) the conditions are equally important, (b) there is a (linear) hierarchy of conditions those higher in hierarchy are more important. Thus, in the second case we have the importance of conditions imposed not by numerical weights, but by their position in the hierarchy. The ranking of the tuples depends on what assumption is made: (a) or (b). In the first case, the count of the conditions in PRF that are satisfied by a tuple is taken into account. In the second case, the lexicographic ordering of the tuples according to their fulfillment of particular conditions belonging to PRF (taken in order imposed by the hierarchy) is employed. Bosc and Pivert [6] proposed a fuzzy operator N to model the hierarchical aggregation described above, in which the contribution of a condition Pi ∈ PRF is less or equal than the contribution of conditions higher in the hierarchy. Let us assume that the conditions are ordered according to the hierarchy, i.e., if i<j then Pi is higher in the hierarchy (is more important) than Pj. The fuzzy operator proposed is defined as a combination of two operators. The first, denoted below with µ 'Pi ,
Flexible Query Languages for Relational Databases: an Overview
23
limits the contribution of condition Pj relatively to the contributions of all preceding conditions Pi (i<j), while the second combines all contributions to obtain the final value for the aggregation of the fuzzy conditions:
µ
N ( P1…Pn )
(t ) =
∑in=1 µ´Pi (t ) n
(66)
where µ´ Pi (t ) = min ( j ≤i ) ( µ Pj (t )). This operator expresses the degree to which a tuple t satisfies the hierarchical aggregation of the fuzzy conditions. The authors also considered another version of their operator replacing the arithmetic mean by the weighted average. Bosc and Pivert adopt a different interpretation of hierarchy of conditions than originally assumed by Lacroix and Lavency. Namely, in the latter case, if no tuple satisfies a condition from, e.g., the k-th level of the hierarchy, then the conditions of the lower levels do play a role in the ranking of tuples. In the former approach, all these lower levels are neglected. P4. Dubois and Prade (cf. [16]) studied the question of conditions Pi with varying degrees of importance forming together a compound condition P via the conjunction. The first model considers some importance weight wi for each elementary condition Pi and the matching degree of the weighted condition Pi against a value u of the corresponding attribute is given by the following equation:
µ P* (u ) = max(µ Pi (u ) ,1 − wi ) i
(67)
where Pi* denotes the condition Pi with an importance associated to it. Then, the matching degree of the condition P is calculated using the standard min operator:
µ P (u ) = min µ P* (u ) = min max(µ Pi (u ) ,1 − wi ) i =1,...,n
i
i =1,...,n
(68)
When the importance is minimal (wi = 0), the condition Pi is not considered in the evaluation. On the other hand, with wi = 1, the evaluation of condition Pi is vital for the evaluation of condition P. This model has been refined (see in [16]) to deal with a variable importance wi – depending on the matching degree of the associated elementary condition. For example, in a specific context it may be useful to assume wi constant for relatively high satisfaction of the elementary condition but for extremely low satisfaction it should be stronger reflected in the overall matching degree by automatically increasing wi. For example, if we want to buy a car and one of our criterion is to have a moderate price, but it is not our primary criterion
Antonio Rosado et al.
24
(condition), then we assume an importance weight smaller than 1.0. However, if a particular car has a very high price, the price criterion becomes more important (wi = 1) and the car is rejected by having a very low satisfaction membership value. The second model, originally proposed by Yager in 1984 (see in [16]) considers a threshold αi for each elementary condition Pi. If condition Pi is satisfied to a degree above threshold αi, that is, µPi(u) ≥ αi, the resulting partial matching degree becomes 1, i.e., µ * (u ) = 1. On the other hand, if the threshold is not reached, i.e., Pi
µPi(u) < αi, then we may consider two ways for the evaluation of the condition: (a) µ P* (u ) = µPi(u) or (b) µ P* (u ) = i
i
µ Pi (u )
αi
. It turns out that both ways may be
expressed by a formula:
µ P (u ) = min µ P* (u ) = min α i → µ Pi (u ) i =1,...,n
i
i =1,...,n
(69)
where → is the implication logical operation. Then, using the Gödel implication (see Eq. (22)) and the Goguen implication (see Eq. (23)) we can obtain (a) and (b), respectively. Note that the first model of importance proposed by Dubois and Prade (see in [16]) and formalized by Eq. (67) is also covered by the general formula of (69) when the Kleene-Dienes implication (see Eq. (20)) is assumed. Still another model of importance applicable to the aggregation of partial matching has been proposed by Dubois and Prade [16] in which they use conditional requirements P1ÆP2 to provide an interpretation for the hierarchical aggregation of fuzzy predicates. The authors consider a similar context to that of the paper by Lacroix and Lavency [30]. An overall condition P is considered to be a sequence of elementary conditions Pi=1,n accompanied by importance weights (called here priorities). It is interpreted in such a way that “P1 should be satisfied (with priority 1) and among the solutions meeting P1 (if any) the ones satisfying P2 are preferred (with priority α2), and among those satisfying both P1 and P2, those satisfying P3 are preferred with priority α3 (α3 < α2 < 1) and so on”. This may be interpreted as nested implication operators: P1Æ(P2Æ(P3Æ… . The overall matching degree (the results of the aggregation) may be thus represented by the following membership function defining a fuzzy set of elements (rows) satisfying P (when P consists of 3 partial predicates):
µ P1 (u ), max(µ P2 (u ),1 − min(µ P1 (u ),α 2 )), µ P (u ) = min max(µ P (u ),1 − min(µ P (u ), µ P (u ),α 3 )) 3 1 2
(70)
Flexible Query Languages for Relational Databases: an Overview where
min(µ P1 (u ),α 2 )
and
min(µ P1 (u ), µ P2 (u ),α 3 )
are
priority
25 levels
(corresponding to wi in Eq. (67)) of fuzzy conditions P2 and P3, respectively. Hence, concerning the predicate P2, its priority is α2 if P1 is fully satisfied and is zero if P1 is not at all satisfied, which reflects the fact that P2 is conditioned by P1. This is another example of the variable importance weight, but this time depending on the satisfaction of the “preceding” partial condition. Notice, that the hierarchy (nesting) of the conditions is here used in the same sense as in Bosc and Pivert's approach rather than in Lacroix and Lavency's sense. G3. SQL extensions In G1 and G2 we reviewed proposals for the application of fuzzy conditions, in the context of crisp relational database querying. In terms of relational algebra we considered them as extensions to the selection operation making it possible to use linguistic terms and flexible aggregation operators in conditions. Here, we discuss two proposals of the most popular extensions of a de facto standard querying language of relational databases, i.e., SQL. Notice that already in Section G1 we discussed Tahani’s approach directly referring to SQL (or SEQUEL). However, both approaches discussed here, the SQLf and FQUERY for Access, differ from Tahani’s approach. The first one is an extension to SQL syntax introducing linguistic (fuzzy) terms, wherever it makes sense, and the second is an example of the implementation of a specific “fuzzy extension” to SQL for Microsoft Access®, a popular desktop DBMS. P5. The previously discussed approaches concentrated on the fuzzification of conditions appearing in the WHERE clause of the SQL’s SELECT instruction. Bosc and Pivert [5], [7], [8] proposed a new language, called SQLf, that is a much more comprehensive and complete fuzzy extension to the crisp SQL language. In SQLf linguistic terms may appear as: 1. Fuzzy values, relations and quantifiers (as aggregation operators) in the WHERE and HAVING clauses; 2. The linguistic quantifiers in addition to the classical EXISTS and ALL quantifiers used together with subqueries. Moreover, the authors observe that in case of complex SQL queries involving linguistic (fuzzy) terms the partial results are fuzzy relations. Thus, all operations of relational algebra (implicitly or explicitly used in SQL’s SELECT instruction) have to be redefined to properly process fuzzy relations. Hence, the union, intersection and difference operations are considered. Special attention is also paid to the division operation which may be interpreted in a different way due to many possible versions of the implication available in fuzzy logic. Other typical operations for SQL require a redefinition, including the partition of relations (fuzzy
26
Antonio Rosado et al.
relations) with the operator (clause) GROUP BY and the operators IN and NOT IN used together with subqueries. All the features of SQL just mentioned were extended in such a way so as to preserve the equivalences that occur in the crisp SQL. To illustrate this work, we describe below an extension of the nesting operator IN and how the partition of relations is adapted to the case of a fuzzy relation in SQLf. SQLf allows fuzzy conditions as described in G2. For example, the query [7]: Q5 - Find the young employees who work in a high-budget department can be expressed in SQLf as [7]: SELECT #emp FROM Employees WHERE age = ‘young’ AND #dep IN (SELECT #dep FROM Departments WHERE budget = ‘high’)
(71)
where the result of the subquery is a fuzzy relation. Consequently, the meaning of the condition a IN E, where a is an element and E is a crisp relation, must be extended to deal with fuzzy relations. Fuzzy sets theory suggests the following definition for the predicate IN [7]:
µ IN (a, A) =
sup
b∈support (A )
min (µ = (a, b ), µ A (b ))
(72)
where a is an element and A is a fuzzy set. In case of the classical identity relation "=" this boils down to µIN(a,A)=µA(a). Bosc and Pivert [7] propose to obtain more flexibility by replacing “=” with another operator referring to the similarity between elements. This leads to a concept of a fuzzy membership, INF. For example, the query [7]: Q6 - Find the employees who work in a department whose budget is about 1000 times their own salary can be expressed in SQLf as [7]: SELECT #emp FROM Employees WHERE sal * 1000 INF (SELECT budget FROM Departments WHERE #dep = Employees.#dep)
(73)
In SQLf, the HAVING clause is extended in two ways: with a fuzzy condition and/or fuzzy quantified proposition. As an example of a fuzzy condition, we can
Flexible Query Languages for Relational Databases: an Overview
27
replace the identity operator “=” with the similarity operator ≈. As an example of using fuzzy quantified propositions, the query [7]: Q7 - Find the departments where most of the young employees are well paid may be expressed in SQLf as [7]: SELECT #dep FROM Employees GROUP BY #dep HAVING most (age = ‘young’) ARE (sal = ‘well paid’)
(74)
Recently, the authors of SQLf are working on the interpretation of SQL’s aggregate functions such as MAX, AVG etc. for the case of fuzzy relations. For example, it is not clear what should be the answer to the query: “Find maximum salary of young employees”. For a discussion of this topic see, e.g., [15],[10],[11] P6. Kacprzyk and Zadrozny [21] started with the syntax of SQL language as it is implemented in the Microsoft Access® DBMS. The authors proposed to include in the language linguistic (fuzzy) terms (predicates) in the spirit of the approaches discussed in Section G1. More specifically, they proposed to take into account the following types of linguistic terms: • fuzzy values (e.g. YOUNG) • fuzzy comparators (e.g. MUCH GREATER THAN) • fuzzy quantifiers (e.g. MOST) The matching degree of relevant rows is calculated according to the previously discussed semantics of fuzzy conditions and linguistically quantified propositions. Kacprzyk and Zadrozny [21] have implemented this extension to SQL as an addin, called FQUERY for Access, in the Microsoft Access package, thus extending the native Access’s querying interface with a capability of manipulating linguistic terms. In FQUERY for Access, the user composes a query using a Query By Example type user interface provided by the host environment, i.e., Microsoft Access. It is executed more or less as a regular Access’s query while FQUERY is responsible for the calculation of matching degrees of the rows, interpreting linguistic terms in an appropriate way. The resulting rows of the answer relation are ordered nonincreasingly with respect to the matching degree. FQUERY for Access was one of the first implementations demonstrating the usefulness of fuzzy querying features for a crisp database. Besides the syntax and semantics of the extended SQL, the authors proposed also a scheme for the elicitation and manipulation of linguistic terms to be used in queries. The problem has been solved in accordance with the relational data model paradigm. Linguistic terms are maintained by FQUERY in a
28
Antonio Rosado et al.
dictionary, “de facto” as another system table storing metadata in regular relational database management systems. The concept of FQUERY for Access has been later developed in two directions. In [55] and [23] the very same concept has been applied in the environment of the Internet WWW service. Another interesting line of development [24], [25] boils down to the addition of some data mining capabilities to the existing fuzzy querying interface. Such a combined interface partially exploits the same modules and data structures and seems to be a promising direction for the development of advanced data analysis tools. G4. PRUF and flexible querying P7. Takahashi [45],[46] proposed a flexible query language (FQL) that is an extension to the domain relational calculus (DRC) Thus, in fact he proposed to use fuzzy logic language instead of classical logic language to express conditions that requested data should meet. The author follows to some extent the idea of Zadeh’s PRUF (see Section 3.6.). However, it seems that the grammar proposed by the author is unnecessarily complicated. As in case of the crisp DRC, the result of a query, expressed as in (7), is a set of rows satisfying a formula. Note that these rows may come directly from the relations defined in a database or can be “constructed” from existing relations (e.g., as in algebraic join operation). There are two problems with the Takahashi’s language. First, he does not discusses any extension of the concept of safe formula [47] for his language so that some queries (formulas) may produce an infinite number of rows as an answer. Second, Takahashi refers to Zadeh’s PRUF that is not fully sound in the context of a querying language. Basically, PRUF provides semantics for a subset of natural language propositions. This semantics is based on possibility theory and thus is more appropriate for the representation of imprecise facts in the database rather than for the interpretation of meaning of a query. Anyway, the calculations overlap to some extent and those proposed by Takahashi for matching degree assessment are still valid. An experimental application for querying a business database using the PRUF translation rules (section 3) that showed the flexibility of the PRUF rules was developed by [40].
5 Flexible query languages (FQLs) for fuzzy relational databases Several fuzzy querying models for modeling incomplete information in fuzzy relational databases have been proposed in the literature (see for instance [3]). Usually, each model requires a specialised querying formalism. In this section we propose a taxonomy for the main proposals found in the literature related to fuzzy
Flexible Query Languages for Relational Databases: an Overview
29
relational databases. The objective of proposing this taxonomy is to offer a structured view about the topic. We hope to shed some light on main differences and similarities between the particular approaches. Further, we hope the taxonomy can offer some guidance and clarification about the most relevant proposals in the area. 5.1 Taxonomy of FQLs for fuzzy databases In Fig. 5 we propose a classification for different FQLs proposed in the literature for fuzzy relational databases.
Taxonomy G1. Possibilistic model G2. Similarity-based model G3. Hybrid models
Proposals of FQLs for fuzzy databases P1. Prade and Testemale [34],[35] P2. Baldwin, Coyne and Martin [1] P3. Bosc and Pivert [8] P4. Buckles and Petry [12] P5. Buckles, Petry and Sachar [13] P6. Shenoi, Melton and Fan [42],[43] P7. Medina, Pons and Vila [32] P8. Galindo, Medina and Aranda [18] P9. Galindo, Medina, Pons and Cubero [20]
Fig. 5. Taxonomy for Flexible Query Languages (FQLs) for fuzzy databases The proposed taxonomy includes three main groups. Group 1 (G1) is devoted to proposals related to possibilistic fuzzy databases. Group 2 (G2) includes proposals relevant for similarity- based models. Group 3 (G3) presents proposals for hybrid models, i.e. combined possibilistic and similarity-based models. The literature on fuzzy databases is much richer and includes among others: [48], [49], [56]. We selected some representative proposals as listed in Fig. 5 and now we will examine them in a more detailed way. G1 Possibilistic model P1. Prade and Testemale [34] generalize the concept of a relational database in such a way that value dij of the j-th attribute in tuple ti may be given as a possibility distribution. This makes it possible to store incomplete or imprecise information. The idea may be best illustrated with an example [34]. Let us consider the PERSON relation (Fig. 6) storing information about students where M1
30
Antonio Rosado et al.
corresponds to the grade in mathematics during the first quarter and NAME and AGE represent name and age of a student. All numerical and non-numerical values in the columns AGE and M1 may be easily represented using appropriate possibility distributions. For instance, the value used for Jill’s AGE attribute means nothing more than that our knowledge about her age is drawn from the proposition: “Jill is young”. This proposition induces the possibility distribution on the domain of the AGE attribute. Because we do not know what is Jill’s exact age we can only assign a possibility degree to all potential numbers representing her age. This distinction between a fuzzy set and an induced possibility distribution is important for approaches dealing with relational databases.
Person:
Name Tom David Bob Jane Jill Joe Jack
Age Young 20 22 about_21 Young about_23 [22, 25]
M1 15 Rather_bad bad_to_very_bad Rather_good around_10 [14, 16] Unknown
Fig. 6. Example from Prade and Testemale [34] Prade and Testemale [34] proposed to adapt the classical relational algebra to the case of the possibilistic database. Thus, all standard operations of selection, Cartesian product, join (see [4] for some problems concerning the possibilistic join), projection, union and intersection are extended. In order to illustrate the algebra, we discuss the selection operation and give a relevant example of this type of query. The selection, σP(R), of a relation R upon the condition P may refer to two types of atomic conditions for Pi: 1. Aj θ a, where Aj is the name of an attribute, θ is a comparison operator (fuzzy or not) and a is a constant (fuzzy or not); 2. Aj θ Ak, where Ak is also an attribute name. More complex conditions can be built from the above atomic conditions and the logical connectives of negation, disjunction and conjunction. The matching degree of an atomic condition and a tuple ti is computed as a pair: possibility and necessity measure (with respect to the possibility distributions dij and dik) of relevant sets. In case (a) it is the set, crisp or fuzzy, of the elements belonging to the domain of Aj and being in relation θ (crisp or fuzzy) with the constant a. In the second case (b) it is the subset of the Cartesian product of
Flexible Query Languages for Relational Databases: an Overview
31
domains of Aj and Ak containing only the pairs of elements being in relation θ. In this case a joint possibility distribution over the Cartesian product of the domains of Aj and Ak is used. Formally, the matching degree for case (a) is computed as follows. Let us denote by F a set (in general fuzzy) whose possibility and necessity measures have to be computed. Its membership function for the elements of the domain of attribute Aj is:
µ F (d ) = sup min( µθ (d , d ' ) , µ a (d ' )) d ∈ Dom( A j ) d '∈D
(75)
Now, the possibility and necessity measures of set F with respect to the possibility distribution π d being the value of dij are computed as: ij
Π dij ( F ) = sup min(π dij (d ), µ F (d ))
(76)
Ν dij ( F ) = inf max( 1 - π dij (d ), µ F (d ))
(77)
d ∈D
d∈D
In case (b) set F comprises the pairs of elements (d, d’), d∈ Dom(Aj), d’∈ Dom(Ak) such that d θ d’ is satisfied. Thus, its membership function is identical to that of θ:
µ F (d , d ′) = µθ (d , d ′)
(78)
Now we compute the possibility and necessity measures with respect to a joint possibility distribution π ( dij ,dik ) :
π ( dij ,dik ) (d , d ' ) = min (π dij (d ) , π dik (d ' ))
∀(d , d ' ) ∈ D × D
(79)
Then, the possibility and necessity measures are computed as previously:
Π ( dij ,dik ) ( F ) = sup min(π (dij ,dik )(d , d ' ), µ F (d , d ' ))
(80)
Ν ( dij ,dik ) ( F ) = inf max( 1 - π ( dij ,dik ) (d , d ' ), µ F (d , d ' ))
(81)
d∈D
d∈D
P2. Baldwin and his collaborators [1] implemented a system for querying a fuzzy relational database that uses semantic unification and the evidential logic rule. The value of an attribute in the database may be either a crisp value or a possibility
32
Antonio Rosado et al.
distribution of values. The queries are composed of one or more conditions (corresponding to attributes of the database), with an importance for each condition and they are represented by a filtering function (similar to the notion of quantifier) and a threshold. The specific feature of their work [1] is the process of semantic unification used for matching the fuzzy values of the conditions with the possibility distributions of different attributes of a tuple in a database. That process is based on the mass assignment theory developed by Baldwin [2] which gives for the matching of two fuzzy sets, an interval [n, p], where n (necessary) is the certain degree of matching and p (possibility) is the maximum possible degree of matching. Next, it is described how that interval is computed. The mass assignments theory [2] provides a bridge between the two forms of uncertainty: probability and fuzziness. A fuzzy set induces a family of probability distributions that can be represented by a function called a mass assignment. The interpretation of this translation (fuzzy set to a mass assignment) may be briefly explained as follows [38]. X A mass assignment is a function m: 2 → [0,1], such that: m(Ai) ≥ 0
∑ m( Ai ) = 1
(82)
i
where Ai are subsets of a set X = {x1, x2, …,xn}, such that Ai = {x1, …, xi}. Hence, A1 ⊂ A2 … ⊂ An. Note that these expressions mean that m represents a family of probability distributions. A fuzzy set is converted to a mass assignment in the following way. The normalized fuzzy set A = x1/µ(x1) + x2/µ(x2) + … + xn/µ(xn), where µ(x1) = 1 and µ(x1) ≥ µ(x2) ≥ … ≥ µ(xn), induces a possibility distribution π such that πx(xi) = µA(xi) (see Eq. (36)). Then, the mass assignment m associated with πx is defined over the subsets Ai as: m(Ai) = mi
(83)
where mi = πx(xi) - πx(xi+1), πx(x1) = 1 and πx(xn+1) = 0. Semantic unification computes the matching of two fuzzy sets calculating the mass assignment m(A|A’), which represents a conditional probability distribution, over the truth set {t, f, u}, of A given A’. The resulting support pair (Sn, Sp) is the sum of the truth- values (t), in the case of Sn, and the sum of the truth-values t with the uncertain values u, for the case of Sp. For example [1], the support pair for matching the two fuzzy sets: cheap = 10/1 + 20/0.5 + 25/0.25 + 30/0.01 average = 20/0.01 + 25/0.5 + 30/1
(84)
Flexible Query Languages for Relational Databases: an Overview
33
is computed calculating the following matrix of mass assignments m(cheap | average): {Mi}: mi {10}: 0.5 {10, 20}: 0.25 {10, 20, 25}: 0.24 {10,20,25,30}: 0.01
{30}: 0.5 f f f t: 0.005
{30, 25}: 0.49 f f u: 0.1176 t: 0.0049
{30, 25, 20}: 0.01 F u: 0.0025 u: 0.0024 t: 0.0001
Fig. 7. Example from Baldwin et al. [1] to which corresponds the following support pair: (Sn,Sp)=([0.005+0.0049+0.0001],[0.0025+0.1176+0.0024+Sn])=(0.01,0.1325) Afterwards, they combine different matching degrees of conditions with their importances using a process called an evidential support logic rule. This rule uses a function called an S function that acts as an aggregation operator OR or AND or even something between these two functions (similar to quantifiers). Formally, the support pair (Sn, Sp) for a tuple is computed in the following way:
(Sn , S p ) = S ∑ w jα j , S ∑ w j β j n
j =1
n
j =1
(85)
where (αj βj) are the supports for the n conditions, wj are their importances and n
∑ w j =1. j =1
Next, we present an example of using semantic unification on querying a fuzzy relational database. The following relation is a simplification of that presented in [1] with some attributes removed. Common_name Pine_marten polecat ferret mink
Upper_fur ({brown:1,black:0.4}(very_dark)) ({brown:0.7,black:0.6}(very_dark)) ({brown:0.5,black:0.7}(dark))
Body_length (average pine_marten) (average polecat) (average polecat) ({brown:1,black:0.7,chocolate:0.8}(very_ (average mink) dark))
Fig. 8. Example from Baldwin et al. [1] The attribute common_name is crisp whereas the attributes upper_fur and body_length are fuzzy. The attribute upper_fur is a compound fuzzy attribute, which means it is composed of two possibility distributions: the first is discrete
34
Antonio Rosado et al.
(e.g., {brown: 1, black: 0.4}) and the second is continuous (e.g., very_dark). Body_length is defined using continuous possibility distributions such as average pine_marten, which specifies that the length is average in the context of pine_marten mammals. The answer for the query [1]: Selection criteria. Threshold = 0. Body_length: (average polecat), Importance: high Upper_fur: ({brown: 1, black: 0.7}(very_dark)), Importance: low is: mink has support (0.40175 1) ferret has support (0.3 0.728) polecat has support (0.426 0.88) pine_marten has support (0.116 0.953846)
The above answer shows that there are two mammals competing for the best solution: the mink and the polecat; the mink has a higher possible support and the polecat has a higher necessary support. Note that the threshold supplied by the user is a necessary threshold, which means that all mammals with a necessary support greater or equal to zero appear in the answer. P3. Bosc and Pivert [9] propose a new type of query for possibilistic databases that do not rely on the fuzzy pattern matching (see P1). Instead, the queries of this new type refer to the representation of the data (possibility distributions). In this case, an answer is either a crisp or fuzzy set depending if the queries are crisp or fuzzy. Moreover, these new queries improve the expression power of the associated FQL: besides the atomic conditions of types Aj θ a and Aj θ Ak, FQLs are enriched with new conditions that we will describe next. First, let us consider such new queries referring to only one possibility distribution. In order to express their conditions Bosc and Pivert [9] define the following three functions: Poss(A,{d1, …,dn}) = min(πA(d1), …, (πA(dn))
(86)
Card_cut(A,λ) = |{d ∈ D: πA(d) ≥ λ}|
(87)
Card_supp(A) = |{d ∈ D: πA(d) > 0}|
(88)
where A is an attribute in the possibilistic database; d1, …,dn are values in the domain, D, of attribute A; πA is a possibility distribution representing the value of A, and λ is a number from the interval [0,1]. Function Poss(A,{d1,…,dn}) supplies
Flexible Query Languages for Relational Databases: an Overview
35
the truth degree of the statement “all the values d1, …, dn are possible for A”. Functions Card_cut(A, λ) and Card_supp(A) supply the number of values that are possible for A with possibility degrees above or equal to, respectively, λ and 0. Therefore, we can easily express queries of the following type [9]: “find the houses for which the price value $100.000 is considered more possible than the value $80.000” or “find the houses for which $100.000 is the only price value which is completely possible”, as Poss(PRICE,{100.000}) ≥ Poss (PRICE, {80.000}) and Poss(PRICE,{100.000}) =1 and Card_cut(PRICE,1) = 1, respectively. These conditions are Boolean and, hence, the respective answers are crisp relations. However, these conditions can be fuzzified and then the respective answers are fuzzy relations (this corresponds to the case of fuzzy queries against crisp data, which was detailed in Section 4.). In order to perform a syntactical comparison between two possibility distributions various comparison techniques may be employed, cf. [36]. Bosc and Pivert [9] use an extended resemblance relation defined on the interval [0,1], called a fuzzy equality measure, which is defined as follows:
µ EQ (π , π ') = minψ (π (u ), π ' (u ))
(89)
u ∈D
where π, π’ are two possibility distributions to be compared and ψ is a resemblance relation, i.e., a reflexive and symmetric relation defined on [0,1]. They also propose an extended formula based both on the resemblance relation over D, denoted RES, and the resemblance relation over [0,1] (a proximity measure, pr,), defined as:
µ s (π ,π ') (u ) =
sup
v∈sup p (π ')
(
)
min µ RES (u , v ), µ pr (π (u ), π ' (v ))
(90)
The equation given above measures the degree to which the possibility distribution π can be replaced by the possibility distribution π’ with respect to an element u belonging to the support of π. Such a replacement is acceptable (the computed degree is high) if there exists v belonging to the support of π’ such that u and v are similar (in the sense of RES) and the values π(u) and π’(v) are similar (in the sense of pr). Then, the degree to which we can replace a possibility distribution π with a possibility distribution π’ with respect to the whole support of π is given by the following equation [9]:
µ repl (π ,π ') =
inf
u ∈sup (π )
(
)
max 1 − π (u ), µ S (π ,π ') (u )
(91)
36
Antonio Rosado et al.
where the resulting degree is the weighted combination of the degrees µs(π,π’)(u) for all the elements u in the support of π and the values π(u) are weights (see Eq. (67)). The query conditions involving two representations can be expressed as: REP(A1) ≈ REP(A2)
(92)
where REP(A1) and REP(A2) are the “shapes” of the possibility distributions to be compared. For example [9], the condition REP(AGE) ≈ REP(middle_aged) will evaluate the extent to which a value of attribute AGE (possibility distribution) is syntactically similar to the possibility distribution induced by the fuzzy set corresponding to the middle_aged concept. Notice that in the case of fuzzy pattern matching a similar query may be used. However, it would produce a possibility/necessity measures of the event that the value of the AGE attribute belongs to the fuzzy set middle_aged provided that all we know about the age is a possibility distribution. G2 Similarity-based model P4. Buckles and Petry [12], [13], [33] introduced a similarity based model for a fuzzy database in which: 1. Each domain Dj is equipped with a similarity relation Sj (see Fig. 9b)), which extends the identity relation used in the crisp relational model. It means that two values of an attribute match not only when they are identical but also if they are similar enough. Similarity relations support basic features of fuzzy querying: a query requesting a given attribute value will also be satisfied by other similar attribute values. 2. The value of tuple i for attribute j, dij, may be any valid (that is, verifying the semantics of the relation) subset of its domain Dj except for the null set. That is:
d ij ∈ 2
Dj
d ij ≠ ∅ . This definition helps represent uncertainty within the
tuples as well as it is the consequence of the similarity relations introduced. Fig. 9c) illustrates a relation in this model; the domain of each attribute and the corresponding similarity relations are shown in Fig. 9a) and Fig. 9b), respectively. The authors also proposed an extension to the relational algebra. The idea of this extension is illustrated on the example of the selection operation (see Fig. 9d)). The syntax of the selection operation [13] is slightly richer in comparison with the one introduced in Section 2 (note that here we use our notation, not the original one):
σP(R) with
(93)
Flexible Query Languages for Relational Databases: an Overview
37
A= {a, b, c, d, e} M= {α, β, χ, δ} a) Domain Sets a b c d e
a 1 0.6 0.3 0.6 0.5
b 0.6 1 0.3 0.7 0.5
c 0.3 0.3 1 0.3 0.3
d 0.6 0.7 0.3 1 0.5
e 0.5 0.5 0.3 0.5 1
α β χ δ
α 1 0.7 0.4 0.5
β 0.7 1 0.4 0.5
χ 0.4 0.4 1 0.4
δ 0.5 0.5 0.4 1
b) Similarity Relations: S1 (left), S2 (right) R2: A a c, e a a
M
R2’: A M a α, δ
α, δ β β χ
c) Relations (one in this example)
a β a χ
R: A M a α, β, δ a χ
d) R ←(σA=a(R2) with α(M) = 0.5)
Fig. 9. Example from [12]: components of a fuzzy relational database (a-c); a fuzzy relational algebra operation: selection (d). where R and P denote the relation and a Boolean expression as previously, while the level condition specifies a similarity threshold, a number belonging to [0,1], for the domain Dj of an attribute Aj appearing in P. This operation selects the tuples from R that satisfy condition P but if a subcondition of P is of the form Aj=a, where Aj is an attribute and a is constant, then it is interpreted as “a is similar to an element of the value of Aj at least to the degree expressed with the level condition” (remember that the values of attributes are, in general, sets). See Fig. 9d) where R2’ is the intermediate result for the query obtained from the tuples satisfying condition A = a, where A is an attribute name and a belongs to A’s domain. Then, the result R is obtained by removing redundant tuples that are identified using the following definition. Two tuples ti and tk are redundant if [12]: I(dij ∪ dkj) ≥ α(Dj),
j = 1,2,…,m
(94)
where dij is the value of tuple i for attribute j; Dj is the domain of attribute j; α (Dj) is the similarity threshold and I is a similarity index defined as:
38
Antonio Rosado et al.
I ( H ) = min S j ( x,y ) x,y∈H
, H ⊆ Dj
(95)
where Sj(⋅,⋅) is a similarity relation associated with the domain Dj . In summary, two tuples are redundant if the values of all corresponding attributes are similar. Two values, dij and dkj of an attribute Aj are similar if the minimum similarity degree between a pair of elements in dij ∪ dkj, I(dij ∪ dkj), is greater than a pre-specified one for this attribute level, α(Dj). All queries should pre-specify these levels for all attributes involved (by default it is assumed to be 1.0). In the example of Fig. 2b), the tuples (a,{α, δ}) and (a,β) in relation R2’ are redundant because a is obviously similar to a (the similarity relation is reflexive) and β is similar to both α and δ, that is: min {S2(α,β), S2(δ,β)} ≥ α(M) = 0.5
(96)
Hence, the tuples (a,{α,δ}) and (a,β) are merged, producing the tuple (a,{α,β,δ}) in the resulting relation R. P5. Buckles et al [13] adapted DRC (Section 2) to the similarity based model. Syntactically, this adaptation manifests itself with the addition of the with clause following each formula. This clause is to be interpreted in a way analogous to the case of their extended relational algebra. Thus, starting with the standard DRC query equivalent formula {( X 1 , … , X n ) Ψ ( X 1 , … , X n )} , Buckles and Petry’s fuzzy domain calculus comprises the following atomic formulae [13]: R(X1,X2, …,Xn) with < domain level conditions >
(97)
X1 θ X2 with
(98)
where R is a database relation, Xi is a constant or a domain variable and the domain level conditions are level conditions for all attributes in R; θ is a comparison operator and the operator level condition is a level condition applying to θ (when omitted its value is 1 by default). In case of Eq. (97) the variables X1, X2, …, Xn are instantiated from the values of a tuple ti = (di1,…,din) producing the following vector of matching degrees [13]: <S(di1,X1), S(di2, X2), …, S(din, Xn)>
(99)
that is, the matching degree, γ(R(.),ti), of R(X1,X2, …, Xn) against tuple ti. The values S(dij,Xj) are defined as follows (provided all variables Xi are instantiated from the same tuple - otherwise S(dij,Xj) = 0 ∀j):
Flexible Query Languages for Relational Databases: an Overview
min S j (u,v) , u∈d ∧v = X j S (d ij ,X j ) = ij min S (u,v) , u∈dij ∧v∈x j j
if X j is a constant
39
(100)
if x j is an instantiation for X j
Tuple ti satisfies Eq. (97) above if for each j, S(dij,Xj) is greater or equal to the similarity threshold corresponding to the attribute Aj. In case of the Eq. (98) the matching degree, γ(X1 θ X2,ti), of formula X1 θ X2 against tuple ti is the minimum value θ(x1,x2) over the pairs (x1,x2) where x1 ∈ di1, x2 ∈ di2 and dij correspond to a constant or an instantiation of the variable. Tuple ti satisfies Eq. (98) above if this matching degree is greater or equal to the threshold specified by the operator level condition. Further, a formula in Buckles and Petry’ fuzzy domain calculus is defined in a similar way to the definition of a domain calculus formula (Section 2). Specifically, it is one of the following expressions [13]: 1. An atomic formula 2. ψ1 ∧ ψ2 with < level condition >, ψ1 ∨ ψ2 with < level condition >, ¬ ψ with < level condition > 3. ∃X(A) ψ with < level condition >, ∀X(A) ψ with < level condition > 4. (ψ), [ψ] where ψ, ψ1, ψ2 are formulas, X is a domain variable associated to attribute A, and level condition applies to any free or bound variable in the formulas ψi. The authors introduce also the concept of the safe formula in a way analogous to the crisp case. Then, they prove that the expressive power of their DRC is at least the same as the previously discussed fuzzy relational algebra (see Proposal P4). P6. Shenoi and Melton [42] extended the fuzzy relational database model of Buckles and Petry (see Proposals P4 and P5) by relaxing the transitivity property required for the similarity relation. They replaced the similarity relation with a proximity relation and showed that general properties of the model are preserved. This provides the user with the freedom to define the closeness among the different elements of the domain. For example [42], if one is close to two with a degree of 0.8 and two is close to three also with a degree of 0.8, then a degree of 0.6 between one and three would not be allowed if we used a similarity relation: the max-min transitivity would impose a value equal or higher to 0.8 which would contradict common sense. Further, Shenoi et al [43] proposed a more general model, an equivalence classes model. The model is based on the following two assumptions [43]: 1. the existence of partitions at the desired level of precision for each non-empty subset of a domain
40
Antonio Rosado et al.
2. the total ordering of partitions related to the same domain but for different precision levels is provided by the concepts of a finer and coarser partition. Assumption 1 is motivated by the postulate that a partitioning of a domain should be defined separately for each subset of elements currently appearing in a database. The authors call such a subset a temporal domain (it is also called an active domain). Assumption 2 expresses an intuitive requirement that elements indistinguishable (i.e., falling into the same equivalence class) at a given level of precision should be still indistinguishable at any lower level of precision. Thus, any two partitions of the same temporal domain taken at different precision levels should be in a coarser/finer relation. This is an abstract model [43] because it does not specify how to partition the domain: the users have a freedom to choose how to define the equivalence classes (information chunks). For instance, Buckles and Petry [12] used similarity relations on scalar domains while Shenoi and Melton [42] used proximity relations on scalar domains. In fact, Shenoi et al. [43] show that any partitions satisfying assumptions 1 and 2 guarantee that the fundamental properties of the relational model are preserved. They also proposed a fuzzy relational algebra for this model. For example [43], Fig. 10 shows a fuzzy relation of candidates to a political election storing the name, age and political view of the candidates and the associated algebra. The scheme of a fuzzy relation is defined by adding the set of precision levels αR used for each attribute:
αR = (αName, αAge, αView)
(101)
In Fig. 10(a-b), the partition of temporal domains into equivalence classes of identical elements results in partitions whose equivalence classes are sets with only one element, corresponding to the special case of the relational model. In fact, since each component of a tuple is a subset of an equivalence class in the partition of its respective attribute, each attribute value in the relation candidates must have only one element. For example, for attribute Name we can have the following partition: PName : {{Cook},{Dean},{Hall},{Kane},{Luce}, {Mann}, {Page}, {Rudd}}
(102)
Flexible Query Languages for Relational Databases: an Overview Name {Cook} {Dean} {Hall} {Kane} {Luce} {Mann} {Page} {Rudd}
41
Age {39} {68} {42} {54} {73} {50} {65} {58}
View {Arch_Conservative} {Ultra_Liberal} Name Age View {Conservative} {Cook} {39} {Arch_Conservative} {Moderate} {Liberal} b) questionable_candidates (α = (αName, αAge, αView)). {Moderate} {Liberal} Name Age View {Conservative} {Dean} {68} {Ultra_Liberal} a) candidates (α = (αName, αAge, αView)). {Kane} {54} {Moderate} {Luce} {73} {Liberal} {Mann} {50} {Moderate} Name Age View {Page} {65} {Liberal} {Cook, Hall} {39, 42} {Arch_Conservative, Conservative} {Rudd} {58} {Conservative} {Dean, Luce, Page} {65, 68, 73} {Ultra_Liberal, Liberal} {Kane, Mann} {50, 54} {Moderate} {Rudd} {58} {Conservative} d) revised_candidates (αR = (αName, αAge, αView)) ← candidates – questionable_candidates c) abstract_candidates (α’ = (α’Name, α’Age, α’View)).
Fig. 10. Example from Shenoi et al. [43]: algebra for the equivalence classes’ model Partitions PName, PAge, PView have implicit precisions, respectively, αName, αAge, αView (see relation candidates in Fig. 10a)). On the other hand, the following partitions: P’Name : {{Cook, Dean, Hall, Kane, Luce, Mann, Page, Rudd}}
(103)
P’Age : {{39,42}, {50,54,58}, {65,68,73}}
(104)
P’View : {{Ultra_Liberal, Liberal},{Moderate},{Conservative, Arch_Conservative}}
(105)
create equivalence classes of more or less identical elements. We can see that the precision of P’Name, α’Name, makes all elements of the temporal domain DName to be indistinguishable whereas the precision of partitions P’Age and P’View, α’Age and α’View, split temporal domains DAge and DView into three equivalent classes. It is obvious that α’ ≤ α means that the corresponding partition P’ is coarser than the corresponding partition P. Then, if we decrease the precisions α (identical) for attributes of relation candidates (see Fig. 10a)) to the new precisions α’ (more or less identical), redundant tuples will appear which must be merged. Fuzzy relation abstract_candidates (see Fig. 10c)) shows the result of merging the tuples in
42
Antonio Rosado et al.
relation candidates considering the new precisions α’Name, α’Age, α’View. We can see, for example, that candidates Kane and Mann are equivalent and hence merged because their names, ages and political views are considered equivalent (see partitions P’Name, P’Age and P’View) but Rudd is not equivalent to Kane and Mann because Rudd’s political view (conservative) is not equivalent to Kane and Mann’ political view (moderate). Note that moderate and conservative do not belong to the same equivalence class in partition P’View. Therefore, Rudd is not redundant and therefore is not merged with any other candidate in candidates’ relation. Finally, the difference operation between the fuzzy relations candidates (see Fig. 10a)) and questionable_candidates (see Fig. 10b)), using precision levels α’Name, α’Age, α’View, is shown in Fig. 10d). We can see that the two tuples, corresponding to candidates Cook and Hall, from fuzzy relation candidates are missing in the result, i.e. fuzzy relation revised_candidates. Besides the tuple appearing in relation questionable_candidates (Cook’s tuple), an extra tuple corresponding to candidate Hall is also removed from candidates because candidate Hall is equivalent to candidate Cook under the more or less identical precision (see P’Name, P’Age, P’View). G3 Hybrid models P7. Medina et al [32] proposed a fuzzy database model, GEFRED (generalized fuzzy relational database) that tries to integrate features of both the possibilistic and similarity based models. The data is represented with generalized fuzzy relations that take into account imprecision as well as uncertainty of information. The latter is dealt with via a compatibility degree associated to each attribute value. More precisely, a generalized fuzzy relation R is composed of two sets [32]: R = (H,B), where H (Head) is the set: H = {(A1 : DG1 [,C1]), (A2 : DG2 [,C2]), ..., (An : DGn [,Cn]) }
(106)
and B (Body) is the set: B = {(A1 : di1 [,ci1]), (A2 : di2 [,ci2]), ..., (An : din [,cin]) }
(107)
where Aj (j = 1…n) is the j-th attribute, DGj (j = 1…n) is the family of all possibility distributions defined over the domain of attribute Aj, which is called a generalized fuzzy domain; Cj is the compatibility attribute of attribute Aj, which may be optional; and dij is the value of attribute Aj in tuple i and cij (j = 1…n) is the compatibility degree of value dij, which is a value in the interval [0,1]. For example, Fig. 11b) (adopted from [32]) shows a generalized fuzzy relation with attributes NAME, ADDRESS, AGE, PRODUCTIVITY, SALARY and compatibility
Flexible Query Languages for Relational Databases: an Overview
43
attribute CAGE. The compatibility attributes associated with other attributes are equal to one and, therefore, they are not shown. Medina et al [32] defined an algebra, a generalized fuzzy relational algebra, to manipulate information stored in the fuzzy database. Next, we will illustrate the selection operation, which is called a generalized fuzzy selection. It is based on a simple condition θG(Aj,a)≥α, where θG is a comparison operator (generalized fuzzy comparator); α is a compatibility threshold (α ∈ [0,1]), and a is a constant. Comparison operator θG is defined as:
θG : DGj × DGj → [0,1]
θ G (π , π ' ) =
sup
(d1 ,d 2 )∈Dk ×Dk
min(θ (d1 , d 2 ), π (d1 ), π ' (d 2 ))
(108) (109)
where π and π’ are possibility distributions and θ can be: • A classical comparison operator such as =, ≠, >, ≥, <, ≤; • A fuzzy comparison operator such as approximately equal, much greater than, etc; • A similarity comparison operator which is defined using a similarity relation on scalar data. In fact, Eq. (109) is similar to Eq. (80) used while querying possibilistic databases. A new element is here an explicitly stated threshold value which resembles queries in the similarity based model. Compatibility degrees of all involved attributes require a special treatment in the matching degree calculation. In fact, these compatibility degrees correspond to partial matching degrees that are immediately aggregated to an overall matching degree in other models. For example, the selection σ=(AGE,old) ≥ 0.6(Emp) of relation Emp (see Fig. 11a)) using the condition =(AGE,old) ≥ 0.6 results in relation S (see Fig. 11b)). We can see from the definition of label OLD (see Fig. 11d)) that tuples corresponding to employees Luis (his age is 30 or 31), Javier (his age is between 30 and 35) do not belong to S because their compatibility degree CAGE is zero and hence the condition is not satisfied. The same is true for tuples corresponding to employees Juan Carlos and Julia, which are YOUNG (see Fig. 11d)). Finally, the compatibility degrees for tuples corresponding to employees Antonio and Francisco are computed from Eq. (109). P8. Galindo et al [18] extended the GEFRED model with a fuzzy domain relational calculus (FDRC) for querying fuzzy relational databases. The FDRC language is described next. Galindo et al.’s fuzzy domain calculus comprises the following atomic formulas [18]:
44
Antonio Rosado et al. H Name B Antonio Francisco Luis
Address Reyes Católicos P. A. Alarcón Recogidas
Age Middle Old {.8/30,1/ 31} Young Young [30,35]
Juan Carlos Camino Ronda Julia Puerta Real Javier Gran Vía a) The generalized fuzzy relation Emp
H Name Address B Antonio Reyes Católicos Francisco P. A. Alarcón
Productivity Fair Excellent Good
Salary 100000 150000 110000
Department Production Comercial Production
Bad Good Fair
90000 130000 105000
Production Comercial Human Resources
Age CAge Productivity Middle 0.75 Fair Old 1 Excellent
Salary 100000 150000
b) Relation S = σ=(AGE,Old) ≥ 0.6 (Emp).
H Department CDepartment B Production 1 S1 Human resources 0.5 S2
1
0
YOUNG MIDDLE
OLD
16 25 30 35 40 45 50 55
c) Relation R: departments with at least one d) Labels definitions for attribute Age. bad (≥ 0.5) employee.
65
80
AGE
Fig. 11. Example from [32]: algebra for the GEFRED model 1. R(X1, X2, ..., Xn) ≥ α, where R is predicate symbol corresponding to a generalized fuzzy relation with n attributes, and each Xi is a constant or a domain variable. This atom requires that the tuple (X1, X2, ..., Xn) belongs to a relation corresponding to R to a degree higher or equal α. For a given instantiation of variables Xi this degree of membership is computed as follows: 2.
R( X 1 , X 2 ,..., X n ) =
max min = (d rc , X c )
r =1,..., m c =1,..., n
(110)
where m is the number of tuples in relation corresponding to R, = is a generalized fuzzy comparator defined by Eq. (109). Thus, the membership degree of a given tuple (X1, X2, ..., Xn) – note that now Xi denotes a constant originally present in the atomic formula or a value substituting the domain
Flexible Query Languages for Relational Databases: an Overview
45
variable – is computed by comparing this tuple with all tuples of relation corresponding to the predicate symbol R. This comparison is done using a fuzzy comparator for all attributes separately; drc denotes a value of the c-th attribute in the r-th tuple. Then, the total result of comparison is taken as the minimum of these per-attribute comparisons. A tuple of the relation for which this maximum value is attained is referred to as the most similar tuple. The fulfillment of threshold α is a value in the interval [0,1] that is the minimum value admissible for R(X1, X2, ..., Xn) in order to make the atom true. 3. θG(X,Y) ≥ α, where θG is a generalized fuzzy comparator, and X and Y are constants or domain variables. This atom expresses that the value X is related to the value Y by the fuzzy comparator θG to a certain truth degree, which is greater or equal to α. Examples of fuzzy atoms are R(X1, Good, X3) and =(X1,Good) ≥ 0.9. In the first case, the threshold is omitted which means that its value is equal to one. Further, a formula in Galindo et al.’s fuzzy domain calculus is defined in a similar way to the definition of a domain calculus formula (Section 2). Specifically, it is either an atomic formula or one of the following expressions: ¬ψ1, ψ1 ∧ ψ2, ψ1 ∨ ψ2, ψ1 ⇒ ψ2,∃X ψ1(X),∀X ψ1(X), where ψ1 and ψ2 are fuzzy formulas and X is a domain variable. Afterwards, they demonstrate an expressive power of FDRC proving that any expression in the fuzzy relational algebra has an equivalent expression in FDRC. Observe that so far we are still operating within the framework of classical logic – due to the threshold value α in an atomic formulae, they are true or false. A possible partial matching is preserved using the concept of a compatibility degree. The result of a query is a set of tuples satisfying it – a new generalized relation. Each attribute value of this new relation may be associated with a compatibility degree expressing how well this specific value matches the query. This degree is computed by a matching function γ which takes three arguments: a formula (query) ψ, a tuple ti = (di1,di2,…,din), and X, a domain variable (attribute) for which a compatibility degree is to be computed. This may be formally described as:
γ(ψ, ti, X) ∈ [0,1] ∪ λ
(111)
Value λ is a value not belonging to [0,1] that asserts that degree γ is not applicable or meaningless. Function γ(ψ,ti,X) is defined depending on the structure of ψ. There are four cases: ψ is an atomic formula, a negation, a disjunction or a formula with an existential quantifier [18]. When ψ is an atomic formula of type R(X1, X2, ..., Xn,K) ≥ α we have: R(K),
γ ( R(X1 , X2 , ..., Xn , K) ≥ α, ti , X ) = min{cij , R(ti )}, λ,
if there are no variables in ψ if X = Aj otherwise
(112)
46
Antonio Rosado et al.
where K is the list of constants present in ψ, the values R(K) and R(ti) are computed using Eq. (110); X = Aj indicates that variable X is an attribute Aj in R; cij is the value of the compatibility attribute Cj for the tuple most similar to ti; if there is no cij associated with this most similar tuple, then cij is assumed to be equal 1.0. The atomic formula θG (Xj,Y) ≥ α is evaluated as:
( (
)
)
θG (d ij , Y ),
γ θG X j , Y ≥ α , ti , X = θG ( X j , Y ), λ ,
if X i = X if X j is a constant
(113)
otherwise
The evaluation of the negation and disjunction expressions is done using the complement and maximum operators, respectively (see (17)-(18)). On the other hand, for formula ψ expressed as ∃ Xn+1 (ψ1 (X1, X2, ..., Xn, Xn+1)): γ (ψ ( X1,..., X n ), ti , X ) =
max
d i (n +1)∈DOM (ψ )
γ (ψ 1 (X1,..., X n , di (n +1) ), ti , X )
(114)
where DOM(ψ) is the set of all symbols that appear in formula ψ or in a tuple of a relation appearing in ψ. The remaining expressions, that is, the conjunction, the implication, and the universal quantifier can be expressed in an equivalent form using the negation, the disjunction and the existential quantifier. Then, the result we obtain for a general query {X1, X2, …, Xn | ψ(X1, X2, ..., Xn)} is a generalized relation R (see Eqs. (106)-(107)) that is computed using the following two steps [18]: 1. Compute all the tuples (dr1,…,drn) that make true the formula ψ(dr1,…,drn); 2. The compatibility values crj (j=1,…,n) for each compatibility attribute Cj, corresponding to the r (r = 1, …, m) tuples of R computed in 1., are computed as: crj = γ(ψ(X1,…,Xn),tr,Xj)
(115)
where tr = (dr1,…,drn) is the r-th tuple of relation R. If crj = λ or crj = 1 for all r = 1, …, m, the attribute Cj is removed from R. For example, the query “show the departments with at least one bad employee (with a degree greater than or equal to 0.5) “ may be expressed in FDRC as:
Flexible Query Languages for Relational Databases: an Overview {d | ∃n, a, ag, p, s (Emp(n,a,ag,p,s,d) ∧ =(p,Bad) ≥ 0.5)}
47
(116)
Which, considering the fuzzy relation in Fig. 11a), produces the resulting fuzzy relation in Fig. 11c). Further, the value C1Department is computed in the following way: C1Department = γ (ψ,t1,d) = ∃n, a, ag, p, s (γ (Emp(n,a,ag,p,s,d),t1,d) ∧ ∧ γ (=(p,Bad) ≥ 0.5,t1,d)) = max {γ (Emp(Antonio, Reyes Católicos, Middle, fair, 100000, d), t1,d) ∧ ∧ γ(=(fair,Bad) ≥ 0.5,t1,d), γ (Emp(Luis, Recogidas, {.8/30, 1/31}, good, 110000, d),t1,d) ∧ ∧ γ(=(good, Bad) ≥ 0.5,t1,d), γ(Emp(Juan Carlos, Camino Ronda, Young, Bad, 90000, d),t1,d) ∧ ∧ γ(=(Bad, Bad) ≥ 0.5,t1,d)} = max {min(1,0.5), min(1,0), min(1,1)} = max {0.5, 0, 1} = 1 Note that the existential quantifier (see Eq. (114)) is replaced with the values of tuples such that department = production (t1). Then, there is computed the matching degree for those tuples (three) and finally the maximum of these degrees is considered. Galindo et al. [19] propose also the inclusion of fuzzy quantifiers in the FDRC language just described. P9. Galindo et al. [20] implemented the FRDB model, GEFRED, on the crisp DBMS Oracle and a FQL, fuzzy SQL (FSQL). They extended the SELECT command of SQL in order to allow for more flexible conditions by choosing between possibility and necessity within fuzzy comparators; retrieving the most (least) important tuples using fulfillment thresholds or allowing fuzzy constants in the right side of the condition. More precisely, the main functionalities added are [20]: 1. Linguistic labels. These labels can be defined for two types of attributes: attributes with an ordered domain or attributes with a scalar domain. In the first case, the labels are defined as trapezoidal possibility distributions and, in the second case, a similarity relation between the labels of the attribute is defined. 2. Fuzzy comparators. They extend the usual comparators =, >, ≥, <, ≤ providing comparators that have two forms, corresponding to the possibility and the necessity cases. For example, the operator FEQ evaluates the possibility of two attributes (or one attribute and a constant) being equal using Eq. (80) (or Eq. (76)). And the NFEQ operator evaluates the necessity of two attributes (or one attribute and a constant) being equal using Eq. (81) (or Eq. (77)). Note that FEQ and NFEQ are instances of the generalized fuzzy comparator θG. Additionally,
48
Antonio Rosado et al.
they provide the fuzzy comparators much greater than (MGT, NMGT) and much less than (MLT, NMLT). 3. Fulfillment thresholds (α). They are specified with the syntax: [THOLD] α
(117)
which is equivalent to having THOLD replaced by ≥, and where the reserved word THOLD may be substituted by a crisp comparator (=, <, …), modifying the meaning of the condition. 4. Function CDEG() shows a column with the compatibility degree for the argument attribute, corresponding to the compatibility degree of the conditions in which the attribute appears. Function CDEG(*) shows the compatibility degree for all the fuzzy attributes appearing in the condition. If we want to see the compatibility columns for all attributes and the compatibility degree of the whole tuples, we will use the % character in the SELECT clause (like SQL does for the * character). 5. Fuzzy constants. FSQL allows for the use of the following constants: UNKNOWN, UNDEFINED, NULL, $[a,b,c,d], $label, [n,m], #n. $[a,b,c,d] is a fuzzy trapezoid function with a ≤ b ≤ c ≤ d. $label is a linguistic label (as described above); [n,m] is an interval, for which a = b = n and c = d = m; and #n means approximately n, in which the fuzzy trapezoid is replaced by a triangle with b = c = n and n – a = d – n. 6. Condition with IS. A condition, IS [NOT] (UNKNOWN | UNDEFINED | NULL)
(118)
is true (without NOT) when the fuzzy attribute value is equal to the fuzzy constant on the right. For example, the query: Q8 - Find the Spanish cities with more than “around 500 thousands” inhabitants may be expressed in FSQL as [20]: SELECT city, CDEG(inhabitants) FROM Population WHERE country = ‘Spain’ AND inhabitants FGEQ $[200,350,650,800] .75 AND inhabitants IS NOT UNKNOWN
(119)
Flexible Query Languages for Relational Databases: an Overview
49
where FGEQ is the fuzzy comparator extending the crisp ≥ comparator; $[200,350,650,800] is a fuzzy constant, and .75 is a fulfillment threshold requiring that the condition on the number of inhabitants is satisfied at least to the degree 0.75. Note that two columns would be displayed: the name of a city and the degree to which its number of inhabitants is greater or equal to around 500 thousands. The cities with unknown number of inhabitants are excluded.
6 Conclusion In the last two decades, imprecision (fuzziness) and uncertainty has been studied in the context of relational DBMS, in particular in the area of querying and in the area of modeling and storing imprecise and uncertain data. The first area has led to the appearance of increasingly flexible query languages (FQLs) which provide more human consistent interfaces in comparison to the classical query languages, by using fuzzy sets theory. In the second area, fuzzy sets theory is used to extend the relational database model, leading to what are usually called fuzzy relational database models, and on the flexible querying of the resulting models. We introduced two taxonomies for FQLs within the context of relational database models to organize the field and to offer a structured view of the topic. One taxonomy organizes the research on FQLs for crisp relational databases and another organizes the research on fuzzy relational databases. Both taxonomies provide a structured view of the main research topics studied and highlight the main differences and similarities between approaches. We believe that our contribution in organizing the field of flexible query languages in relational databases can shed some light on the most relevant proposals in the area, as well as guide designers and interested users in understanding and selecting the best approaches to suit their aims.
References [1] Baldwin JF, Coyne MR, Martin TP (1993) Querying a database with fuzzy attribute values by iterative updating of the selection criteria. In: Proceedings of International Joint Conference on Artificial Intelligence (IJCAI'93) [2] Baldwin JF, Martin TP, Pilsworth BW (1995) FRIL - Fuzzy and evidential reasoning in artificial intelligence. John Wiley & Sons, Inc, New York [3] Bosc P (1999) Fuzzy Databases. In: Bezdek J, Dubois D, Prade H (eds) Fuzzy sets in approximate reasoning and information systems, The Handbooks of Fuzzy Sets Series. Kluwer Academic Publishers, pp 403-468
50
Antonio Rosado et al.
[4] Bosc P, Lietard L, Pivert O (2000) About ill-known data and equi-join operations. In: Larsen HL, Kacprzyk J, Zadrozny S, Andreasen T, Christiansen H (eds) Flexible query answering systems. Recent advances. Physica-Verlag, Heidelberg New York, pp 65-74 [5] Bosc P, Pivert O (1992) Fuzzy querying in conventional databases. In: Zadeh LA, Kacprzyk J (eds) Fuzzy logic for the management of uncertainty. John Wiley & Sons, pp 645-671 [6] Bosc P, Pivert O (1993) An approach for a hierarchical aggregation of fuzzy predicates. In: Proceedings of 2nd IEEE International Conference on Fuzzy Systems (FUZZ-IEEE´93), USA, pp 1231-1236 [7] Bosc P, Pivert O (1995) SQLf: A relational database language for fuzzy querying. IEEE Transactions on Fuzzy Systems 3: 1-17 [8] Bosc P, Pivert O (1997) Fuzzy queries against regular and fuzzy databases. In: Andreasen T, Christiansen H, Larsen HL (eds) Flexible query answering systems. Kluwer Academic Publishers, pp 187-208 [9] Bosc P, Pivert O (1997) On representation-based querying of databases containing ill-known values. In: Proceedings of International Symposium on Methodologies for Intelligent Systems (ISMIS' 97), pp 477-486 [10] Bosc P, Pivert O, Lietard L (2001) Aggregate operators in database flexible querying. In: Proceedings of 9th IEEE International Conference on Fuzzy Systems (FUZZ-IEEE'2001), Melbourne, Australia, pp 1231-1234 [11] Bosc P, Pivert O, Lietard L (2003) On the comparison of aggregates over fuzzy sets. In: Bouchon-Meunier B, Foulloy L, Yager RR (eds) Intelligent systems for information processing. From representation to applications. Elsevier, pp 141-152 [12] Buckles BP, Petry FE (1985) Query languages for fuzzy databases. In: Kacprzyk J, Yager RR (eds) Management decision support systems using fuzzy sets and possibility theory. Verlag, TUV Rheinland, pp 241-251 [13] Buckles BP, Petry FE, Sachar HS (1986) Design of similarity-based relational databases. In: Prade H, Negoita CV (eds) Fuzzy logic in knowledge engineering. Verlag TUV Rheinland, pp 3-7 [14] Codd EF (1970) A relational model of data for large shared data banks. Communications of the ACM 13(6): 377-387 [15] Dubois D, Prade H (1990) Measuring properties of fuzzy sets: A general technique and its use in fuzzy query evaluation. Fuzzy Sets and Systems 38(2): 137-152 [16] Dubois D, Prade H (1997) Using fuzzy sets in flexible querying: why and how?. In: Andreasen T, Christiansen H, Larsen HL (eds) Flexible query answering systems. Kluwer Academic Publishers, pp 45-60 [17] Fodor J, Yager RR (2000) Fuzzy set-theoretic operators and quantifiers. In: Dubois D, Prade H (eds) Fundamentals of fuzzy sets. Kluwer Academic Publishers, 125-193
Flexible Query Languages for Relational Databases: an Overview
51
[18] Galindo J; Medina JM; Aranda GMC (1999) Querying fuzzy relational databases through fuzzy domain calculus. International Journal of Intelligent Systems 14(4): 375-411 [19] Galindo J, Medina JM, Cubero JC, García MT (2000) Fuzzy quantifiers in fuzzy domain calculus. In: Proceedings of 8th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU´2000), Spain, pp 1697-1702 [20] Galindo J, Medina JM, Pons O, Cubero JC (1998) A server for fuzzy SQL queries. In: Andreasen T, Christiansen H, Larsen HL (eds) Flexible query answering systems. LNAI: 1495, Springer, pp 164-174 [21] Kacprzyk J, Zadrozny S (1995) FQUERY for Access: fuzzy querying for Windows-based DBMS. In: Bosc P, Kacprzyk J (eds) Fuzziness in database management systems. Physica-Verlag, Heidelberg, pp 415-433 [22] Kacprzyk J, Zadrozny S (1997) Implementation of OWA operators in fuzzy querying for Microsoft Access. In: Yager RR, Kacprzyk J (eds) The ordered weighted averaging operators: theory and applications. Kluwer, Boston, pp 293-306 [23] Kacprzyk J, Zadrozny S (1999) Fuzzy querying via WWW: implementational issues. In: Proceedings of 7th IEEE International Conference on Fuzzy Systems (FUZZ-IEEE'1999), Seoul, Korea, pp 603-608 [24] Kacprzyk J, Zadrozny S (2000) On a fuzzy querying and data mining interface. Kybernetika 36: 657-670 [25] Kacprzyk J, Zadrozny S (2000) On combining intelligent querying and data mining using fuzzy logic concepts. In: Bordogna G, Pasi G (eds) Recent research issues on the management of fuzziness in databases. Physica-Verlag, Heidelberg New York, pp 67-81 [26] Kacprzyk J, Zadrozny S, Ziolkowski A (1989) FQUERY III+: a 'humanconsistent` database querying system based on fuzzy logic with linguistic quantifiers. Information Systems 6: 443-453 [27] Kacprzyk J, Ziolkowski A (1986) Database queries with fuzzy linguistic quantifiers. IEEE Transactions on Systems, Man and Cybernetics SMC 16: 474-479 [28] Kerre EE, De Cock M (1999) Linguistic modifiers: an overview. In: Chen G, Ying M., Cai K-Y (eds) Fuzzy logic and soft computing. Kluwer Academic Publishers, pp 69-85 [29] Klir GJ, Folger TA (1988) Fuzzy sets, uncertainty and information, PrenticeHall. [30] Lacroix M, Lavency P (1987) Preferences: putting more knowledge into queries. In: Proceedings of 13rd International Conference on Very Large Databases (VLDB' 87), Brighton (GB), pp 217-225 [31] Liu Y, Kerre EE (1998) An overview of fuzzy quantifiers (I). Interpretations. Fuzzy Sets and Systems 95: 1-21
52
Antonio Rosado et al.
[32] Medina JM, Pons O, Vila MA (1994) GEFRED: a generalized model of fuzzy relational databases. Information Sciences 76(1-2): 87-109 [33] Petry FE (1996) Fuzzy databases: principles and applications. Kluwer Academic Publishers [34] Prade H, Testemale C (1984) Generalizing database relational algebra for the treatment of incomplete or uncertain information and vague queries. Information Sciences 34: 115-143 [35] Prade H, Testemale C (1987) Representation of soft constraints and fuzzy attribute values by means of possibility distributions in databases. In: Bezdek JC (ed) Analysis of fuzzy information, vol. II, CRC Press. pp 213-229 [36] Raju KVSVN, Majumdar AK (1988) Fuzzy functional dependencies and lossless join decomposition of fuzzy relational database systems. ACM Transactions on Database Systems 13: 129-166 [37] Ramakrishnan R, Gehrke J (2000) Database management systems, McGrawHill. [38] Ribeiro RA (1993) Application of support logic theory to fuzzy multiple attribute decision problems, University of Bristol, UK [39] Ribeiro RA, Moreira AM (1999) Intelligent query model for business characteristcs. In: Proceedings of IEEE/WSES/IMACS CSCC'99 Conference, Greece [40] Ribeiro RA, Moreira AM (2003) Fuzzy query interface for a business database. International Journal of Human Computer Studies 58(4): 363-391 [41] Schmucker KJ (1984) Fuzzy sets, natural language computations, and risk analysis, Computer Science Press [42] Shenoi S, Melton A (1989) Proximity relations in the fuzzy relational database model. Fuzzy Sets and Systems 31: 285-296 [43] Shenoi S, Melton A, Fan LT (1990) An equivalence classes model of fuzzy relational databases. Fuzzy Sets and Systems 38: 153-170 [44] Tahani V (1977) A conceptual framework for fuzzy query processing: a step toward very intelligent database systems. Information Processing and Management 13: 289-303 [45] Takahashi Y (1991) A fuzzy query language for relational databases. IEEE Transactions on Systems, Man and Cybernetics SMC 21: 1576-1579 [46] Takahashi Y (1995) A fuzzy query language for relational databases. In: Bosc P, Kacprzyk J (eds) Fuzziness in database management systems. PhysicaVerlag, Heidelberg, pp 365-384 [47] Ullman JD (1982) Principles of database systems, Computer Science Press. [48] Umano M (1982) FREEDOM-0: a fuzzy database system. In: Gupta M, Sanchez E (eds) Fuzzy information and decision processes. North-Holland, Amsterdam, pp 339-347
Flexible Query Languages for Relational Databases: an Overview
53
[49] Umano M, Fukami S (1994) Fuzzy relational algebra for possibilitydistribution-fuzzy relational model of fuzzy data. Journal of Intelligent Information Systems 3: 7-27 [50] Yager RR (1994) Interpreting linguistically quantified propositions. International Journal of Intelligent Systems 9: 541-569 [51] Zadeh LA (1965) Fuzzy sets. Information and Control 8: 338-353 [52] Zadeh LA (1975) The concept of a linguistic variable and its application to approximate reasoning - II. Information Sciences, 8: 219-269 [53] Zadeh LA (1978) PRUF -a meaning representation language for natural languages. International Journal of Man-Machine Studies 10: 395-460 [54] Zadeh LA (1983) A computational approach to fuzzy quantifiers in natural languages. Computational Mathematics Applications 9: 149-184 [55] Zadrozny S, Kacprzyk J (1998) Implementing fuzzy querying via the Internet/WWW: Java applets, ActiveX controls and cookies. In: Andreasen T, Christiansen H, Larsen HL (eds) Flexible query answering systems. LNAI: 1495, Springer, Berlin Heidelberg, pp 382-392 [56] Zemankova-Leech M, Kandel A (1984) Fuzzy relational databases - A key to expert systems. Koln, Germany, TUV Rheinland.
Vacuity-Oriented Generalized Yes/No Queries Addressed to Possibilistic Databases Patrick Bosc and Olivier Pivert IRISA-ENSSAT Technopole Anticipa BP 80518 22305 Lannion France bosc, [email protected]
1 Introduction In this chapter, extended relational databases are considered where some attribute values are imprecisely known. The need for imperfect data is more and more recognized and imprecise information can appear in diverse situations such as data warehouses, forecasts, incomplete archives, structured data extracted from texts, or systems where information issued from automated recognition procedures is stored. Different formalisms can be used to represent imprecise information (see [7] for instance), and the possibilistic setting is assumed in the rest of the chapter. A key question is to define a sound semantics for queries addressed to imprecise databases. Since imprecise data are represented as (possibly infinite) sets of acceptable candidates, an imprecise database can be seen as a set of regular databases, called worlds, associated with a choice for each attribute value. This approach provides a rational starting point for the definition of a query in the sense that its result is the set of the results obtained for each world (or interpretation). Unfortunately, such an approach is intractable, obviously in the case of an infinite number of worlds, but also due to the possibly huge number of worlds when it is finite. This observation leads to consider only specific queries which can be processed directly against the possibilistic database (the processing is then called “compact”), while delivering a result equivalent to the one defined in terms of worlds. The principle of the approach advocated is summarized in figure 1. A compact calculus valid for a subset of the relational algebra has been devised (see [4] for details). One of its characteristics is to deal with queries containing binary operations allowing for the composition of relations. In this context, the result of a query is a possibilistic relation whose interpretations correspond to more or less possible results, equivalent to those which would have been obtained with a calculus applied to worlds. This achievement is P. Bosc and O. Pivert: Vacuity-Oriented Generalized Yes/No Queries Addressed to Possibilistic Databases, Studfuzz 203, 55–73 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
56
Patrick Bosc and Olivier Pivert
interesting from a methodological point of view, but the use by an end user of the result which is delivered, is probably somewhat delicate. So, one can find it convenient to define queries which are more specialized (or targeted) to fit user needs. Vacuityoriented generalized yes/no queries, considered in the rest of this chapter, are intended to meet this goal. Their generic form is: “is the answer to Q nonempty?”. In the presence of precise data, the answer to such a query is either “yes” (if the result of Q is a nonempty relation), or “no” (in the opposite situation). However, in the possibilistic setting, imprecise data come into play, and the answer is tainted with uncertainty. It can be characterized by a double degree of possibility and certainty, both depending on the degrees of the worlds in which the result is empty or not. To reflect the uncertain nature of the result, vacuityoriented generalized yes/no queries are reformulated as follows: “to what extent is it possible and certain that the answer to Q is not empty?”.
Fig. 1. Principle for a sound compact processing against a possibilistic database.
To answer such a query, a two-step processing will take place: • first, Q is evaluated in a compact fashion (i.e., directly against the possibilistic database, see figure 1),
Vacuity-Oriented Yes/No Queries in Possibilistic Databases
57
• then, the two desired degrees are computed from the resulting relation obtained. The structure of the rest of the chapter is the following. In section 2, some basic notions on possibility theory are recalled and the notion of possibilistic relational databases is introduced. The family of yes/no queries considered in the chapter, is presented in section 3. Then, the data model required for a valid compact processing of algebraic queries is described in section 4. It is based on two simple appropriate extensions of the basic model advocated in [8] Section 5 is devoted to the presentation of the two-step processing of vacuityoriented generalized yes/no queries. Finally, the conclusion summarizes the contributions of the chapter and outlines some perspectives for future work.
2 Possibilistic relational databases 2.1 A brief reminder about possibility theory Possibility theory [9] provides an ordinal model for uncertainty where imprecision is represented by means of a preference relation coded by a total order over the possible situations. This framework is closely related to fuzzy set theory [10] since the idea is to constrain the values taken by a variable thanks to a (normalized) fuzzy set called a possibility distribution π. π(a) expresses the degree to which a is a possible value for the considered variable. The normalization condition imposes that at least one of the values of the domain (a0 ) is completely possible, i.e., π(a0 ) = 1. This setting is particularly suited to take into account un-certainty represented by linguistic terms such as “tall”, “medium” or “recent”. In the discrete case assumed later on, a possibility distribution is written {π1 /a1 , ..., πn /an } where ai is a candidate value and πi its degree of possibility. Any event E defined on the power-set of X is characterized by two measures: Π its possibility and N its certainty. The necessity N of E is defined as: N (E) = 1 − Π(E) where is the event opposite to E. The following results, where E, E1 and E2 denote events, are of interest in the context of this chapter (see [6] for more details): Π(E1 ∪ E2 ) = max(Π(E1 ), Π(E2 ))
(1)
Π(E1 ∩ E2 ) = min(Π(E1 ), Π(E2 )) if E1 and E2 are non-interactive N (E1 ∩ E2 ) = min(N (E1 ), N (E2 ))
(2) (3)
N (E1 ∪ E2 ) = max(N (E1 ), N (E2 )) if E1 and E2 are non-interactive Π(E) < 1 ⇒ N (E) = 0
(4) (5)
The two measures Π and N provide a total order over the set of regular (non fuzzy) events. They can be ordered according to Π for those which are not at all certain and according to N for those which are completely possible. This capability for rank-ordering events is important in the context of the problem dealt with in this chapter as it will be seen in section 5.
58
Patrick Bosc and Olivier Pivert
2.2 Possibilistic relational databases In contrast to a regular database, a possibilistic relational database D may have some attributes which take imprecise values. In such a case, a possibility distribution is used to represent all the more or less acceptable candidate values for the attribute. The first version of a possibilistic database model was introduced in [8]. From a semantic point of view, a possibilistic database D can be interpreted as a set of usual databases (also called worlds) W1 , ..., Wp , denoted by rep(D), each of which being more or less possible. This view establishes a semantic connection between possibilistic and regular databases. This relationship is particularly interesting since it offers a canonical approach to the definition of queries addressed to possibilistic databases as it will be seen later. Any world Wi is obtained by choosing a candidate value in each possibility distribution appearing in D. One of these (regular) databases, let us say Wk , is supposed to correspond to the actual state of the universe modeled. Any world Wi corresponds to a conjunction of independent choices and according to formula 2, the degree assigned to it is the minimum of the degrees tied to each of the chosen candidate values in the original possibilistic database D. Therefore, at least one of the worlds is completely possible, i.e., is assigned the degree of possibility Π = 1. Example 1. Let us consider the possibilistic database D involving two relations: im and pl whose respective schemas are IM(#i, ac, date, place) and PL(ac, lg, sp). The relation im describes satellite images of aircrafts and each image, identified by a number (#i), taken on a certain location (place) a given day (date) is supposed to include a single plane (ac). The relation pl gives the length (lg) and maximal speed (msp) of each aircraft. With the extensions of im and pl given hereafter, four worlds can be drawn, W1 , W2 , W3 and W4 , since there are two candidates for date in the first tuple of im and two candidates for ac in the second one. im
#i ac date place i1 a1 {1/d1 ,0.7/d3 } c1 d1 c2 i3 {1/a3 ,0.3/a4 }
pl
ac a1 a2 a3 a4
lg 20 25 18 20
msp 1000 1200 800 1200
Each of these worlds involves the relation pl which has only precise values (and then has a single interpretation) and one of the four following regular relations im1 to im4, issued from the possibilistic relation im:
Vacuity-Oriented Yes/No Queries in Possibilistic Databases im1 #i ac date place Π=1 i1 a1 d1 c1 i3 a3 d1 c2 im3 #i ac date place Π=0.3 i1 a1 d1 c1 i3 a4 d1 c2
59
im2 #i ac date place Π=0.7 i1 a1 d3 c1 i3 a3 d1 c2 im4 #i ac date place Π=0.3 i1 a1 d3 c1 i3 a4 d1 c2
The value Π specified is that of each world Wi made of the pair of relations (imi , pl) and as it is expected, one of them is completely possible. It is worth mentioning that, as far as a query Q concerns the values taken by the attributes (not their description, see [2]), the answer to Q becomes uncertain (even if in certain cases, it happens that no uncertainty occurs). This is due to the fact that the elements appearing in the answer generally depend on the choices that are made, i.e., vary with the world onto which the query may apply. This observation holds in particular for yes/no queries when imprecise values come into play (see next sections). Example 2. Let us come back to the database of example 1 and consider the query looking for the places where an aircraft whose maximal speed is over 900 km/h, has been seen. The answer contains c1 for sure (Π = N = 1), since the aircraft a1 whose maximal speed is 1000 km/h has been seen there. On the contrary, some uncertainty affects c2 , since it is in the answer only if the aircraft seen there is a4 , which is possible at the degree Π = 0.3 and therefore completely unsure (N = 0) since it is completely possible that a3 (which does not match the selection imposed as to the maximal speed) has been seen instead of a4 .
3 Vacuity-oriented generalized yes/no queries As mentioned in the introduction, the format of vacuity-oriented generalized yes/no queries is: Q1:=“to what extent is it possible and certain that the answer to Q is non-empty?” where Q is a relational query. They generalize queries of the form: Q2:=“to what extent is it possible and certain that t belongs to the answer to Q?” where t is a specified tuple, studied in [5] Let us mention that Q2-type queries have been studied in [1] in the context of null values (the difference being that
60
Patrick Bosc and Olivier Pivert
their generic formulation is “is it true that tuple t belongs to the answer to Q?”). In queries of type Q2, the specification of a desired target tuple is assumed and this may be impossible in some situations for Q1-type queries. On the other hand, it is always possible to transform a Q2-type query into a Q1-type one, by inserting additional conditions (namely those corresponding to the selection of tuple t) into Q. Example 3. Let us come back again to the database of example 1. If one wants to know the extent to which it is possible and certain that an image was taken at date d1 with an aircraft whose length is 20, one can use the query: “to what extent is it possible and certain that (d1 , 20) belongs to the answer to Q?”, where Q is the algebraic query: project(equi-join(im, av, {ac}, {ac}),{date, lg}), where project(r, Y) denotes the projection of relation r onto the set of attributes Y and equi-join(r, s, X, Y) stands for the join of relations r and s with the joining condition: X = Y. But it it also possible to call on the query: “to what extent is it possible and certain that the answer to Q’ is non-empty?”, where Q’ is the following algebraic query: equi-join(select(im, date = ’d1 ’), select(av, lg = 20), {ac}, {ac}). On the contrary, if one looks for the existence of images with aircrafts whose maximal speed is under 900 km/h and different from “ATR42”, the desired result of the underlying algebraic query cannot be characterized as a single tuple (indeed, it corresponds to a set of acceptable tuples, which can be specified only if all the names of aircrafts and all the possible speeds under 900 km/h are known). It is then necessary to call on a Q1-type query, namely: “to what extent is it possible and certain that the answer to Q” is non-empty?”, where Q” is the algebraic expression:
Vacuity-Oriented Yes/No Queries in Possibilistic Databases
61
equi-join(select(im, ac = ’ATR42’), select(av, msp < 900), {ac}, {ac}). The result of a vacuity-oriented generalized yes/no query (Q1-type above) depends on the nature (empty or not) of the answer to query Q in the worlds associated with the considered possibilistic database. Due to property 5, four cases can be distinguished: • Π = 0: it is certain that the answer to Q is empty, i.e., whatever the world chosen, Q delivers an empty answer, • 0 < Π < 1: it is not at all certain that the answer to Q is not empty (there is at least one world completely possible where the answer to Q is empty), but this is somewhat possible there is one world possible at the degree Π where the answer to Q is not empty (and no such world possible at a larger degree), • 0 < N < 1: it is somewhat certain that the answer to Q is not empty (the highest possible world where the answer to Q is empty is possible at the degree 1 − N ) and this is completely possible (there is one world completely possible where the answer to Q is not empty), • N = 1: it is completely certain (or sure) that the answer to Q is not empty (the answer to Q is not empty in any world).
4 An extended possibilistic data model 4.1 Objective As mentioned before, a calculus based on the processing of the query (Q) against every more or less possible database (world) associated with the possibilistic database is indeed intractable. Then, a compact approach to the calculus of the desired possibility and necessity degrees must be found out. In order to reach this goal, it is necessary to be provided with a data model and operations which have good properties: • the data model must be closed for the considered operations, i.e., any operation takes elements of the model as input(s) and returns such an element, • any compact query (applying to the possibilistic database D) must behave so that the interpretations of its result correspond exactly to the results of this query applied to all interpretations (worlds) drawn from D. This latter point is the characteristic property of data models called strong representation systems and this can be formalized as follows. Let O be a set of algebraic operators {op1 , , opn } and Q be a query calling only on operators of O. A data model associated with a possibilistic database D is a strong representation system for O if:
62
Patrick Bosc and Olivier Pivert
rep(Qc(D)) = Q(rep(D)) (P1) where rep(D) denotes the set of worlds associated with D and Qc stands for the query calling on the compact version of the operations. It turns out that the initial relational possibilistic model suffers from deficiencies which prevents it to comply with property P1 in two respects: • the handling of “missing tuples”, i.e., tuples of a possibilistic relation which may have no representative in some worlds, • the accounting for dependencies between candidate values, since up to now it is assumed that candidate values are independent. This means that the set of all choices is obtained as the Cartesian product of all the candidates, or alternatively that the choice of a given candidate in a distribution does not depend on any choice in any other distribution. These two aspects are detailed in the two following subsections. 4.2 Representing possibly missing tuples There is a need at the compact level for expressing that some tuples has no representative in some worlds. Indeed, some operations (e.g., selections, joins) lead to discard candidate values from a distribution in order to be compatible with the world-based interpretation (contrary to what is done in the approach suggested by Prade and Testemale, [8]). However, one must be able to compute the degree of all the worlds that can be derived from the answer, including those in which some tuples are not represented (because some of their candidate values have been discarded). As a consequence, it is mandatory to have some information in the model serving as a basis for deriving all the (more or less possible) worlds with their accurate associated degree. A solution compatible with the relational data model is to introduce a new attribute, denoted by N , which states whether or not it is legal to build worlds where the corresponding tuple is not represented, and, if so, the degree of possibility tied to this choice. The value t.N of N in a tuple t is meant to express the certainty of the presence of a representative of tuple t in any world. So doing, it is possible to generate the worlds from which a tuple is not represented, by taking into account the degree of possibility of its absence, which, according to possibility theory, is given by (1t.N). An extended possibilistic relation is a set of tuples: {< N1 /t1 >, , < Np /tp >} where Ni stands for the certainty of the event “tuple ti is represented in any interpretation of the relation”. It is worth mentioning that Ni equals 1 for tuples of initial possibilistic relations which makes initial relations compatible with the model.
Vacuity-Oriented Yes/No Queries in Possibilistic Databases
63
Example 4. Let us consider a relation r of schema is R(A, B, C). According to the previous notation, N is not considered a regular attribute and does not appear in the relation schemas. Let us assume that a selection applies to r with the condition “A ∈ {a1 , a2 }”. Let u be a tuple of t which takes an imprecise value on A, e.g.: u = 1/ < {1/a2 , 0.7/a1 , 0.7/a3 , 0.4/a4 , 0.1/a5 }, b, c > . To conform with the selection condition, the candidate values a3 , a4 and a5 must be removed from the tuple resulting from the selection. According to the approach proposed before, some track is also kept (thanks to the special attribute N ) about the fact that worlds do exist where the values for A were not solely a1 and a2 . Hence, the resulting tuple will be represented as: 0.3/ < {1/a2 , 0.7/a1 }, b, c > stating that the choice associated with the absence of a representative of this tuple in a given world is possible at the degree: 0.7 = 1 − 0.3 One may wonder about the fact that only the possibility of the most possible candidate value is taken into account. This is indeed sufficient because in possibility theory if a given event has several occurrences (with possibly different grades), these occurrences are merged into a single one which is assigned the highest of the degrees. Example 5. Let us consider a relation r whose schema is R(A, B, C) with the extension given hereafter: r
A B {1/a1 , 0.6/a2 , 0.3/a3 } b1 b2 a1 b2 a2
C c1 c2 c3
N 1 1 1
and the selection condition “A = a1 ”. The resulting relation is: res
A a1 a1
BC N b1 c1 0.4 b2 c2 1
This relation has two interpretations (worlds): one with the two tuples which is completely possible (min(min(1, 1, 1), min(1, 1, 1)) = 1) and one with only the second tuple which is possible at the degree
64
Patrick Bosc and Olivier Pivert 1/r1 A a1 a1 a2
B b1 b2 b2
C c1 c2 c3
1/r’1 A B C a1 b1 c1 a1 b2 c2
0.6/r2 A a2 a1 a2
B b1 b2 b2
C c1 c2 c3
0.6/r’2 A B C a1 b2 c2
0.3/r3 A a3 a1 a2
B b1 b2 b2
C c1 c2 c3
0.3/r’3 A B C a1 b2 c2
min(1−0.4, min(1, 1, 1)) = 0.6. From the three worlds that can be issued from r: the following resulting relations would be obtained after selection: which reduce to only two different worlds that are exactly those derived from the compact relation res, namely: A B CΠ=1 a1 b1 c1 a1 b2 c2
A B C Π = 0.6 a1 b2 c2
and it can be observed that the two relations r’2 and r’3 have the same content and they are merged into a single one with the degree of possibility max(0.6, 0.3) = 0.6. 4.3 Possibility distributions over several attributes Another aspect of the data model which needs to be revisited is related to the fact that it is sometimes necessary to express relationships (dependencies) between candidate values coming from different attributes of a same tuple. For example, let us consider a given tuple t where the two attributes A and B take the imprecise values t.A = {a1, a2} and t.B = {b1, b2, b3}, whatever the associated grades which are omitted here. If an operation retains only the pairs (a1, b1) and (a2, b3), it is impossible to represent this situation with a Cartesian product of subsets of t.A on the one hand and t.B on the other hand. In other words, A and B values cannot be kept separate (which would mean that they are independent) and the correct associations must be explicitly represented. This requires that the model incorporates attribute values defined as possibility distributions over several domains. This is feasible in the scope of the relational data model thanks to the concept of a nested relation. More precisely, candidates will be represented as weighted tuples which have a disjunctive semantics, i.e., they are mutually exclusive. Therefore, non-nested tuples keep their conjunctive meaning, whereas nested ones have a disjunctive interpretation since they represent alternative candidates. In the remainder, the notation R(A1 , , Am , X1 (Ap , , Aq ), , Xn (Ak , , Ar )) stands for a schema where A1 to Am are elementary attributes (also called level-one attributes)
Vacuity-Oriented Yes/No Queries in Possibilistic Databases
65
whose values are either precise or not (possibility distributions) and Xi (Ah , , Aj ) represents a “structured” attribute Xi whose values are possibility distributions made of tuples built over attributes Ah to Aj which are called “nested” attributes. Obviously, such relations have an interpretation in terms of worlds as it is the case for ordinary possibilistic relations. When one moves to a given world, a structured candidate value is split into atomic values and the schema becomes a non-nested one. Example 6. Let us consider a relation r whose schema is R(A, B, C, D), with the extension: r
A B C DN a1 {1/8,0.8/10,0.4/12} {1/9,0.7/15} d1 1 5 {1/11,1/12} d3 1 a2
and the selection condition “B < C”. The resulting relation will include a nested relation gathering the attributes B and C and the result is: res
X D N B C a1 {1/<8,9>,0.7/<8,15>,0.7/<10,15>,0.4/<12,15>} d1 0.2 {1/<5, 11>, 1/<5, 12>} d3 1.0 a2 A
Two points are worthy of comments: • the certainty degree of the first tuple is 0.2 since the candidates 0.8/<10,9> and 0.4/<12,9> have been removed and the degree of the most possible one is 0.8, • although no candidate is discarded in the second tuple, its representation must be modified in order to cope with the structure of the resulting relation.
5 Processing vacuity-oriented generalized yes/no queries 5.1 Principle of the approach The principle of the method described hereafter is outlined in figure 2, which is an adaptation of figure 1 to extended yes/no queries. By construction, the compact processing, which is sound, general and tractable is based on a twostep mechanism: • a compact calculus of an algebraic query such that the equivalence with the individual worlds is guaranteed, which builds a compact possibilistic relation (complying with the data model presented before),
66
Patrick Bosc and Olivier Pivert
Fig. 2. Principle for processing vacuity-oriented generalized yes/no queries.
• a post-processing aimed at the calculation of the two degrees constituting the final answer to the user’s query. The first step relies on a processing strategy that has been pointed out for dealing with algebraic queries [4] Of course, the price to pay for the compliance with property P1 is that only a restricted number of operations are allowed in Q with respect to the usual relational algebra. 5.2 Compact version of relational operators It turns out that only four operators can be processed in a compact way for the following reason. In order to comply with property P1, any operator o must necessarily be so that the following property holds (P2): when a tuple t1 of an input relation is used by o, any part of t1 involving imprecise information must appear in at most one tuple of the output relation Proof. Let us consider a tuple u of a relation r whose A-component is imprecise and described by means of a possibility distribution: π = {d1 /a1 ,..., dn /an }. Let us assume that u.A (i.e., π) appears in two tuples: v1 = N1 / < x1 , π, y1 >, v2 = N2 / < x2 , π, y2 > of the output relation (whose schema is (X, A, Y)). Tuples v1 and v2 are supposed to stem from u (and possibly some other tuples from another relation). Due to the independence hypothesis assumed when worlds are built, it is possible to derive the following world from the output relation res:
Vacuity-Oriented Yes/No Queries in Possibilistic Databases
X ... x1 x2 ...
A ... ai aj ...
67
Y ... y1 y2 ...
However, this world is illegal (if ai = aj ), since in any original world, the value of A is either ai , or aj . The consequence of the preceding constraint (P2) is that Cartesian products (and then joins), intersections and differences (all of them in their most general form) cannot be expected to be acceptable. Now, the objective is mainly to present the key points governing the four legal operators. The interested reader will find more details about them in [4], in particular, in terms of proofs of compliance with properties P1 and P2. The selection When imprecise data come into play, two aspects must be handled for a selection: • the determination of the schema of the output relation, which depends on the schema of the input relation and the nature of the selection condition. The schema of the resulting relation remains unchanged for a condition of the form “attribute θ constant” (θ ∈ {=, =, >, <, ≥, ≤}), as well as for a conjunction of such conditions. On the other hand, if the condition is of the form “attribute1 θ attribute2 ” or “attribute1 θ1 const1 or attribute2 θ2 const2 ”, or more generally “cond1 (attribute1 ) or cond2 (attribute2 )”, it is necessary to introduce distributions defined over both attribute1 and attribute2 ; • for each tuple t of the input relation, a tuple of the result is produced according to the following procedure (provided that at least one candidate value satisfies the selection condition): – removal of any candidate value of t (possibly defined over several attributes in the case of nested relations) concerned with the selection condition which does not satisfy the selection condition, – update of N by taking into account the removal of the most possible candidate value (whose possibility degree is denoted by πM ). More precisely, the value of N in the resulting tuple is the minimum of πM and its value in the initial tuple t (t.N ). The selection is illustrated in example 5. The projection The regular projection is intended for the removal of “useless” attributes. Four aspects have to be taken into account for the projection of a relation r on a subset Y of its attributes in the presence of imprecise values:
68
Patrick Bosc and Olivier Pivert
• the impact of the projection on the value of the attribute N in each tuple: indeed, this value is not affected by a projection and it remains unchanged in the output tuple; • the role of duplicates: their handling is a little bit more delicate because tuples in level-one relations are conjunctive, whereas they are mutually exclusive in a nested relation where they represent alternative candidates. The appropriate way of doing is the following: – duplicate tuples may appear in level-one tuples, because possibility distributions are representations and not values strictly speaking. This is necessary to recover all the legal interpretations of the projected relation. For instance, if relation r whose schema is R(A, B) contains the tuples < {π1 /a1 , ..., πn /an }, b1 > and < {π1 /a1 , ..., πn /an }, b2 >, the projection over A results in a relation with two identical tuples. This makes it possible to have a world with two tuples < ai > and < aj >, which would be quite impossible with a single tuple kept in the projection, – duplicates are removed inside nested relations, since these tuples represent alternative choices for which duplicates are meaningless. According to possibility theory, the highest possibility degree is assigned in such a case; • the structure of the resulting relation: two cases involving a nested relation are worthy of comments: – if all the attributes of a nested relation are suppressed, the nested relation totally disappears, – if only one of the attributes of a nested relation is retained, the nested relation becomes a level-one attribute. • the grades: when an attribute with an imprecise value or a nested relation disappears, the possibility of the most possible candidate is forwarded (by means of a minimum) to one of the remaining attributes. So doing, it is guaranteed that no world can be drawn with a possibility greater than that of the corresponding world before projection. Example 7. Let us consider a relation of schema (A, X(B, C), D) involving a nested relation: r
A
X B C a1 {1/< b1 , c1 >,0.7/< b1 , c2 >,0.7/< b2 , c2 >, 0.4/< b3 , c2 >} a2 {1/< b1 , c4 >, 1/< b1 , c2 >}
D
N
{0.6/d1 } 0.2 d3
1
The projection of r on the attributes A and C produces the following result : It is worth noticing that the degree 0.6 initially attached to d1 has impacted the attribute A in the first tuple of the result (it would have been possible to forward it to C as well, or even to both A and C).
Vacuity-Oriented Yes/No Queries in Possibilistic Databases res
69
A B N {0.6/a1 } {1/c1 ,0.7/c2 } 0.2 {1/c4 ,1/c2 } 1 a2
The foreign-key join (fk-join) Although there is no hope for defining an acceptable compact version of the join in general, it turns out that the necessary condition highlighted at the beginning of subsection 5.2 (P2) can be satisfied by a specific join, namely the fk-join. The fk-join is a binary operator which allows for the composition of two relations. Let us consider the relation schemas R(W, Z) and S(W, Y), a possibilistic relation r(R) where W and and Z may take imprecise values and a regular relation s(S) where the functional dependency W →Y is valid. W is then the key of s and a foreign key of r. The principle of the fk-join consists in completing relation r by adding the image of the W-component (via W → Y) into each tuple. By definition, this leads to a resulting relation whose schema (X(W, Y), Z) involves a nested relation, which “connects” the pairs of candidates over W and Y. The degree of possibility of any structured candidate value is the one of the value that has been completed. Similarly to the selection, the attribute N is updated to keep track of the most possible candidate value for which no match occurred. This operator works in a similar way when the attribute(s) involved are already part of a nested relation. For instance, with relations r and s whose respective schemas are R(X1(W, T), V) and S(W, Z), the schema of the resulting relation is (X2(W, Z, T), V), i.e., the set of attributes Z is inserted in the nested relation of the result. An illustration of the fk-join is given in the example worked out in 5.4. The union An interesting point is the fact that there is no interaction between tuples to produce an element of the union of two compatible relations (i.e., whose attributes are pairwise defined on the same domain). Then, the necessary condition (P2) evoked before is satisfied and the only remaining condition in order to comply with property P1 concerns the independence of the two input relations (i.e., choices of candidates among tuples can be made without any restriction). If this assumption is valid, two points are worthy of comments: • the possible presence of nested relations inside the input relations. It may happen that two possibilistic relations have compatible attributes, but involved in nested relations whose schemas differ. In such a situation, it is necessary to define a common schema, which will be that of the resulting relation. For instance, the union of relations r and s whose respective schemas are (X1(A, B), C, D, X2(E, F), G) and (A, B, C, X3(D, E), F,
70
Patrick Bosc and Olivier Pivert
G) makes sense and the common schema is (X4(A, B), C, X5(D, E, F), G); • the handling of duplicates. For reasons similar to those explained for the projection, duplicates must not be removed in order to be able to recover all the possible worlds (interpretations). Finally, when attributes of the relations may take imprecise values, the union is defined by putting all their tuples into the result (along with their degree of certainty) after a possible homogenization of the schema (if necessary). 5.3 Computation of the two final degrees Once the algebraic component Q involved in a vacuity-oriented generalized yes/no query has been processed according to the previous strategy, a compact resulting relation res is produced. The degrees of possibility and necessity, which constitute the answer to the vacuity-oriented generalized yes/no query, can be deduced from res in the following way: • if res is empty, then the possibility degree equals 0. Due to formula 5, N also equals 0. In other words, it is certain that the answer to Q is empty; • if res contains at least one tuple tj whose N-value (Nj ) is strictly positive, the necessity degree that the answer to Q is not empty is given by: maxti∈res Ni . Proof. Let res be made of the tuples {N1 /t1 , ..., Np /tp }. The only empty world that can be derived from res is the one where neither t1 , ..., nor tp has a representative. According to formula 2, the degree of possibility of such a world is: min(1 N1 , ..., 1 Np ) since the events considered are non-interactive. In particular, there is no such world if one of the Ni ’s equals 1. Thus: N(the answer to Q is not empty) = 1 −Π(res has an empty representative) = max(N1 ,..., Np ). The contraposition of formula 5 (N > 0 ⇒ Π = 1) allows to state that the possibility degree is 1. In other words, it is completely possible that the answer to Q is not empty (and it may be certain in the special case where N itself is 1); • if the N -value of all the tuples of res equals 0, the certainty that the answer to Q is not empty is 0 (cf. the previous proof) and, according to formula 1, its possibility is given by the possibility of the highest possible interpretation of the tuples involved in res, i.e., the one obtained in taking the highest degree attached to the tuples appearing in res. 5.4 An example This example is intended for illustrating the overall functioning of the procedure designed for answering vacuity-oriented generalized yes/no queries
Vacuity-Oriented Yes/No Queries in Possibilistic Databases
71
addressed to possibilistic databases. Let us consider the possibilistic database composed of the relations im1(IM), im2(IM) and pl(PL) where IM and PL are the schemas introduced in example 1. The two relations im1 and im2 are assumed to contain images of aircrafts taken by two distinct sources (e.g., satellites). Let us take the query: “to what extent is it possible and certain that there exists at least one shot with an aircraft of length 20 and maximal speed greater than 900 km/h, taken on date d1?”. More formally this query is: “to what extent is it possible and certain that the result of Q is nonempty?”, where Q = fk-join(union(select(im1, date = ’d1’), select(im2, date = ’d1’)), select(pl, lg = 20 and msp > 900), {ac}, {ac}), which, optionally, a projection on the attribute #i can be added to. With the extensions of relations im1, im2 and pl given hereafter: im1
#i ac date place N i1 a3 {1/d1 ,0.7/d3 } c1 1 c2 1 i2 {1/a2 ,0.7/a1 } d1
im1
#i ac date place N i3 {1/a5 ,0.4/a3 } {0.6/d4 ,1/d1 } c3 1
pl
ac a1 a2 a3 a4 a5
lg 20 25 18 20 20
msp 1000 1200 600 1200 1000
the selection on im1 (resp. im2, pl) creates the intermediate relation im11 (resp. im21, pl1): im11
#i ac date i1 a3 {1/d1 } i2 {1/a2 ,0.7/a1 } d1
place N c1 0.3 c2 1
im21
#i ac date i3 {1/a5 ,0.4/a3 } {1/d1 }
place N c3 0.3
pl1
ac a1 a4 a5
lg 20 20 20
msp 1000 1200 1000
After the union of im11 and im21, the fk-join with pl1 delivers the result: From this table, it can be deduced that the possibility and the certainty that the answer to Q is not empty are respectively: Π = 1, N = 0.4
72
Patrick Bosc and Olivier Pivert res
#i i2 i3
X date place N ac lg msp {0.7/} d1 c2 0 {1/} {1/d1 } c3 0.4
If a final projection appears in Q, the resulting table obtained is: #i N {0.7/i2 } 0 0.4 i3 and the same values of possibility and necessity are derived from it.
6 Conclusion This chapter addresses the issue of querying relational databases where some attribute values are imprecisely known and represented as possibility distributions. For tractability reasons, a specific family of queries, called vacuityoriented generalized yes/no queries, has been investigated. The general format of such queries is: “to what extent is it possible and certain that the answer to Q is non-empty?” , where Q is an algebraic query. It has been shown that these queries could be processed in a compact (and then reasonable from a performance viewpoint) way at the price of some restrictions as to the operations present in Q. Four of the regular algebraic operators have been extended to work directly on possibilistic relations. To this aim, it is necessary to enrich the initial possibilistic relational data model. In the more specific case of queries calling only on the possibility, another compact processing strategy can be devised where a wider range of algebraic operations are allowed [3]. This work opens several lines for future research. One of them is related to the influence of the model of uncertainty chosen to represent ill-known information. We conjecture that it should be feasible to replace possibility distributions by probability distributions, because the processing strategy is founded on the equivalence between the treatment on the possibilistic database D and the one on the worlds, i.e., the regular databases, associated with D. Of course, the definition of the compact operators, as well as the final calculus of degrees should be adapted appropriately to the probabilistic framework. Another subject of interest would concern the introduction of fuzzy algebraic queries in the body of vacuityoriented generalized yes/no queries. Since, fuzzy sets and possibility theory share the same semantics of degrees (i.e., preferences over candidates), there is no fundamental obstacle to deal with the two aspects: imprecise data represented as possibility distributions and flexible queries with gradual predicates and connectors modeled in the framework of fuzzy sets.
Vacuity-Oriented Yes/No Queries in Possibilistic Databases
73
References [1] S. Abiteboul, P. Kanellakis, and G. Grahne. On the representation and querying of sets of possible worlds. Theoretical Computer Science, 78:159187, 1991. [2] P. Bosc, L. Duval, and O. Pivert. Value-based and representation-based querying of possibilistic databases. In Recent Issues of Fuzzy Databases, pages 3–27. Physica Verlag, 2000. [3] P. Bosc, L. Duval, and O. Pivert. An initial approach to the evaluation of possibilistic queries addressed to possibilistic databases. Fuzzy Sets and Systems, 140:151166, 2003. [4] P. Bosc and O. Pivert. Towards an algebraic query language for possibilistic databases. In Proc. of the 12th Conference on Fuzzy Systems (FUZZ-IEEE’03), 2003. [5] P. Bosc and O. Pivert. About yes/no queries against possibilistic databases. Journal of Intelligent Systems, To appear in, 2005. [6] D. Dubois and H. Prade. Fuzzy Sets and Systems Theory and Applications. Academic Press, 1980. [7] T. Imielinski and W. Lipski. Incomplete information in relational databases. Journal of the ACM, 31:761791, 1984. [8] H. Prade and C. Testemale. Generalizing database relational algebra for the treatment of incomplete or uncertain information and vague queries. Information Sciences, 34:115–143, 1984. [9] L. A. Zadeh. Fuzzy sets as a basis for a theory of possibility. Fuzzy Sets and Systems, 1:3–28, 1978. [10] L.A. Zadeh. Fuzzy sets. Information and Control, 8:338–353, 1965.
On the Applicability of Extended Possibilistic Truth Values in Flexible Database Modelling and Querying Guy de Tr´e and Rita de Caluwe Ghent University Department of Telecommunications and Information Processing Sint-Pietersnieuwstraat 41, B-9000 Ghent (Belgium) {Guy.DETRE,Rita.DECALUWE}@UGent.be
1 Introduction The mathematical logic, supporting a database model, largely determines the behavior and interaction of the model on constraint satisfaction and on database querying. For traditional database models that only handle perfectly described data and do not explicitly deal with missing information, the classical two-valued Boolean logic is well suited: both integrity constraints and querying criteria can be described by Boolean expressions, which should evaluate to true in order to be satisfied. If pseudo-descriptions, commonly known as ‘null’-values, are admitted into a database model to cope with missing information, a two-valued logic is no longer sufficient in order to define the impact of transformations and modifications in the presence of nulls [1; 2]. In his approach, Edgar Codd extends the relational database model based on an underlying three-valued logic [2] in order to formalize the semantics of null values in traditional relational databases. This approach has even been further extended by using a four-valued logic, hereby making an explicit distinction between nulls that represent unknown information (i.e. information that exists but is not available) and nulls that represent inapplicable information (i.e. information that is missing because it is not applicable) [4]. Due to the fact that the law of the excluded middle no longer holds for many-valued logics, such extensions have been the subject of criticism [5]. If one wants to model imperfect information more efficiently or if one wants to enhance the database access facilities by dealing with flexible database querying, the demand for a more elaborated logic to support database modelling even becomes more stringent. In these situations, query satisfaction and constraint satisfaction become a matter of degree and require a logical
G. de Tr´ e and R. de Caluwe: On the Applicability of Extended Possibilistic Truth Values in Flexible Database Modelling and Querying, Studfuzz 203, 75–90 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
76
Guy de Tr´e and Rita de Caluwe
framework that allows us to reflect the knowledge of the actual truth of a proposition in a rather natural way. In this chapter, it is discussed how a many-valued logic, based on extended possibilistic truth values (EPTVs), meets these requirements and moreover helps to overcome some of the problems experienced in the null value approach in a more natural way. The main research results on EPTVs are brought together in the framework of flexible database modelling and compared with other traditional and ‘fuzzy’ approaches. The remainder of the chapter is organized as follows: In Section 2, preliminary definitions on EPTVs are presented. In Section 3, the suitability of the presented logic to support (flexible) data and database modelling is discussed. Section 4 deals with the support of flexible querying. An elaborate example, illustrating both database modelling and database querying, is given in Section 5. In Section 6 a comparison with other approaches is given. Finally, conclusions are given in Section 7.
2 Some preliminaries on EPTVs The concept of an EPTV [7] is an extension of the concept of a possibilistic truth value (PTV) as originally presented by Henri Prade [15] and further developed by Gert De Cooman [6]. The basic idea behind EPTV’s is that one considers a universe I ∗ of three truth values: true (T ), false (F ) and undefined (⊥), the latter being a pseudo-description for those situations where information is inapplicable and a Boolean truth value could not be used without a lost of information. For example, with this consideration the truth value of the proposition ‘is salary higher than 2.000 Euro ?’ for a retired person is ⊥, which denotes that a (condition on) salary is not applicable for retired persons. A truth value F should in this case be less informative and just indicate that the person does not have a salary that is higher than 2.000 Euro. EPTVs are defined as disjunctive fuzzy sets [17] over I ∗ , where the membership grades are interpreted as degrees of possibility. This means that each EPTV has the form {(T, µT ), (F, µF ), (⊥, µ⊥ )} where µT , µF , µ⊥ ∈ [0, 1]. Furthermore, µT denotes the possibility that the EPTV is true, µF denotes the possibility that the EPTV is false and µ⊥ denotes the possibility that the EPTV is undefined. For example, the EPTV {(T, 1), (F, 0.2)} expresses ‘either completely possible true or less possibly (with possibility 0.2) false’. The associated membership grades allow for a gradual representation of truth and uncertainty about truth and provide more flexibility to support flexible database modelling and flexible database querying. Special cases of EPTVs are: The formal definition of EPTVs and an overview of their most important properties and logical operators are given in [7].
EPTVs in Flexible Database Modelling and Querying
77
Table 1. Special cases of EPTVs. EPTV {(T, 1)} {(F, 1)} {(T, 1), (F, 1)} {(⊥, 1)} {(T, 1), (F, 1), (⊥, 1)}
Semantics True False Unknown (either true or false) Inapplicable No information
In order to be of practical usefulness, definitions of the basic logical opera˜ ), disjunction (∨ ˜ ) and negation (˜ tors for conjunction (∧ ¬) have to be provided. A notation with tilde is used here, to explicitly denote that that these operators are defined on EPTVs. This way, they can easily be distinguished from their counterparts in a classical three-valued logic. In order to preserve compatibility with classical logic, it is important that truth-functionality is as complete as possible (complete truth-functionality can not be obtained in many-valued logics [14]). This means that the behavior of a logical operator should be mirrored in a logical function combining EPTVs, i.e., with the understanding that P represents the universe of all propositions, t : P → {T, F } is the mapping that associates a traditional ˜ ∗ ) is the mapping that associates truth value with each p ∈ P and t˜∗ → ℘(I ∗ an EPTV with each p ∈ P (where ℘(I ˜ ) denotes the set of fuzzy sets on I ∗ ), the following rules should hold ˜ t˜∗ (q) • p, q ∈ P , q = ¬p : t˜∗ (p ∧ q) = t˜∗ (p)∧ ∗ ∗ ˜ ˜ ˜ t˜∗ (q) • p, q ∈ P , q = ¬p : t (p ∨ q) = t (p)∨ ∗ ∗ ˜ ˜ ˜ t (p) • p ∈ P : t (¬p) = ¬ Several definitions for logical operators have been developed, all of them with their own advantages and drawbacks. An overview and comparison of definitions that are respectively based on Zadeh’s extension principle, the minimum t-norm operation and the algebraic t-norm operation can be found in [11]. As an illustration the definitions, based on the minimum t-norm operation, are given. With the understanding that ℘(I ˜ ∗ ) denotes the set of fuzzy sets on I ∗ , the conjunctive aggregation operator ˜ : ℘(I ˜ ∗ ) → ℘(I ˜ ∗) ∧ ˜ ∗ ) × ℘(I ˜ , V˜ ) → U ˜∧ ˜ V˜ (U is defined by • µU˜ ∧˜ V˜ (T ) = min(µU˜ (T ), µV˜ (T )) • µU˜ ∧˜ V˜ (F ) = max(µ U˜ (F ), µV˜ (F )) min(µU˜ (T ), µV˜ (⊥)), • µU˜ ∧˜ V˜ (⊥) = max min(µU˜ (⊥), µV˜ (T )), min(µU˜ (⊥), µV˜ (⊥))
78
Guy de Tr´e and Rita de Caluwe
The disjunctive aggregation operator ˜ : ℘(I ˜ ∗ ) → ℘(I ˜ ∗) ∨ ˜ ∗ ) × ℘(I ˜ , V˜ ) → U ˜∨ ˜ V˜ (U is defined by • µU˜ ∨˜ V˜ (T ) = max(µU˜ (T ), µV˜ (T )) • µU˜ ∨˜ V˜ (F ) = min(µ U˜ (F ), µV˜ (F )) min(µU˜ (F ), µV˜ (⊥)), • µU˜ ∨˜ V˜ (⊥) = max min(µU˜ (⊥), µV˜ (F )), min(µU˜ (⊥), µV˜ (⊥)) Finally, the negation operator ¬ ˜ : ℘(I ˜ ∗ ) → ℘(I ˜ ∗) ˜ )˜ ˜ (U ¬U is defined by • µ¬˜ U˜ (T ) = µU˜ (F ) • µ¬˜ U˜ (F ) = µU˜ (T ) • µ¬˜ U˜ (⊥) = µU˜ (⊥)
3 Supporting flexible database modelling Equipped with operators that almost preserve truth-functionality, EPTVs are well suited to support the modelling of imperfect and missing information in databases. This is especially the case if attribute domains are extended with domain specific symbols (⊥t ) that are used to model the inapplicability of a regular domain value (and thus support nulls that represent inapplicability) and if possibility distributions, defined over the attribute domains, are used to model vague, imprecise or uncertain attribute values, as outlined in [16]. In such cases, completely unknown information can be modelled by uniform possibility distributions that associate the value one with each regular domain value. In fact, a seamless integration of EPTVs in the type system of a database model is obtained by extending the domain of the Boolean type with a truth value ⊥ and by allowing the use of possibility distributions that are defined over this extended domain. Additionally, the definitions of all operators within the type system should be extended in order to cope with the extra domain values ⊥t . In most cases this could be done as follows op(v1 , ⊥t2 ) = op(⊥t1 , v2 ) = ⊥t where op is an operator that has to be extended, ⊥ti , i = 1, 2 is the extra value of the domain domti of argument type ti , vi , i = 1, 2 is a value of domti and ⊥t is the extra value of the domain of the result type t of op.
EPTVs in Flexible Database Modelling and Querying
79
EPTVs are also well suited to express constraint satisfaction and therefore support the use of (flexible) integrity constraints in database modelling. Traditionally, integrity constraints are seen as properties of the universe and, as such, they impact database records when updates are performed. As any other constraint, an integrity constraint can be softened. For some database management tasks, as for example data integration, this can be advantageously [10]. Typical for flexible integrity constraints is that they can impose soft restrictions on the database, which can be useful in situations where one can not express the semantic constraints on the data by means of a strict restriction. With respect to an update, insertion or deletion of a database record, three situations can be considered: 1. All (flexible) integrity constraints are completely satisfied. In this case, the constraints behave as if they were regular integrity constraints. The update, insertion or deletion can be safely executed on the database. 2. At least one of the flexible integrity constraints is partially satisfied. In this case, deletion should not be allowed, as it would make the database invalid. Insertion and update are allowed and have as effect that the record under consideration should only partially belong to the database. This can be modelled by means of EPTVs as described and illustrated below. 3. None of the (flexible) integrity constraints is satisfied at all. In this case, the constraints also behave as regular integrity constraints. The update, insertion or deletion operation is not allowed. For example, instead of limiting the valid temperatures in Belgium within the range [−25 ◦ C, 40 ◦ C], a soft restriction, e.g., allows it to limit these temperatures to the more flexible range from ‘about −20 ◦ C’ to ‘about 40 ◦ C’. Constraint satisfaction in the presence of soft restrictions may not be known with certainty and therefore requires a gradual representation. EPTVs can be used for this purpose. If an EPTV {(T, µT ), (F, µF ), (⊥, µ⊥ )} is used to express constraint satisfaction, µT denotes the possibility that the constraint is satisfied, µF denotes the possibility that the constraint is not satisfied and µ⊥ denotes the possibility that the constraint does not apply to the considered database record. Furthermore, to each record in a database an EPTV, resulting from the aggregation of the EPTVs of all integrity constraints that are defined for the record, can be attached. Such an EPTV then expresses to which extent the record satisfies all integrity constraints that have been defined for it. In the next section, it will be illustrated how these attached EPTVs can be calculated and aggregated. With the description and illustration given above, it should be clear that EPTVs are suited to support flexible database modelling. More details about
80
Guy de Tr´e and Rita de Caluwe
the generality of the presented approach and a further description of different kinds of flexible integrity constraints are beyond the scope of this chapter, but can be found in [9].
4 Supporting flexible database querying Because a (soft) selection condition imposed by a (flexible) query can be seen as a special case of a constraint, EPTVs can also be used to express query satisfaction. What is initially needed are appropriate evaluation functions for comparison operators. The basic comparison operators can be defined as follows: assume that ˜ and V˜ are fuzzy op ∈ {=, =, ≤, ≥, <, >} denotes a comparison operator and U sets that are defined over the domain domt of an attribute with associated type t. Furthermore, assume that domt contains the extra value ⊥t , which is used to model inapplicability. Then the EPTV of the comparison ˜ op V˜ U is defined by • µU˜ • µU˜ • µU˜
op V˜ (T ) op V˜ (F )
= supx,y∈(domt )2 :x op y min(µU˜ (x), µV˜ (y)) = supx,y∈(domt \{⊥t })2 :¬ (x op y) min(µU˜ (x), µV˜ (y)) (⊥) = min(µU˜ (⊥t ), µV˜ (⊥t )) op V˜
Another frequently used comparison operator is the so-called ‘is’ operator. This operator allows to compare stored (labels of) possibility distributions with (labels of) fuzzy sets provided by the user in a query. The general form of the ‘is’ operator is A is L where A is the name of a database attribute and L is a linguistic term that is given by the user. For example, if A is the Salary attribute of an employee record whose current value is denoted by the label ‘about 3.000 Euro’ that is modelled by a possibility distribution function π‘about 3.000 Euro’ and L is the label of the fuzzy set that represents ‘high salary’, then the calculated EPTV of the proposition ‘Salary is high salary’ will express to what extent ‘about 3.000 Euro’ is a ‘high salary’. From the possibilistic logic viewpoint, the ‘is’ operator should be interpreted as follows. L is represented by a fuzzy set with membership function µL and πA is a possibility distribution being a value of the attribute A at given record. p = A IS L is treated as a proposition of a multi-valued logic, thus possessing a fuzzy set of models, M . On the other hand, πA is a possibility distribution on the space of interpretations. An interpretation is meant here as an assignment of a value to the attribute A. Evaluation of the proposition p corresponds to the computation of an EPTV and is, in accordance with [13], defined by
EPTVs in Flexible Database Modelling and Querying
81
• µA is L (T ) = supx∈domt min(πA (x), µL (x)) • µA is L (F ) = supx∈domt \{⊥t } min(πA (x), 1 − µL (x)) • µA is L (⊥) = min(πA (⊥t ), 1 − µL (⊥t )) In flexible querying, weights may be associated with the query criteria, hereby denoting that some criteria might be more relevant (or stringent) than others. In order to cope with such weights, some more advanced aggregation operators are needed. From a conceptual point of view, a distinction can be made between static weights and dynamic weights. Static weights are fixed, known in advance and can be directly derived from the formulation of the query. These weights are independent of the values of the record(s) on which the query criteria act and are not allowed to change during query processing. A further distinction can be made between static weight assignments, where it is also known in advance with which constraint a weight is associated (e.g. in a situation where the user explicitly states his/her preferences) and dynamic weight assignments, where the associations between weights and criteria depend on the actual attribute values of the record(s) on which the query criteria act (e.g. in a situation where most criteria have to be satisfied, but it is not important which ones). In aggregation with dynamic weights, neither the weights nor the associations between weights and criteria are known in advance and depend on the attribute values of the record(s) on which the query criteria act. This kind of flexibility is required to avoid some unnatural behavior of the query evaluation in cases where, e.g., a criterion is of limited importance only within a given range of values, for example, if the criterion ‘high salary’ is not important, unless the salary value is extremely high. Several advanced aggregation operators for EPTVs have been studied by the authors. An example of an advanced aggregation operator for EPTVs that deals with fixed, statically assigned weights is the operator based on residual (co)implicator functions presented in [8]. However, more research on aggregation operators within the framework of EPTVs is necessary to be able to cope with all the different kinds of weight handling as described above. Another important aspect of flexible querying is the ranking of the possible alternatives in the result of a query. In fact, because query satisfaction is no longer a matter of bivalence (true/false) but a matter of degree, the supporting logical framework should provide adequate ranking facilities. Such facilities are necessary for the efficient interpretation and satisfactory representation of the (relative importance of the) results of a query. The simplest way to rank EPTVs is by using the so-called analytical ranking function ra that maps each EPTV {(T, µT ), (F, µF ), (⊥, µ⊥ )} onto its associated ranking value (1 + (µT − µF )(1 − µ⊥ /2)) ∈ [0, 1] 2 An alternative approach is to assume the following initial ordering of the ‘basic’ truth values
82
Guy de Tr´e and Rita de Caluwe
{(T, µT )} > {(⊥, µ⊥ )} > {(F, µF )} and to consider the impact of the (F)alse component on the ranking process to be more negative than the impact of the ⊥-component. This can be obtained by introducing a threshold value α for the ⊥-component and by subsequently ranking the EPTVs as if there were PTVs. The EPTVs are then ranked with decreasing possibilities of being true (i.e. µT ) and the additional restriction that their respective possibilities of being inapplicable (i.e. µ⊥ ) must not be greater than α. The resulting mapping function rα then maps each EPTV {(T, µT ), (F, µF ), (⊥, µ⊥ )} onto its associated ranking value µT + (1 − µF ) if µ⊥ ≤ α 2 0 if µ > α ⊥
More information on ranking functions for EPTVs, including a third ranking method, has been presented in [12]. In this section it is illustrated that EPTVs are suited to support flexible database querying. To preserve generality and to be independent of any database model, only selection conditions are dealt with. However, when dealing with a specific database model, as for example the relational database model [3], other data manipulation operators need to be considered. This is beyond the scope of this chapter. More details about the handling and behavior of such operators within a logical framework based on EPTVs can be found in [9].
5 An example As an illustrative example a simple relational database [3] consisting of a single relation ‘High buildings’ is considered. 5.1 Flexible database modelling The relation High buildings contains some facts about buildings, which can generally be categorized as being high. Each tuple t in High buildings represents a building and is characterized by a unique building name (name) —which is the primary key attribute—, the city where the building is located (location),the height of the building in metres (height), the number of storeys in the building (storeys) and the year when the building the construction of the building was finished (year). To assure that the relation only contains tuples that represent high buildings, a soft integrity constraint ‘height IS high’
EPTVs in Flexible Database Modelling and Querying
83
Fig. 1. The modelling of ‘high’ buildings.
is defined for the relation. Hereby, high is a linguistic term being represented by the fuzzy set with membership function as depicted in Figure 1. As extra information an EPTV, expressing the truth of the proposition that the tuple belongs to the relation is attached to each tuple (Contains). In fact, this EPTV expresses the integrity constraint satisfaction of each tuple. Each attribute Ai has an associated domain domAi of valid values. Each associated domain domAi contains a domain specific ‘undefined’ element ⊥Ai that is used to model cases where a regular domain value is not applicable, as described in [16]. Table 2. Example of instances of relation High buildings. N ame Location Height Storeys Y ear Contains Empire State New York 381 102 1931 {(T, 1)} Petronas Towers Kuala Lumpur 452 88 1998 {(T, 1)} Taipei 101 Taipei 509 101 2004 {(T, 1)} Chrysler Building New York 319 77 1930 {(T, 1)} Eiffel Tower Paris 300.5±0.2 3 1889 {(T, 1)} Central Plaza Hong Kong 374 78 1992 {(T, 1)} Sears Tower Chicago 442 110 1974 {(T, 1)} Great Pyramid Gizeh 140±5 N/A ±2570BC {(T, 0.2), (F, 0.8)} Trump Building New York 283 72 1930 {(T, 0.9), (F, 0.1)} Warsaw TV Mast Warsaw 646 N/A 1973 {(T, 1)} CITIC Plaza Guangzhou 391 80 1996 {(T, 1)}
Table 2 is a tabular representation of an example of High buildings (each row represents a tuple, also called an instance of the relation). The values of the ‘height’- and ‘year’-attributes are labels, each of them being characterized by a possibility distribution function that is associated with the attribute and is defined over the domain of the attribute. For example, a label denoting a ‘crisp’ number like 381 corresponds to a possibility distribution function with a discrete support, labels like ‘300.5±0.2’ correspond to triangular possibility distribution functions. These possibility distributions model that the attribute value is imprecise. For example, the height of the Eiffel Tower is known to be 300.51 metres (± 15cm depending on temperature). Likewise, at construction
84
Guy de Tr´e and Rita de Caluwe
the Great Pyramid was about 146 metres, but due to erosion its current height is about 137 metres, what is modelled by the label ‘140±5’. The label ‘N/A’ stands for ‘not applicable’ and corresponds to a possibility distribution function ΠN/A (x) = 0, if x ∈ domAi \ {⊥Ai } = 1, if x = ⊥Ai This label is used to denote that the storeys attribute does not apply for the ‘Great Pyramid’ and the ‘Warsaw TV Mast’. In a similar way, one can consider a label ‘UNK’ that stands for ‘unknown (but applicable)’ and corresponds to a (rectangular) possibility distribution function ΠU N K (x) = 1, if x ∈ domAi \ {⊥Ai } = 0, if x = ⊥Ai and a label ‘UNA’ that stands for ‘unavailable’ and consequently corresponds to the possibility distribution function ΠU N A (x) = 1, ∀x ∈ domAi Due to the soft integrity constraint ‘height IS high’ the ‘Great Pyramid’ is still considered as being a high building, but to a lesser extent than other more recent buildings. This is reflected by the associated EPTV in the Contains attribute field, that is calculated by applying the formula for ‘IS’ predicates, i.e. • µ140±5 is high (T ) = supx∈domheight min(π140±5 (x), µhigh (x)) = 0.2 • µ140±5 is high (F ) = supx∈domheight \{⊥height } min(π140±5 (x), 1 − µhigh (x)) = 0.8 • µ140±5 is high (⊥) = min(π140±5 (⊥height ), 1 − µhigh (⊥height )) = 0 For the same reasons ‘Trump Tower’ is almost considered as being a high building with associated EPTV: • µ283 is high (T ) = supx∈domheight min(π283 (x), µhigh (x)) = 0.9 • µ283 is high (F ) = supx∈domheight \{⊥height } min(π283 (x), 1 − µhigh (x)) = 0.1 • µ283 is high (⊥) = min(π283 (⊥height ), 1 − µhigh (⊥height )) = 0 5.2 Flexible database querying In flexible database requests, the query criteria themselves may be flexible and contain labels. Consider, for example, the following query with a
EPTVs in Flexible Database Modelling and Querying
85
non-compound flexible query condition: Query 1: SELECT name, location, height FROM High buildings WHERE height IS About 300m This query selects the name, location and height of all buildings that are about 300 meters high. For the sake of the example, it is assumed that the linguistic term ‘About 300m’ stands for 300m±50m and is modelled by a fuzzy set with triangular membership function as depicted in Figure 2.
Fig. 2. The modelling of ‘About 300m’.
The tuples belonging to the result set of Query 1 are given in Table 3. This result set illustrates that EPTVs can usefully be applied to express the ‘degree’ of query satisfaction and moreover adequately can handle the occurrence of missing information in flexible database requests. Table 3. Result set of Query 1. N ame Eiffel Tower Trump Building Chrysler Building Empire State Petronas Towers Taipei 101 Central Plaza Sears Tower Great Pyramid Warsaw TV Mast CITIC Plaza
Location Paris New York New York New York Kuala Lumpur Taipei Hong Kong Chicago Gizeh Warsaw Guangzhou
Height 300.5±0.1 283 319 381 452 509 374 442 140±5 646 391
Contains {(T, 0.99), (F, 0.01)} {(T, 0.66), (F, 0.34)} {(T, 0.62), (F, 0.38)} {(F, 1)} {(F, 1)} {(F, 1)} {(F, 1)} {(F, 1)} {(F, 1)} {(F, 1)} {(F, 1)}
ra 0.99 0.66 0.62 0 0 0 0 0 0 0 0
With this example it is illustrated that EPTVs allow to model the partial satisfaction of a flexible query condition. Tuple ‘Eiffel Tower’ almost completely satisfies the query condition, whereas tuples ‘Trump Building’ and ‘Chrysler Building’ only partly satisfy the query condition. All other tuples do
86
Guy de Tr´e and Rita de Caluwe
not satisfy the condition but are shown for completeness. If required, threshold values on the associated EPTVs can be used to limit the query results. The associated EPTVs in the result set all result from the conjunction of the associated EPTV from the original relation and the EPTV that results from the evaluation of the query condition. Hereby, the conjunction operator based on the minimum t-norm operation, has been applied. For example, the associated EPTV for the tuple ‘Trump Building’ is obtained from the conjunction ˜ {(T, 0.66), (F, 0.34)} {(T, 0.9), (F, 0.1)}∧ where {(T, 0.9), (F, 0.1)} is the EPTV from the relation High buildings and {(T, 0.66), (F, 0.34)} is the EPTV that results from the evaluation of the query condition ‘height IS About 300m’, that is calculated by applying the formula for ‘IS’ predicates, i.e. • µ283 is About 300m (T ) = supx∈domheight min(π283 (x), µAbout 300m (x)) = 0.66 • µ283 is About 300m (F ) = supx∈domheight \{⊥height } min(π283 (x), 1 − µAbout 300m (x)) = 0.34 • µ283 is About 300m (⊥) = min(π283 (⊥height ), 1 − µAbout 300m (⊥height )) = 0 As extra information for the reader, the derived analytical ranking value ra is added in the last column of the result set. Usually, this derived information should not be stored in the database. Queries with compound query conditions (crisp or flexible) are handled similarly as queries with compound crisp query conditions: the EPTVs of the tuples in the result set are calculated from the EPTVs of the individual ˜ ’ and ‘∨ ˜ ’. conditions in the composition, using the logical operators ‘˜ ¬’, ‘∧ In order to illustrate the use of the third truth value ⊥, consider the following query: Query 2: SELECT name, location, storeys FROM High buildings WHERE storeys > 100 This query selects the name, location and number of storeys of all buildings that have more that 100 stories. The tuples belonging to the result set of Query 2 are given in Table 4. As with the previous example the associated EPTVs again result from the conjunction of the associated EPTV from the original relation and the EPTV that results from the evaluation of the query condition. Again, the conjunction operator based on the minimum t-norm operation, has been applied. By looking at the result set it can be observed that the framework makes a clear distinction between the situation where data exist, but the query condition is not satisfied and the situation where the query data does not apply.
EPTVs in Flexible Database Modelling and Querying
87
Table 4. Result set of Query 2. N ame Empire State Taipei 101 Sears Tower Warsaw TV Mast Great Pyramid Eiffel Tower Trump Building Chrysler Building Petronas Towers Central Plaza CITIC Plaza
Location New York Taipei Chicago Warsaw Gizeh Paris New York New York Kuala Lumpur Hong Kong Guangzhou
Storeys 102 101 110 N/A N/A 3 72 77 88 78 80
Contains {(T, 1)} {(T, 1)} {(T, 1)} {(⊥, 1)} {(F, 0.8), (⊥, 0.2)} {(F, 1)} {(F, 1)} {(F, 1)} {(F, 1)} {(F, 1)} {(F, 1)}
ra 1 1 1 0.5 0.14 0 0 0 0 0 0
For example, the facts that the ‘Great Pyramid’ and the ‘Warsaw TV Mast’ do not have ‘storeys’ are communicated to the users by means of the extra truth value ⊥, which differs from zero. The ‘Great Pyramid’ only satisfies the query condition to a minimal extent because it not fully belongs to the original relation. The associated EPTV {(F, 0.8), (⊥, 0.2)} is in fact the result of the conjunction ˜ {(⊥, 1)} {(T, 0.2), (F, 0.8)}∧ where {(T, 0.2), (F, 0.8)} is the EPTV from the relation High buildings and {(⊥, 1)} is the EPTV that results from the evaluation of the query condition ‘storeys > 100’.
6 Comparison with other approaches Classical two-valued Boolean logic does not have the semantic richness that is necessary to efficiently support flexible database modelling and flexible database querying. Extensions like three-valued Kleene logic and four-valued Belnap logic provide an extra truth value to represent unknown information and can been used to support null values. However, as described in [5], such approaches introduce problems of incomplete truth-functionality. Neither do they give satisfaction to support the requirements of more advanced techniques for the modelling and querying of imperfect information as described in the previous sections. In more advanced (fuzzy) database modelling, the most frequently used framework is based on a [0, 1]-valued possibility measure (and an associated necessity measure) [13]. Typical for the approach is that a possibility measure P os and an associated necessity measure N ec are defined over the Cartesian product domt1 × domt2 × · · · × domtm of the domains of the associated types of the characterizing attributes A1 , A2 , . . . , Am of the resulting record set R
88
Guy de Tr´e and Rita de Caluwe
of a query. As a consequence, every record r of the result set has been assigned a possibility P os({r}) ∈ [0, 1] and a necessity N ec({r}) ∈ [0, 1], which respectively indicate to what extent it is possible that r belongs to the result set and to what extent it is certain (or necessary) that r belongs to the result set. Conjunction of possibility measures and disjunction of necessity measures is obtained by P os(A ∪ B) = max(P os(A), P os(B)) and N ec(A ∩ B) = min(N ec(A), N ec(B)) Furthermore, ∀ A, P os(A) = 1 − N ec(A), where A denotes the complement of A. The main advantage of using this approach is that possibility theory provides mathematical tools and evaluation functions that allow to calculate the possibility measures and necessity measures stemming from the comparison of possibility distributions that model uncertain, imprecise or vague information [13]. The approach presented in this chapter is an extension of this framework, which additionally allows to cope explicitly with the inapplicability of information what regularly occurs in database applications. In fact, the similarity between both approaches becomes clear if the approach with EPTVs is restricted to the regular truth values T and F . If the EPTV t˜∗ (p) represents the truth of a proposition p, then the membership grade µt˜∗ (p) (T ) is the possibility that the proposition is true and can be interpreted as the possibility of the proposition, whereas the membership grade µt˜∗ (p) (F ) of F is the possibility that the proposition is false, which might be interpreted as the complement of the necessity of the proposition. Beside the main advantage of being able to cope explicitly with inapplicability of information, this new approach also provides the user with an efficient and convenient way to handle expressions. Expressions can be evaluated just by applying the extended conjunction, disjunction, negation, implication and equivalence operators as presented in [7]. EPTVs provide the user with the maximum of information available: if some parts of an expression are not applicable at evaluation, this would be clearly expressed by means of the third component ⊥. In those cases where a ranking of EPTVs is required, a ranking function as presented in Section 4 can be applied in order to obtain a single ranking value. This ranking value needs to be interpreted as extra information, associated with the EPTV, and does not replace the EPTV. The EPTV itself remains connected with the database records as illustrated in the examples of the previous section, thereby providing more information about the truth value under consideration than possibility and necessity measures do.
EPTVs in Flexible Database Modelling and Querying
89
7 Conclusions In this chapter, the usefulness of a logical framework based on EPTVs in order to support flexible database modelling and querying has been discussed. This logical framework is based on possibility theory and has specific facilities to deal with the inapplicability of information, the inapplicability of integrity constraints and the inapplicability of querying criteria. Although many-valued logics cannot be fully truth-functional, the truth-functionality of the presented framework is as complete as possible. This allows for a better support for the handling of imperfect and missing information. Moreover, because with EPTVs no specific notations are required to model unknown information, some of the problems that come along with traditional approaches, that are based on a three or four-valued logic that has a specific truth value ’unknown’, have been solved. As indicated in this chapter further research on EPTVs is necessary. Especially within the field of advanced aggregation. Future work of our research group will further explore this field. Another topic of interest concerns the further study and development of more advanced querying mechanism, including other and operators and evaluation functions. Up to now the focus of our research was on the applicability of EPTVs within object oriented database models. In the near future attention will be paid to the incorporation of the presented framework in both the regular relational database model and the possibilistic relational database models.
References [1] J. Biskup. A formal approach to null values in database relations. In H. Gallaire, J. Minker, and Nicolas J., editors, Advances in Data Base Theory, pages 299–341. Plenum Press, New York, USA, 1981. [2] E.F. Codd. Missing information (applicable and inapplicable) in relational databases. ACM SIGMOD Record, 15(4):53–78, 1986. [3] E.F. Codd. A relational model of data for large shared data banks. Communications of the ACM, 13(6), 1986. Republished in (1983) Communications of the ACM 26(1). [4] E.F. Codd. More commentary on missing information in relational databases (applicable and inapplicable information). ACM SIGMOD Record, 16(1):42–50, 1987. [5] C.J. Date. Null values in database management. In Relational Database: Selected Writings, pages 313–334. Addisson-Wesley Publishing Company, Massachusetts, USA, 1986. [6] G. de Cooman. From possibilistic information ot kleene’s strong multivalued logics. In Dubois, D. et al., editor, Fuzzy sets, logics and reasoning about knowledge, pages 315–323. Kluwer Academic Publishers, Boston, USA, 1999.
90
Guy de Tr´e and Rita de Caluwe
[7] G. de Tr´e. Extended possibilistic truth values. International Journal of Intelligent Systems, 17:427–446, 2002. [8] G. de Tr´e and B. De Baets. Aggregating constraint satisfaction degrees expressed by possibilistic truth values. IEEE Transactions on Fuzzy Systems, 11(3):361–368, 2003. [9] G. de Tr´e and R. de Caluwe. A constraint based fuzzy object oriented database model. In Z. Ma, editor, Advances in Fuzzy Object-Oriented Databases: Modeling and Applications, pages 1–45. Idea Group Publishing, Hershey, USA, 2005. [10] G. de Tr´e, R. de Caluwe, and H. Prade. Null values revisited in prospect of data integration. Lecture Notes in Computer Science, 3226:79–90, 2004. [11] G. de Tr´e, R. de Caluwe, J. Verstraete, and A. Hallez. Conjunctive aggregation of possibilistic truth values and flexible database querying. Lecture Notes in Artificial Intelligence, 2522:344–355, 2002. [12] G. de Tr´e, T. Matth´e, K. Tourn´e, and B. Callens. Ranking the possible alternatives in flexible querying: An extended possibilistic approach. Lecture Notes in Computer Science, 2869:204–211, 2003. [13] D. Dubois and H. Prade. Fuzzy Sets and Systems Theory and Applications. Academic Press, 1980. [14] D. Dubois and H. Prade. Possibility theory, probability theory and multiple-valued logics: A clarification. Annals of Mathematics and Artificial Intelligence, 32(1–4):35–66, 2001. [15] H. Prade. Possibility sets, fuzzy sets and their relation to lukasiewicz logic. In Proc. of the 12th International Symposium on Multiple-Valued Logic, pages 223–227, 1982. [16] H. Prade and C. Testemale. Generalizing database relational algebra for the treatment of incomplete or uncertain information and vague queries. Information Sciences, 34:115–143, 1984. [17] L.A. Zadeh. Fuzzy sets. Information and Control, 8:338–353, 1965.
Preference based Quality Assessment and Presentation of Query Results Stefan Fischer, Werner Kießling and Timotheus Preisinger Chair for Databases and Information Systems University of Augsburg D-86139 Augsburg {fischer, kiessling, preisinger}@informatik.uni-augsburg.de
1 Introduction As long as there have been database search engines there has been the problem of what to present to the user when there is no perfect match and how to present that query result to the user. Respecting the user’s search preferences is the suitable way to search for best matching alternatives. Modelling such preferences as strict partial orders in “A is better than B” semantics has been proven to be user intuitive in various internet applications. The better the search result, the better is the psychological advantage of the presenter. Thus, there is the necessity to know the quality of the search result with respect to the search preferences. This chapter introduces a novel personalized and situated quality assessment for query results. Based on a human comprehensible linguistic model of five quality categories a very intuitive framework for valuations is defined for numerical as well as for categorical search preferences. These quality valuations provide human comprehensible presentation arguments. Moreover, they are used to compute the situated overall quality of a search result. For delivery of the results a flexible and situated filter decides which results to present, e.g. by respecting quality requirements of the user. A so called presentation preference determines which results are predestined to be especially pointed out to a user. Eventually, it will be evaluated how ecommerce applications will profit from the use of a preference based search in combination with the introduced human comprehensible quality assessment. Considering the procurement of goods via internet the idea is simple. A customer expects to have at least the service he or she has when directly contacting a human sales person. That means the customer wants to be treated individually according to his or her needs. But the misery already begins with the first step, the usage of the search engine.
S. Fischer et al.: Preference based Quality Assessment and Presentation of Query Results, Studfuzz 203, 91–121 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
92
Stefan Fischer et al.
1.1 Database search engines – a continuing misery The lack of effectiveness of database search engines is as old as database search engines themselves. If there is no perfect match with respect to the search conditions a best alternative must be delivered. Even Amazon.com, market leader in the B2C domains of books and audio CDs, is not able to present a simple alternative to the desired book “Diary for Robin” by the author “James Patterson”, although there is a book by this author with the similar title “Diary for Nicholas” (see Fig. 1). This phenomenon is known as empty result effect. As a solution, by now most search engines are equipped with the option
Fig. 1. Amazon’s failing search engine.
to combine the search conditions with a logical or. E.g. the search engine of the company B2B-Perfect.com uses this simple technology and promises a powerful search. Especially in B2B, where the goods are much more complex than books, the effect is clear, a flooding effect with lots of irrelevant results. Fortunately, the misery can be stopped by respecting the customer’s search preferences as soft conditions. Modelling preferences as strict partial orders as “A is better than B” semantics [15] has been proven to be user intuitive in various internet applications [19]. 1.2 Quality of a query result But more than this it is necessary to provide a similarly good service as a human sales person. Since a customer wants to understand why the presented results are the best available ones for him, it is necessary to know about the quality of the query results w.r.t. the customer’s wishes. Normally, to be convinced of the presented pre-selection, as stated by the sales psychologist Becker [2], a customer wants an impression of the search quality of the presented query results as decision support. Some advanced search engines deliver ad hoc and very often not intuitively computed alternatives. The presentation of these results is often equipped with a valuation not human comprehensible that aims to tell how close the result is to the search conditions. E.g. the search engine for the documentation of the Oracle database system scores the results with values between 1% and 100% and orders the results descending.
Preference based Quality Assessment and Presentation of Query Results
93
We will shortly leave the field of querying standard databases and have a look at structured and semi-structured data. The search engine used by the portal of the scientific association ACM.org does a little better by valuating the results within five categories (see Fig. 2), which is the intuitive number of what a human being normally differentiates according to Zadeh [26], a thesis also supported by psychologist Miller’s researches on perception [20].
Fig. 2. Quality of search results at ACM.
But the user of this search engine can hardly understand why there are ratings of the second and fourth category when searching only for the word “Kießling”. A closer look to the first result shows Kießling’s works are often cited, which seems to be the reason for the applied second highest valuation. But it is not comprehensible why the second result only gets a rating of the fourth category although Kießling is coauthor of the paper and his work is also cited very often therein. The problem lies in the non-personalized and non-situated measurement of key data used in such technologies which obviously fail. Maybe one user focusses the author, another one the number of citations. Such problems have been widely discussed in the context of information retrieval. It seems to be a typical problem of this domain not directly connected to usual database searches, but it suffers of the same problems as common keyword search does in ordinary databases: On the one hand, users can hardly understand which methodology is used for result computation. On the other hand, there are barely any possibilities for a personalization of the search request – although being fundamental for satisfying search results.
94
Stefan Fischer et al.
A further and very interesting aspect is that in a sales scenario of course the better the search result, the better is the psychological advantage of the vendor. Therefore, an internet store should have the information about the quality of the search results for being able to provide a good reasoning when offering the results. All the well known sales psychology models (e. g. [21; 9; 11]) emphasize that knowledge about the quality of the offered goods with respect to the customer’s preferences is a major factor for a sales dialog and for consumer choice behavior. The preferences of each of the customers of course differ. They are even different for one customer in various situations, e.g. someone may in general prefer the color green, but not when considering the color of a car. These personality and situation dependant preferences are also known as conditional preferences in [4], although the approach used there rather focuses on situations. Of course, in a sales scenario the question is not only how to valuate a query result, but also in which order the results should be presented. Is there a result to pro-actively point out? Is there a possibility to influence the consumer choice behavior? For these questions this chapter is organized as follows. Next, the foundations of the underlying intuitive preference model will be summarized. In section 3 a novel approach for the quality assessment of query results will be introduced, including psychological aspects of query result presentation regarding consumer choice behavior. Integrated into a prototype experiences with human customers will demonstrate the valuable effects in section 4. Summary and outlook will conclude this chapter.
2 Fundamental preference concepts revisited To handle user preferences, a comprehensive and theoretical well-founded framework is necessary for a flexible and intuitive usage. Therefore, the foundations of Kießling [15; 16] for preferences in databases are briefly repeated. They are the basis for an intuitive and flexible valuation of search results. Afterwards we will have a look at some other preference models. 2.1 Preference modelling – foundations People naturally express their wishes in terms like “I like A better than B”. This kind of preference modelling is universally applied in any kind of situation and understood by everybody. Even just thinking having tea or coffee for breakfast shows such a preference. People are intuitively used to deal with such preferences. As shown and designed in [15; 16], all these preferences can be formulated as strict partial orders and even be engineered to complex, multidimensional preference constructs without loss of intuitive semantics.
Preference based Quality Assessment and Presentation of Query Results
95
Preferences and their engineering In [15; 16] a preference is formulated on a set of attribute names with an associated domain of values. When combining preferences P1 and P2 , the attributes of P1 and P2 may overlap – allowing multiple and even conflicting preferences on the same attributes. Definition 1. Domain values of a set of attributes Let A = A1 , A2 , . . . , Ak denote a non-empty set of attribute names Ai associated with domains of values dom(Ai ), 1 ≤ i ≤ k. The domain of A is defined as dom(A) := ×Ai ∈A dom(Ai ). Definition 2. A preference as strict partial order Given a set A of attribute names, a preference P is a strict partial order P := (A,
96
Stefan Fischer et al.
will see some of the various intuitive preference constructs introduced in [15]. The following preference constructs enable one to easily model the most frequent expressions. Definition 4. Categorical or non-numerical base preferences Given a finite POS-set S ⊆ dom(A) of preferred values, P := POS(A, S) is called a POS preference if ∀x, y ∈ dom(A) : x
Preference based Quality Assessment and Presentation of Query Results
97
Given an interval [low, up] ∈ dom(A) × dom(A), the distance is defined as ∀v ∈ dom(A) : dist(v, [low, up]) := if v ∈ [low, up] then 0 else if v < low then low − v else v − up. P := BETWEEN(A, [low, up]) is a BETWEEN preference if ∀x, y ∈ dom(A) : x dist(y, [low, up]). The user specifies an interval with his preferred value. P := AROUND(A, z) is called AROUND preference if P := BET W EEN (A, [z, z]) The user specifies a single preferred value. Given a function f : dom(A) → R and the ‘less-than’ order ‘<’ on R. P := SCORE(A, f ) is called SCORE preference ∀x, y ∈ dom(A) : x
2 are obvious. Definition 6. Pareto preference Given two preferences P1 , P2 on two domains A1 , A2 and two tuples (x1 , x2 ), (y1 , y2 ) ∈ dom(A1 ) × dom(A2 ). A preference P := P1 ⊗ P2 is called a Pareto preference if (x1 , x2 )
98
Stefan Fischer et al.
maximum temperature during winter days should be between 10 and 16 degrees Celsius. These three preferences can be expressed as P := P1 ⊗P2 ⊗P3 , where P1 := AROU N D(population, 500000), P2 := P OS(state capital, yes), P3 := BET W EEN (degreesCelsius, [10, 16]). A graph with edges xi → xj for every two xi , xj with xi
Las Vegas (500000, no, 16)
Phoenix (1400000, yes, 21)
San Diego (1300000, no, 18)
Fig. 3. Graph for Lisa’s Pareto preference.
If a preference is more important than another then the following prioritized preference supports the intuitive modeling. Definition 7. Prioritized preference Given two preferences P1 , P2 on two domains A1 , A2 and two tuples (x1 , x2 ), (y1 , y2 ) ∈ dom(A1 ) × dom(A2 ). A preference P := P1 & P2 is called a prioritized preference if (x1 , x2 )
& P2
(y1 , y2 ) ⇔ x1
In a prioritization, a tuple y is better than a tuple x if it is better in the more important contained preference. Given an equal rating there, y is rated better than x if it is better in the less important preference. If they are rated uncomparable in the more important contained preference, the overall rating is as well uncomparable. 2.2 The BMO query model Preferences are defined in terms of values from dom(A), representing the realm of wishes. In database applications it is assumed that the real world is mapped into appropriate instances that are called database sets. A database set R may e.g. be a view or a base relation in SQL or a DTD-instance in XML. Whether preferences can be satisfied depends on the current database contents. Thus a match-making between wishes and data has to be made. To this purpose the “Best Matches Only” (BMO) query model has been introduced in [15; 16].
Preference based Quality Assessment and Presentation of Query Results
99
Definition 8. BMO result The Best Matches Only result set contains only the best matches w.r.t. the strict partial order of a preference P . It is a selection of unordered result tuples. All tuples t, t ∈ BM O have equal or uncomparable values regarding the preference P . For prioritized preferences this BMO model can lead to a possible disregard of less important preferences. When comparing tuples, a specific preference is only evaluated if all the more important preferences show equal results. In difference to number based preference models we do not imply any kind of order to the elements of the BMO set. For no two of them we can say which one is better according to the user’s preferences at this point. This will be the subject of section 3. Chomicki’s winnow operator [5] provides similar semantics. Currently, there are two query languages extended by soft conditions based on the presented preference model, namely Preference SQL [19] for relational databases and Preference XPath[18] for XML structured data. 2.3 Other preference models Basically, there are two different ways of expressing preferences in databases, quantitative and qualitative. The quantitative approach uses scoring functions for computing a numeric score for every tuple. The results are returned in order of this score. Agrawal and Wimmers present numerical value based preferences in [1]. A preference is modelled as a floating point value between 0 and 1. Although this approach enables a user to express preferences at a very fine grained level, it is not easy understandable for many people. People are not used to rating anything on a floating point number base [20]. The combination of different preferences they propose is a kind of prioritization between these. Using a fuzzy set approach, Bosc and Prade not only use soft query conditions but also propose a way of storing imprecise data in [3]. Our focus however lies on product catalog search and recommendations. As product descriptions in catalogs do not contain uncertain data, we do not have to deal with this issue. The soft conditions in [3] are also based on floating point values between 0 and 1. Dubois and Prade use this fuzzy preference model as a base for their research in [7]. For the connection of equally important preferences they propose different ways of computing including use of minimum and maximum functions and averaging. Although not explicitly stated, the selection of the computation type should be made following personalized and situated principles. Using five discrete levels of fulfilling search conditions it combines result selection and a kind of quality assignment. Qualitative preference models like ours use directly specified preferences between tuples. Users are enabled to express their preferences declaratively.
100
Stefan Fischer et al.
This is easy to accomplish especially for unexperienced users and therefore provides a very intuitive interface.
3 Personalized quality assessment and presentation of query results Operating the search engine of a database is a very powerful position and decisive question of design [6]. E.g., considering the power of the decision which results to present first at Google.com’s search engine, the first listed links have much more visits, with all the positive consequences of this. When setting up the search engine in a database, there are five steps: 1. 2. 3. 4. 5.
Preparing the sources Indexing Interpreting the query Searching Presentation of the results
Steps 1 to 4 are part of the database management system. To improve the search quality by avoiding the empty result effect and the flooding effect, a preference search technology (see subsection 2.2) should be used for step 3. The importance of step 5 is still nearly ignored. Especially in a sales scenario, it is very important which results and alternatives to present at all, which ones to put in front, or even to recommend. Here also the question is whether to respect presenter preferences. There are attempts to cover these problems, e.g. the provider of e-catalog technology Poet.com allows specifying “favorites” by flags in the database. Whenever favorites are in the result set they are placed at the top of the result table. But that respects the presenter preferences only statically. Furthermore, there is no respect to any customer preferences or to aspects of specific situations. One major and still unsolved key issue in these difficulties is the quality of the search result with respect to the customer preferences. The knowledge about the qualities of search results is absolutely necessary to do a good reasoning when presenting results. In this section a novel personalized and situated presentation framework is introduced. A valuation of query results is, of course, necessary only for advanced search engines that are able to present alternatives when there are no perfect matches. In the sequel, the BMO result model as described in Definition 8 is applied, because it is very user intuitive and has been shown to be suitable for a cooperative product search in various applications [19]. 3.1 Design principles and workflow For a successful deal, the product presentation is a decisive factor. An example of an advanced search result presentation is shown in the following scenario.
Preference based Quality Assessment and Presentation of Query Results
101
Example 3. Sample scenario of advanced search result presentation Marge is a computer dealer and tells her vendor Apu: “I am interested in notebooks. The clock frequency must be at least 2 GHz. The order quantity should be around 40. It is equally important that the main memory capacity should be between 512 and 1024 MB-RAM, and it is equally important that the price should be between 100 and 1200 Euro. Equally important, my preferred manufacturers are Toshiba and Hewlett Packard.” An answer from the vendor side might be the result set shown in Table 1. Table 1. Sample BMO result set t1 t2 t3 t4
Make CPU-GHz MB-RAM Quantity Price per unit (Euro) Elitegroup 2.0 256 40 1150 Gerion 2.0 374 50 1199 HP 2.2 512 50 1249 Toshiba 2.4 768 40 1378
Even in this not too complex example the customer would be glad about advice and assistance. Additionally, a proactive recommendation of a product supports an effective selling process [2]. A cooperative result presentation could now pick out one result. This could be the Toshiba notebooks, because the vendor’s utility function of maximizing the turnover is supported and, besides the hard condition of the CPU-GHz, three of four preferences are perfectly fulfilled. This delivers the following sales arguments: “There are four best matches with respect to your preferences. I recommend the Toshiba notebook position which very well meets your preferences. It perfectly matches your desired manufacturer and order quantity and has even a faster CPU and perfectly the MB-RAM as you demanded. I think the moderately higher price is acceptable for this high-quality product.” The first aspect of the following design principles serves to avoid incomprehensible and non-personalized ad hoc attempts of presenting search results with incorrect valuations and disrespect of the preferences of customer and presenter, i.e. the vendor side. Moreover, the second aspect serves to enable advanced search engines to do a cooperative and effective presentation of the search results following the strategy of the presenter: 1. Consideration of user preferences as well as presenter preferences From the search to the final presentation the preferences of the user as well as the preferences of the presenter must be flexibly combinable with respect to the specific situation. 2. Providing plausible arguments about the quality of the search results For an effective presentation of search results the vendor needs arguments to underpin his decision why to present or point out the specific results. Therefore, it is necessary to know about the search quality with respect
102
Stefan Fischer et al.
to single search conditions as well as about an overall impression of the quality of a search result for the specific situation. 3. Consideration of filter criteria applied to the search result set In various cases not all computed alternatives should be shown to the user. The reason may be e.g. search quality requirements of the user or results the presenter only wants to show under specific circumstances and different situations. Therefore it must be possible to apply flexible filter conditions to the search result set. 4. Semantically well-founded marked favorites within the result set To enable the vendor to point out one or more of the results, these have to be marked following flexible and situated criteria such as e.g. a utility function of a vendor or the result quality with respect to a customer’s search preferences. Definition 9 describes the major steps of the workflow of a personalized advanced query result presentation based on the BMO query model as described above, respecting the just mentioned design principles. Definition 9. Workflow of a result presentation The presentation of BMO query results is defined in seven steps: 1. Query composition with search preferences 2. Preference search 3. Computing the quality valuations of the BMO results 4. Computing aggregated quality valuations 5. Applying “But-only” filter conditions to the BMO results 6. Marking out favorites via presentation preferences 7. Result presentation and consideration of customer feedback In the sequel the steps of this Definition are outlined. 1. Query composition with search preferences: Search preferences must be known before starting a qualitative product search. While customers may express theirs in a search mask, vendor preferences can be regarded as a-priori filters. Both types can be intuitively expressed in our preference model described in Definitions 2 and 3. 2. Preference search: With a given search preference best matches can be queried from a database using a preference query language (see section 2.2). The results are only best matches according to the BMO model of Definition 8. This provides a clearly arranged pre-selection without bothering the customer by lots of search iterations with adjustments of the search criteria. 3./4. Computing the quality valuations of the BMO results and aggregated quality valuations: Knowing the quality of a search result with respect to each occurring base preference provides valuable presentation arguments. People normally do not think in numbers when rating a quality. As Zadeh found out [26] they think in so called linguistic variables. We use a system of five quality levels
Preference based Quality Assessment and Presentation of Query Results
103
each mapped to a linguistic variable. Of course our approach of quality assessment would work as well with more or less levels. Thereby we get a basis for an objective quality computing with respect to complex preferences and therefore to an overall quality computation for every search result. The following Definition 10 specifies the quality function of search results for the underlying situation. Definition 10. Quality function QUAL Let V := (‘perfect’, ‘very good’, ‘good’, ‘acceptable’, ‘sufficient’) be a descending ordered list of 5 linguistic quality terms and let C(s) := {C1 (s), . . . , C5 (s)} be a partition of dom(A) into 5 parts depending on the situation s. Then the quality function QU ALP,s : BM O → V for a tuple t ∈ BM O with t[A] ∈ dom(A) regarding a preference P := (A,
Via online request tuple t3 is offered to the customer with a quality of q1 . After thinking about the offer he decides to order the notebook but in this moment the notebook t3 is sold out. The vendor offers t2 as a further alternative and argues with quality q2 for this result, where q1 < q2 . The
104
Stefan Fischer et al.
customer would not understand why the obviously better result regarding his preferences would be valuated worse than the worse alternative. For him the situation has not changed. Therefore, QU ALP,s must fulfill the following postulate of Definition 11. Definition 11. Quality postulate For a given preference P := (A,
Preference based Quality Assessment and Presentation of Query Results
105
Definition 13. “But-only” filter applied to query results Let BOF specify a hard filter condition over BMO++ . Then the result set of the selection on the extended pre-selection BMO++ with application of the “but-only” filter BOF is declared as BM O∗ := σBOF (BM O++ ), where σcondition (R) denotes the well-known hard selection over the relation R. A condition can be stated in terms of quality, e.g. results which only have a very low overall quality should be disregarded. Another condition might concern the vendor’s utility function, e.g. only show the five results with the highest profit. Of course, any combinations are possible as well. 6. Marking favorites via presentation preferences: BMO∗ will be presented to the user, but for a promising presentation, the system should proactively point out a result and give reasons why this is an appropriate result. In a sales scenario, a vendor would act smart and according to the rules of sales psychology, personalized to each customer. There is a wide range of rules how to decide the presentation order. Thus, presentation preferences PP need to be flexible to enable the use of different sales strategies as well as vendor preferences, e.g. the maximization of profit. The selection for determination of the favorites with the presentation preference PP can be declared as follows. Definition 14. Presentation preference over BMO∗ Let PP be the presentation preference over the enhanced result set BMO∗ . Then FAV := [PP](BMO∗ ) denotes the result set of favorite query results according to the presentation preference PP. If FAV consists of more than one result then a random pick out of this result set is a proper choice to determine which result to present in the first row. Yet, the empty result effect never occurs, since the selection of favorites is defined as preference. 7. Result presentation and consideration of customer feedback: Finally, if there is no agreement for a result there might be several reasons. On the one side, if the number of results is still too large, then a filtering within the results should be done according to the customer’s feedback. The results of this selection are computed with the customer filter criterion CFC by σCF C (BM O∗ ). Because of the very good filter effect of the BMO query model (see [17]) this will not often be necessary. On the other side, if the customer is not satisfied with this selection then according to his feedback a further preference query should be started or results hidden due to step 5 could be presented. If the customer has correctly expressed his search preferences this will not often be necessary, because he can be sure that all (but the possibly hidden) relevant results have been presented. Considering these 7 steps of the workflow of Definition 9 step 1 and 2 were discussed in chapter 2 in detail. Step 7 depends on the application and shall not be the focus here. For step 3, the next three subsections introduce an
106
Stefan Fischer et al.
intuitive and situative model for quality valuation of query results regarding preferences. Step 4 can easily be managed by applying the aggregate functions of the used query language and therefore needs no further consideration. In subsections 3.4 and 3.5 approaches for “but-only” filters and presentation preferences for a personalized and smart presentation dialog are introduced. 3.2 Quality information for base preferences In contrast to ad hoc valuations and non-intuitive approaches in the sequel a universal and semantics based approach is developed. The preference model of [15; 16] provides semantic information according to various intuitive constructs to formulate a human’s wish. Based on this framework, the quality of a search result is determined under consideration of this semantics. Thus, the obtained qualities are available with an according context and so they provide comprehensible, suitable arguments for the presentation of search results. Categorizing the qualities of a result tuple within its queried base preferences is a very sensitive design decision and depends on many factors of the respective situation. The quality valuation function QUALP,s of Definition 10 will be specified in a well-formed way for each mentioned base preference. Because of the sensitivity of the quality valuations to the current situation, partitioning parameters bi (s) ∈ R are defined, depending on the situations. Please note that consequently QUALP,s is instantiated for a BMO result set as defined in Definition 10. In [10] it is shown that the following valuation approach is not restricted to the intuitive BMO model. SCORE preference Definitely a perfect match for a preference P := SCORE(A, f) is a match where the scoring function is maximal. But often the scoring function f has no upper limit in a specific domain. Thus for a quality valuation additional knowledge about the situation must be regarded. Depending on the specific situation the knowledge engineer has to formulate QUALP,s as declared in Definition 15. Definition 15. QUALP,s -function of a SCORE preference For a given preference P := SCORE(A, f ) in a situation s and a result tuple t ∈ BM O, QUALP,s is defined as follows, where b1 (s) ≥ b2 (s) ≥ b3 (s) ≥ b4 (s): ‘perfect’, b1 (s) ≤ f (t[A]) b2 (s) ≤ f (t[A]) < b1 (s) ‘very good’, b3 (s) ≤ f (t[A]) < b2 (s) QU ALP,s (t) := ‘good’, ‘acceptable’ b 4 (s) ≤ f (t[A]) < b3 (s) ‘sufficient’, f (t[A]) < b4 (s)
Preference based Quality Assessment and Presentation of Query Results
107
BETWEEN preference Regarding a preference P := BETWEEN(A, [low, up]), a search result t is marked as a perfect match if and only if t[A] ∈ [low, up]. For the quality classifications except ‘perfect’ the knowledge engineer has to declare the suitable ranges, under involvement of the respective situation. There, a result with a distance d > 0 from the upper bound of the optimal range intuitively must be of the same quality like a result with distance d from the lower bound. Therefore, the following symmetric partition is defined. Definition 16. QUALP,s -function of a BETWEEN preference For a given preference P := BETWEEN(A, [l, u]) in a situation s for a result tuple t ∈ BM O, QUALP,s is of the following form, where 0 ≤ b1 (s) ≤ b2 (s) ≤ b3 (s) and the function dist(v, [low, up]) is known from Definition 5: ‘perfect’, dist(t[A], [l, u]) = 0 ‘very good’, 0 < dist(t[A], [l, u]) ≤ b1 (s) b1 (s) < dist(t[A], [l, u]) ≤ b2 (s) QU ALP,s (t) := ‘good’, ‘acceptable’, b 2 (s) < dist(t[A], [l, u]) ≤ b3 (s) ‘sufficient’, b3 (s) < dist(t[A], [l, u]) AROUND preference The AROUND preference is a special case of the BETWEEN preference. It’s QUALP,s -function can be described equally to the one of the BETWEEN preference when setting up = low =: z. It is clear that a result tuple t is denoted as perfect match iff it exactly has the value z. HIGHEST preference Just like for the SCORE preference, in a HIGHEST preference the biggest values are best. And as well there need not be a upper limit for values. So the knowledge engineer has to decide which quality valuation a search result deserves. With knowledge about the situation he can design the following well-formed QUALP,s -function. Definition 17. QUALP,s -function of a HIGHEST preference For a given preference P := HIGHEST(A) in a situation s for a result tuple t ∈ BM O, QUALP,s is of the following form, where b1 (s) ≥ b2 (s) ≥ b3 (s) ≥ b4 (s): ‘perfect’, b1 (s) ≤ t[A] (s) ≤ t[A] < b1 (s) ‘very good’, b 2 b3 (s) ≤ t[A] < b2 (s) QU ALP,s (t) := ‘good’, ‘acceptable’ b4 (s) ≤ t[A] < b3 (s) ‘sufficient’, t[A] < b4 (s)
108
Stefan Fischer et al.
The LOWEST preference, being dual to the HIGHEST preference has a nearly identical definition for the QUALP,s -function. The function for LOWEST preferences is defined by replacing ‘<’ by ‘>’ and analogous ‘≤’ by ‘≥’ in Definition 17, so that smaller values are rated better. POS/NEG preference A result tuple t is a perfect match referring to a preference P := POS/NEG(A, POS-set; NEG-set) iff t[A] ∈ POS-set and it must therefore be marked as being of ‘perfect’ quality. Accordingly if t[A] ∈ NEG-set, then this result t has to be declared of ‘sufficient’ quality. The knowledge engineer’s only decision is how to valuate results t[A] ∈ / POS-set ∪ NEG-set. Once again, there is no wisest selection. The method to be used depends on user and situation. Definition 18. QUALP,s -function of a POS/NEG preference For a given preference P := POS/NEG(A, POS-set; NEG-set) in a situation s for a result tuple t ∈ BM O, QU ALP,s is of the following form: t[A] ∈POS-set ‘perfect’, M ‘very good’, N QU ALP,s (t) := ‘good’, ‘acceptable’ O ‘sufficient’, t[A] ∈NEG-set There are three possible approaches: 1. Optimistic: M := t[A] ∈ / POS-set ∪ NEG-set, N := ∅, O := ∅. 2. Moderate: N := t[A] ∈ / POS-set ∪ NEG-set, M := ∅, O := ∅. 3. Pessimistic: O := t[A] ∈ / POS-set ∪ NEG-set, M := ∅, N := ∅. Example 4. Sample QUALP,s -function for a POS/NEG preference In Example 1 we learned that Marge had a POS/NEG preference for cities with Minnesota as the only element of the POS-set and a NEG-set containing Los Angeles and Orlando. Assuming an optimistic quality valuation, we get the quality values stated in Table 3. Table 3. Sample result quality for a POS/NEG preference City Quality rating Minnesota perfect New York very good Los Angeles sufficient
It depends on the situation which approach a knowledge engineer selects. Sales psychology tells a more pessimistic valuation may be more believable for some customers. For other customers a more offensive presentation with an optimistic valuation leads to a successful deal.
Preference based Quality Assessment and Presentation of Query Results
109
POS/POS preference / P OS1 A tuple t with t[A] ∈ P OS1 -set is a perfect match. If t[A] ∈ set ∪ P OS2 -set then this t[A] is not mentioned by the customer and therefore the quality valuation must be ‘sufficient’. Only for values t with t[A] ∈ P OS2 set the knowledge engineer has to decide which quality valuation to use. Definition 19. QUALP,s -function of a POS/POS preference For a given preference P := P OS/P OS(A, P OS1 -set; P OS2 -set) in a situation s for a result tuple t ∈ BM O, QUALP,s is of the following form: ‘perfect’, t[A] ∈ P OS1 -set M ‘very good’, N QU ALP,s (t) := ‘good’, ‘acceptable’ O ‘sufficient’, t[A] ∈ / P OS1 -set ∪ P OS2 -set There are three possible approaches: 1. Optimistic: M := t[A] ∈ P OS2 -set, N := ∅, O := ∅. 2. Moderate: N := t[A] ∈ P OS2 -set, M := ∅, O := ∅. 3. Pessimistic: O := t[A] ∈ P OS2 -set, M := ∅, N := ∅. NEG preference The quality valuation for a result t ∈ BM O with respect to a NEG preference P := NEG(A, NEG-set) is intuitively clear and leaves no scope for a knowledge engineer. A result t has to be declared of ‘perfect’ quality when t[A] ∈ / NEGset, because that is exactly what the user desires. If t[A] ∈ NEG-set this result must be stated as ‘sufficient’, according to the preference model philosophy. Definition 20. QUALP,s -function of a NEG preference For a given preference P := NEG(A, NEG-set) in a situation s for a result tuple t ∈ BM O, QUALP,s is of the following form: ‘perfect’, t[A] ∈NEG-set / QU ALP,s (t) := ‘sufficient’, t[A] ∈NEG-set POS preference The dual of the NEG preference is the POS preference. Thus, the design modelling is intuitively the dual design of the NEG preference: Elements of the POS-set are rated ‘perfect’, all others ‘sufficient’. Definition 21. QUALP,s -function of a POS preference For a given preference P := POS(A, POS-set) in a situation s for a result tuple t ∈ BM O, QUALP,s is of the following form: ‘perfect’, t[A] ∈POS-set QU ALP,s (t) := ‘sufficient’, t[A] ∈POS-set /
110
Stefan Fischer et al.
Lemma 1. For the QUALP,s -functions of all mentioned base preferences it holds that a) ∀ t, t | t[A], t [A] ∈ dom(A) : t
Preference based Quality Assessment and Presentation of Query Results
111
P2 := P3 ⊗ P4 is considered to be in the form P := P1 ⊗ P3 ⊗ P4 . This reflects that each of the equally important preferences P1 , P3 , and P4 must have the same impact for the quality valuation. Obviously, in a numerical average valuation the first consideration would lead to a wrong quality. Definition 22. Optimistic valuation of a Pareto preference result Assume a Pareto preference P := P1 . . . Pd , d ≥ 2, in a situation s where for all Pj , j = 1, . . . , d, it holds that Pj is no Pareto preference and QUALPj ,s (t) fulfills the postulate of Definition 11. The optimistic valuation for a result tuple t ∈ BM O is defined as follows regarding the order of quality terms of Definition 10: QU ALP,s (t) := max{{QU ALPj ,s (t)|j = 1, . . . , d}}, where {{. . . }} denotes a multi-set. Thus, for the optimistic valuation the best quality valuation of all Pareto combined preferences Pj within P is decisive. The contrary approach, the pessimistic valuation defines the quality of a result tuple t as worst quality valuation of all preferences Pj in P . Example 5. Sample optimistic and pessimistic quality valuation Marge and Homer are interested in a bundle of notebooks and consider seven equally important characteristics. Formulated in preference algebra they have the Pareto preference P := Pmake ⊗ Pmhz ⊗ Pram ⊗ Poquan ⊗ Pdeltime ⊗ Pweight ⊗ Pprice . For these seven base preferences Marge and Homer have the same situational parameters and the same opinions, whether these characteristics are suitably fulfilled or not. In Fig. 4 the preference quality information for each Pj is given for one result tuple t regarding to the notebook characteristics. In this, five stars for one Pj denote a ‘perfect’ quality, four stars denote a ‘very good’ quality, and so on until one star denotes a ‘sufficient’ quality for the QU ALPj ,s -function. Thus, e.g. QU ALPmake ,s (t) = ‘very good’. Homer is the carefree customer and only considers the positive aspects of a search result. Therefore, for Homer the optimistic quality valuation would be appropriate. With the valuation of Definition 22 the quality for P is computed as follows: QU ALP,s (t) = max{{ ‘very good’, ‘acceptable’, ‘very good’, ‘very good’, ‘very good’, ‘acceptable’, ‘good’}} = ‘very good’ Marge, in contrast to Homer, is always very skeptical. For her, the pessimistic quality valuation might be suitable. Analog to Definition 22 it is calculated as follows: QU ALP,s (t) = min{{ ‘very good’, ‘acceptable’, ‘very good’, ‘very good’, ‘very good’, ‘acceptable’, ‘good’}} = ‘acceptable’
112
Stefan Fischer et al.
Fig. 4. Sample qualities of notebook characteristics for one result tuplet.
The qualities for these two examples are visualized in Fig. 4. Besides these two valuations often a more moderate approach is appropriate in various situations. Therefore, a statistically more robust techniques is introduced. Even more ways of computation can be found in [10]. Definition 23. Equidistant linguistic average valuation of a Pareto preference result Assume a Pareto preference P := P1 ⊗ . . . ⊗ Pd , d ≥ 2, in a situation s where for all Pj , j = 1, . . . , d, it holds that Pj is no Pareto preference and QU ALPj ,s (t) fulfills the postulate of Definition 11. The equidistant linguistic average valuation for a result tuple t is defined as follows regarding the order of quality terms of Definition 10: ‘perfect’, y=5 y=4 ‘very good’, y=3 QU ALP,s (t) := ‘good’, ‘acceptable’ y =2 ‘sufficient’, y=1 where y := round 5, 4, where g(x) := 3, 2, 1,
d 1 d
g(QU AL (t)) P ,s j j=1
x = ‘perfect’ x = ‘very good’ x = ‘good’ x = ‘acceptable’ x = ‘sufficient’
Quality valuation of a prioritized preference result According to the philosophy of a prioritized preference P1 & . . . & Pd the preference P1 is the dominating preference and thus must have the decisive impact. As P2 , . . . , Pd are subordinated, P1 as the only decisive factor leads to the following valuation.
Preference based Quality Assessment and Presentation of Query Results
113
Definition 24. Quality valuation of a prioritized preference result Assume a prioritized preference P1 & . . . & Pd , d ≥ 2, in a situation s where for all Pj , j = 1, . . . , d, it holds that Pj is no prioritized preference and QU ALPj ,s (t) fulfills the postulate of Definition 11. The valuation for a result tuple t is defined as follows: QU ALP,s (t) := QU ALP1 ,s (t) There is no further correct valuation of a prioritized preference result [10]. Please note that the information of QU ALP2 ,s (t), . . ., QU ALPd ,s (t) of a prioritized preference is not necessary for the quality of the prioritized preference, but provides valuable arguments for a result presentation. In the last two sections the quality valuation functions were defined for each base and complex preference in the form QU ALP,s (t). It is obvious that the structure of the complex preferences can be used to calculate recursively the quality of each combined preference. Thus, all presentation arguments regarding the quality can be computed. With this the complex attribute Q of BM O+ of Definition 12 can be calculated. 3.4 Filter criterion “but-only” The quality valuations with respect to the base preferences as well as for the complex preferences have been constructed. Based on them, the decision which results to present can be made now. These results will be visible for the user while the others will be hidden and never or in a second phase be presented to the user. There are various reasons for filter criteria BOF (”but-only” filter) to compute BM O∗ := σBOF (BM O++ ) (see Definition 14). Three obvious criteria are hidden preferences of the knowledge engineer, issues of presentation style, and quality claims of a user, but of course the framework is not limited to these. E.g. in a web shop several vendor preferences could be integrated in a search preference P. Many criteria can only be handled after the soft selection, for instance a vendor wants to present only one result per manufacturer, but with highest possible quality. Lots of other and combined approaches are imaginable. 3.5 Selection criterion for pointing out a search result Suppose the decision has been made which results to present to the customer. Still in many scenarios it is necessary to point out one or more results. The question which results are the best ones to present first is very situation and domain dependent. Agrawal and Wimmer [1] suggest bringing the results into a total order by rating the results with a sum over weighted preferences. But there is only a limited expressiveness in their search technology/preference
114
Stefan Fischer et al.
model and also limited semantics for a smart presentation order. As Pareto preferences and prioritized preferences cannot in general be transformed into total orders [16; 5], several different presentation strategies will be pointed out. It becomes clear that the preference model of [15; 16] and the introduced quality valuation of search results form a very powerful framework for an easy and intuitive declaration of presentation preferences. By avoiding the empty result effect when using this model for presentation we can also make sure that there will always be at least one result predestined to point out. General selection criteria One straight forward approach is to pick out a result with the best available overall quality, namely the highest AQoverall . Assume a result set BM O∗ . The presentation preference regarding the highest overall quality within BM O∗ is defined as PP := HIGHEST(AQoverall ). If this presentation preference PP delivers more than one result then a random pick out of these is an appropriate way. To get many presentation arguments for a good reasoning, it is very useful to pick out the results with the highest number of high quality single characteristics, i.e. with the most perfectly fulfilled base preferences. This leads to the next selection criterion. Once again assume a result set BM O∗ with the aggregated attributes AQ‘perfect’ , AQ‘very good’ , AQ‘good’ , and AQ‘acceptable’ , counting the quality valuations with respect to the base preferences for a result t. Then the selection criterion for the highest quantity of most positive result characteristics is defined as the following complex presentation preference: PP := HIGHEST(AQ‘perfect’ ) & HIGHEST(AQ‘very good’ ) & HIGHEST(AQ‘good’ ) & HIGHEST(AQ‘acceptable’ ) This selection criterion first picks out results with most ‘perfectly’ satisfied base preferences. If there is an equal number of ‘perfect’ statements then within these results the results with the most ‘very good’ valuations are delivered and so forth. Of course, there are many further intuitive criteria. In the next section the focus is on criteria wrt. a sales scenario. Presentation criteria in sales scenarios As a special domain, selection criteria for a presentation within a sales scenario, are discussed in this section. Especially in a sales dialog it is very important to choose the adequate products to point out, i.e. to follow a special sales strategy [22; 12]. The style of a good vendor is to actively recommend one or more results and to argue why this is an appropriate product. The vendor can improve his believability when he is able to adequately valuate the alternatives [2]. Of course, the better the quality of a search result the higher is the psychological advantage for the vendor within a result presentation.
Preference based Quality Assessment and Presentation of Query Results
115
There is no single perfect sale strategy. Many cognitive and environmental factors have an impact on the success of a sales scenario [2]. Some people can be convinced by a more blatant offering. Others react positively to an offer, when positive and negative aspects of the search result are discussed. Also vendors have different utility functions. There can even be conflicting aspects, e.g. when there is a result in the result set with lots of high-quality presentation arguments and there is a further result with a higher profit for the vendor. Each selection criterion is declaratively defined as a preference according to the model of [15; 16]. As shown these preferences can flexibly be combined into more complex preferences, even if there are conflicts in the preferences. With the following criteria a framework is given to support wellknown aspects of sales strategies concerning the selection of one item out of the result set being actively recommended. This describes a prime example for the usage of soft selections. In literature about consumer choice behavior all models treat the quality of the offered goods wrt. the customer’s preferences as a major cognitive factor [21; 9; 11]. Thus, the rules discussed in the last subsection that focus on good arguments regarding the preferences are also highly relevant for a sales strategy driven query result presentation. But it is not always suitable to present the best alternatives at first. Hansen [11] emphasizes that it depends on the situation, i.e. on the person involved, the location, and many other factors. He also points out that it is very important for a customer to consider alternatives. It depends on the customer whether he wants to the best available alternatives presented to him or to discover them by himself. Possibly, the selection criterion PP := AROUND(AQoverall ) might be useful, e.g. in order to start with an alternative of middle quality. A further very important selection criterion according to the sales psychologist B¨ansch [2] is the price. He argues that on the one side recommending a result with a high price shows the customer a high appreciation of his financial strength. But on the other side, this is embarrassing if the customer actually does not have this financial strength. Thus, the rule is when knowing about the highest price preference of a customer, then the result with the highest price and hence the semantically supposed item of highest product quality should be presented. If there is no knowledge about such a financial strength or the will to buy the most expensive alternative, then an alternative of a middle price up to a high price or even with the second highest price should be recommended. This shows appreciation of the customer, but it is not embarrassing for him when he cannot afford this product. And, in case of financial strength there is even the possibility for the customer to step up to a product of higher quality and price, which boosts his ego. Of course, all the discussed presentation preferences can be combined with vendor preferences like a maximal turnover, a maximal profit, or promotion of slow sellers. Moreover, every imaginable result set could be used as input within this framework. This framework is universal, because it satisfies the
116
Stefan Fischer et al.
postulate of Definition 11 and the valuation itself is based on the flexible, intuitive and semantics based preference model of [15; 16].
4 Evaluation by means of an automated sales agent The quality assessment framework including lots of presentation preferences was realized as a Java component and evaluated within the automated sales agent COSIMAB2B . 4.1 The prototype COSIMAB2B COSIMAB2B is the prototype of an autonomous sales agent that automates a cost intensive e-procurement process. Together with industrial partners a typical B2B use case was modelled. The product domain comprises boxes, in particular storage, transport, and waste containers. On the client side the customer interface is additionally equipped with some optional features. A female character agent named COSIMA embodies the electronic sales agent. COSIMA talks to the customer via speech synthesis in real time. Text templates are used for the text generation to realize speech output e.g. for different sales strategies and to dynamically integrate the discussed quality knowledge about the search results. Upon the start of COSIMAB2B the embodied character agent welcomes the customer. Then the customer iteratively composes the content of his shopping cart by searching the product database. In Figure 5 for example the customer is searching for a red storage container made of polyethylene with a volume of about three liters and a width of 100 till 150 millimeters. Actually, there is no perfect match for these search preferences in the product database. Thus, best alternatives are offered. As shown in Fig. 5 COSIMA smartly presents the search results. Following a given sales strategy, COSIMA presents the article with most perfectly fulfilled base preferences. Here she especially emphasizes the perfectly matched red color and fairly mentions the nearly matched volume of 2.7 liters. As the width is perfectly in the customer’s preferred range and also the material is exactly the desired one, COSIMA completes her arguing by emphasizing the ‘perfect’ overall quality. Finally, COSIMA proactively draws the customer’s attention to optional accessories (cross selling). 4.2 Evaluation In an experimental setting 30 test customers had to act as purchasers for a company. They had five tasks, i.e. to buy five exactly specified storage boxes. If there was no such box available they had to select a best alternative on their own responsibility. For the quality valuations for all customers the same
Preference based Quality Assessment and Presentation of Query Results
117
Fig. 5. Smart search result presentation.
adjustments were used. All but two people deemed the quality valuations correct. Thus, a personalization of these parameters is necessary, but obviously an approach of user modelling, e.g. of preference patterns [25] would be appropriate for most of the customers. By applying different sales strategies, customers were divided into two groups of 15 people. In the first group the strategy “best overall quality before most perfect arguments” (strategy b) was applied. For the second group the strategy “second highest price before most perfect arguments” (strategy s) was in use. Please note that the result set is the same for both groups, but the presentation order may differ. Querying the properties of the first task, seven boxes in exactly the same order for both strategies were delivered by COSIMA. She always recommended the first result. Fig. 5 shows there are 5 results visible at first sight. It is remarkable that the very seriously deciding test customers briefly scrolled to lower-listed products, but did not consider their properties. As illustrated in Fig. 6 only results visible at first sight were selected. Two more people followed the “higher quality has a higher price” argumentation of strategy s (product 1 is more expensive than 2, 3, and 5) by selecting the recommended result in contrast to people with strategy b. This result is also emphasized by the results of the non-illustrated tasks 2, 3, and 5. Exemplarily, at last for this experimental setting the experiences of task 4 are illustrated in Fig. 7. Here different sales strategies followed a different order of the resulting products named V, W, X, and Y. The arguments about the quality of the search result with respect to the search preferences (strategy b) leads to selections of the first ordered products, whereas the “higher product
118
Stefan Fischer et al.
Fig. 6. Selections regarding the presentation order of task 1.
quality has a higher price” argumentation (strategy s) leads to more selections of the product W. Here product Y is the most expensive one.
Fig. 7. Selections regarding the presentation orders of task 4.
Summarizing, it can be stated that different sales strategies had indeed different impacts on the test customers. Not each customer appreciated each argumentation, e.g. some liked to hear only positive arguments, others expected to hear at least one not perfectly fulfilled search preference in the argumentation. Thus, a personalization is necessary. If there is no knowledge about the customer, moderate strategies like ‘second highest price’ should be applied. As shown in Fig. 6 this is a powerful instrument to direct the customer choice behavior. Moreover, a good argumentation was experienced as believable and the test customers showed a satisfied attitude during their shopping which is very important for a successful customer relationship [21; 9; 11].
5 Summary and outlook After analyzing the continuing misery of today’s search engines and the problematic point of search result presentation, a review of the preference model of [15; 16] has been given. Moreover, it has been shown that knowledge about the quality of search results with respect to search preferences is a major issue with regard to several aspects, e.g. for consumer choice behavior in e-commerce applications [21; 9; 11]. A novel effective approach of quality assessment for search result presentations has been found. Following the philosophy of [26] a linguistic model for the search result quality has been introduced. An intuitive framework has been elaborated for
Preference based Quality Assessment and Presentation of Query Results
119
situated quality valuations, satisfying the postulate that worse results must not have a higher valuation than better results. Defining quality valuation functions for all base preference constructors of [15; 16] with an output of linguistic quality terms an extensible framework for a human comprehensible search result presentation was created. Based on these results, various situated quality valuations for complex preferences [15; 16] have been introduced. After respecting a so called “but-only” filter, i.e. filter criteria applied over the search results, various intuitive selection criteria have been introduced for deciding which result to present first or which results to proactively offer to a customer, respectively. Especially for the case of a sales process, selection criteria have been presented. For the first time this enables the flexible application of personalized sales strategies [22; 12] within an e-commerce sales process. This can be done in a declarative manner, even coping with conflicting presentation preferences. In combination with a preference search engine, the search and presentation process can be managed very efficiently for users/customers. They do not have to iterate through several pages and moreover get helpful arguments why the results are the suitable ones. Experimental settings showed the applicability of the introduced quality assessment approaches according to the effect on human beings. Fields of applications for the quality assessment could be comparison shops like COSIMA [17]. In that platform a fixed heuristic determining the presentation order is implemented equally for all customers. With this introduced quality assessment, in such an application each customer could be easily and flexibly treated according to his search quality claims and with respect to his presentation preferences. Moreover, some strategies of the service provider could be respected, e.g. to flexibly push sponsored results without conflicting with the customer’s preferences. Obviously, this technology is also predestined for managing personalized decision support. The knowledge for the adjustment of the partitioning parameters is at the moment a manual task for the knowledge engineer of the search engine. Many influences may change the quality sensation of a customer and preferences may vary with the according situation. This dynamic aspect for the definition of a situation is discussed e.g. by the sociologist Thomas [24] or the human factors researcher Endsley [8]. For the detection of partitioning parameters for the quality functions an automated process would be very helpful. An imaginable solution could be a technique like Preference Mining [14; 13].
References [1] R. Agrawal and E. Wimmers. A framework for expressing and combining preferences. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 297–306, 2000. [2] W. Becker. Verkaufspsychologie - Theoretische Grundlagen und praktische Anwendungen. Profil Verlag, 1998.
120
Stefan Fischer et al.
[3] P. Bosc and H. Prade. Uncertainty Management in Information Systems, chapter An Introduction to the Fuzzy Set and Possibility Theory-Based Treatment of Flexible Queries and Uncertain or Imprecise Databases, pages 284–324. Kluwer, Dordrecht, 1994. [4] L. Cholvy and C. Garion. An attempt to adapt a logic of conditional preferences for reasoning with contrary-to-duties. Fundamenta Informaticae, 48(2-3):183–204, 2001. [5] J. Chomicki. Preference formulas in relational queries. ACM Transactions on Database Systems (TODS), 28(4):427–466, 2003. [6] R. Dewan, M. Freimer, and P. Nelson. Impact of search engine ownership on underlying market for goods and services. In Thirty-Fourth Annual Hawaii International Conference on System Sciences (HICSS), Maui, Hawaii, USA, 2001. [7] D. Dubois and H. Prade. Flexible Query Answering Systems, chapter Using fuzzy sets in flexible querying: Why and how? Kluwer Academic Publishers, Norwell, Massachusetts, USA, 1997. [8] M. R. Endsley. Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1):32–64, 1995. [9] J. F. Engel, R. D. Blackwell, and D. T. Kollat. Consumer Behavior. Dryden Press, USA, 1978. [10] S. Fisher. Personalized query result presentation and offer composition for e-procurement applications. Master’s thesis, University of Augsburg, Augsburg, Germany, 2004. [11] F. Hansen. Consumer choice behavior - a cognitive theory. The Free Press, 1972. [12] S.E. Heiman and D. Sanchez. New Strategic Selling: Unique Sales System Proven Successful By World’s Largest Company’s. Warner Books, 1998. [13] S. Holland, M. Ester, and W. Kießling. A novel approach on mining user preferences for personalized applications. In Proceedings of the 7th European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD ’03), pages 204–216, Dubrovnik, Croatia, 2003. [14] S. Holland and W. Kießling. Situated preferences and preference repositories for personalized database applications. In 23rd International Conference on Conceptual Modeling (ER 2004), pages 511–523, Shanghai, China, 2004. [15] W. Kießling. Foundations of preferences in database systems. In Proceedings of the Twenty-eighth International Conference on Very Large Databases, pages 311–322, Hong Kong, China, 2002. [16] W. Kießling. Preference queries with sv-semantics. In Proceedings of the 11th International Conference on Management of Data (COMMAD 2005), pages 15–26, Goa, India, 2005. [17] W. Kießling, S. Fischer, S. Holland, and T. Ehm. A smart and speaking e-sales assistant. In Proceedings of the Third International Workshop on Advanced Issues of E-Commerce and Web-Based Information Systems (WECWIS 2001), pages 21–30, San Jose, CA, USA, 2001.
Preference based Quality Assessment and Presentation of Query Results
121
[18] W. Kießling, B. Hafenrichter, S. Fischer, and S. Holland. Preference xpath: A query language for e-commerce. In Internationale Tagung Wirtschaftsinformatik 2001, pages 427–440, Augsburg, Germany, 2001. [19] W. Kießling and G. K¨ ostler. Preference sql - design, implementation, experiences. In Proceedings of the Twenty-eighth International Conference on Very Large Databases, pages 990–1001, Hong Kong, China, 2002. [20] G. Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review, 63:81–97, 1956. [21] F. M. Nicosia. Consumer Decision Process. Prentice-Hall, Inc., NJ, USA, 1966. [22] N. Rackham. Major Account Sales Strategy. McGraw-Hill Trade, 1989. [23] C. P. Simon and L. Blume. Mathematics for Economics. W. W. Norton & Company, 1994. [24] W. I. Thomas. The Unadjusted Girl. Little, Brown, and Co. Boston, 1923. [25] Q. Wang, W. T. Balke, W. Kießling, and A. Huhn. P-news: Deeply personalized news dissemination for mpeg-7 based digital libraries. In 8th European Conference on Digital Libraries (ECDL 2004), pages 256– 268, Bath, UK, 2004. [26] L. A. Zadeh. The Concept of a linguistic variable and its application to approximate reasoning. Elsevier Pub. Co., New York, NY, USA, 1973.
Modeling Fuzzy Information in the EER and Nested Relational Database Models Zongmin M. Ma Department of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning 110004, China
1 Introduction A major goal for database research has been the incorporation of additional semantics into database models. It is recognized that the relational database model has semantic and structured drawbacks when it comes to modeling some emerging applications such as computer aided design (CAD), geographical information systems (GIS), and artificial intelligence. In response to this problem, some attempts to relax the first normal form (1NF) limitation, which is the most fundamental normalization constraint in the relational databases, are made and one kind of data model, called non-first normal (or nested) relational database model, has been introduced. In common sense, the nested relational database model means that attribute values in the relational instances are either atomic or set-valued and even relations themselves. In addition, the next generation of database models takes the form of rich data models such as the object-oriented database model and the semantic (conceptual) data models. In real world applications, information is often vague or ambiguous. Therefore, different kinds of imperfect information have been extensively introduced into relational databases [8, 18, 25, 26]. However, the classical relational database model and its extensions with imprecision and uncertainty does not satisfy the need of modeling complex objects with imprecise and uncertain information. So, many researches have concentrated on conceptual data models [10, 31, 38] and object-oriented databases (OODB) [2, 12-17, 22] to deal with complex objects and uncertain data together. It should be pointed out that, being the extension of the relational database model, the nested relational database model is able to handle complex-valued attributes and may be better suitable to some complex applications (e.g., office automation systems, information retrieval systems and expert database systems [33]). In [33], an 2 extended nested relational database model (also known as an NF data model) Z.M. Ma: Modeling Fuzzy Information in the EER and Nested Relational Database Models, Studfuzz 203, 123–146 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
124
Zongmin M. Ma
was introduced for representing and manipulating complex and uncertain data in databases, and the extended algebra and the extended SQL-like query language were thereby defined. Also physical data representation of the model, and the core operations that the model provides, were also introduced in [33], in which the uncertain data were represented based on similarity relationship [8]. In [20], based on possibility distribution and the semantic measure of fuzzy data developed in [18], a fuzzy nested relational database model was briefly presented to cope with imperfect as well as complex objects in the real world. Generally speaking, the classical relational databases are designed by developing high-level conceptual data models first, and then the developed conceptual models are mapped to actual implementations, i.e., database models. The ER model [11], for example, is extensively used for conceptual design of the relational databases [29]. In addition, conceptual data models, EER, IFO [1] and UML have been used for conceptual design of the object-oriented databases [23, 24]. It should be noticed, however, that less research has been done in developing the design methodologies for implementing the databases with imprecise and uncertain information although the conceptual data models and logical database models have been extended for representing and handling imperfect information. In this chapter, we focus on fuzzy data modeling at conceptual level, in the EER model, a data model that is extensively used in conceptual data modeling, and, at logical level, in the nested relational databases based on possibility distribution. A complete extension of nested relational databases for extended-based fuzzy data is presented in the chapter. In particular, the formal approach to mapping a fuzzy EER model to a fuzzy nested relational database (FNRDB) schema is developed. The remainder of this chapter is organized as follows. Section 2 describes related work. The fuzzy extended entity-relationship (FEER) model and fuzzy nested relational databases are introduced in Section 3. In Section 4, the approaches to mapping the fuzzy EER model to the fuzzy nested relational schema are developed. Section 5 concludes this chapter.
2 Related work Fuzzy information has been extensively investigated in the context of the relational database model [8, 18, 25, 26] since Zadeh proposed fuzzy set theory [34]. In connection to the representations of fuzzy data, two basic extended data models for fuzzy relational databases can be identified: one is based on possibility distribution [25, 26] and the other one is based on similarity (proximity or resemblance) relation [8]. The former can further be classified into two categories, i.e., tuples associated with possibility degrees and attribute values represented possibility distributions, which respectively correspond to the type-1 fuzzy relational data model and the type-2 fuzzy relational data model [26]. The form of an n-tuple in each of the above-mentioned fuzzy relational models can be expressed, respectively, as:
Modelling Fuzzy Information in the EER
125
t1 = , t2 = <πA1, πA2, ..., πAi, ..., πAn,> and t3 = , where pi ⊆ Di with Di being the domain of attribute Ai, ai ∈ Di, d ∈ (0, 1], πAi is the possibility distribution of attribute Ai on its domain Di, and πAi (x), x ∈ Di, denotes the possibility that x is the actual value of t[Ai]. Notation d is used to denote the membership degree of the tuples in the corresponding relation. The value d of the tuple may be interpreted either as a possibility measure of association of the attribute values or as a truth value of a fuzzy predicate associated with the relation. Viewed from the expressive format, pi is a set value containing some elements of Di, in which these elements have similarity relations. In other words, pi is a fuzzy data represented by similarity relations in the elements of Di. It is clear that, based on the above-mentioned basic fuzzy relational models, there should be one kind of hybrid fuzzy relational model, called extended possibility-based fuzzy relational model [13], in which possibility distribution and similarity (proximity or resemblance) relation arise in a relational database simultaneously. Based on various fuzzy relational database models, many studies have been done for data integrity constraints [6, 7, 26, 27]. Also there have been research studies on fuzzy query languages [5, 28] and fuzzy relational algebra [30, 21]. In [5], an existing query language, namely SQL, for fuzzy queries was extended and some fuzzy aggregation operators were developed. In [36], a fuzzy relational data base (FRDB) model architecture and query language were presented and the possible applications of the FRDB in imprecise information processing were discussed. In order to model complex objects with imprecise and uncertain information, current efforts have been concentrated on some fuzzy non-traditional data models for databases such as the conceptual data models [10, 31, 38], the nested relational model [20, 33], and the object-oriented data model [2, 12-17, 22]. The conceptual data models can sever as the tool for designing databases. In [33], null values, set values, range values (partial values and value intervals), and 2 fuzzy values were all modeled in the NF data model based on similarity 2 relationship and the extended NF algebra was developed. In [20], fuzzy data were introduced into nested relational databases to model complex objects with imprecise and uncertain information, where fuzzy data are the extended possibility-based fuzzy data expressed by possibility distributions over a universe of discourse, and meanwhile a resemblance relation on the universe is in effect. In [20] the structure of the fuzzy nested relational model was briefly presented. Regarding modeling imprecise and uncertain information in the objectoriented databases (OODBs), Zicari and Milano first introduced incomplete information, namely, null values in [37], where incomplete schema and incomplete objects can be distinguished. From then on, the incorporation of
126
Zongmin M. Ma
imprecise and uncertain information into the object-oriented databases has increasingly received the attentions and fuzziness is witnessed at the levels of object instances and class hierarchies. Based on similarity relationship, in [16], the range of attribute values was used to represent the set of allowed values for an attribute of a given class. Depending on the inclusion of the actual attribute values of the given object into the range of the attributes for the class, the membership degrees of an object to a class can be calculated. The weak and strong class hierarchies were defined based on monotone increase or decrease of the membership of a subclass in its superclass. Based on the extension of a graphs-based object model, a fuzzy object-oriented data model was defined in [2]. The notion of strength expressed by linguistic qualifiers, which can be associated with the instance relationship as well as an object with a class, was proposed. Fuzzy classes and fuzzy class hierarchies were thus modeled in OODBs. In [3], the graph-based operations to select and browse a fuzzy object oriented database were further defined for the fuzzy graph-based model that manages both crisp and fuzzy information. The UFO (uncertainty and fuzziness in an object-oriented) database model was proposed in [17] to model fuzziness and uncertainty by means of fuzzy sets and generalized fuzzy sets, respectively. The fact that the behavior and structure of the object are incompletely defined results in a gradual nature for the instantiation of an object. The partial inheritance, conditional inheritance, and multiple inheritances were permitted in fuzzy hierarchies. Based on possibility theory, vagueness and uncertainty were represented in class hierarchies in [15], where the fuzzy ranges of the subclass attributes defined restrictions on that of the superclass attributes and then the degree of inclusion of a subclass in the superclass was dependent on the inclusion between the fuzzy ranges of their attributes. Focusing on the fuzzy types, Marín, Pons and Vila [22] discussed two different strategies for adding fuzzy types to an object-oriented database system and presented how the typical classes of an OODB can be used to represent a fuzzy type and how the mechanisms of instantiation and inheritance can be modeled using this kind of new type on an OODB. Recent efforts have been paid on the establishment of consistent framework for a fuzzy object-oriented model based on the standard for the Object Data Management Group (ODMG) object data model [12, 13]. In [14], an object-oriented database modeling technique was presented based on the concept ‘level-2 fuzzy set’ to deals with a uniform and advantageous representation of both perfect and imperfect ‘real world’ information. Also it was illustrated and discussed how the ODMG data model can be generalized to handle ‘real world’ data in a more advantageous way. Concerning the conceptual data modeling of fuzzy information, Zvieli and Chen [38] introduced three levels of fuzziness in the ER model based on fuzzy set theory. At the first level, entity sets, relationships, and attribute sets may be fuzzy, namely, they have membership degrees in the data model. The second
Modelling Fuzzy Information in the EER
127
level is related with the fuzzy occurrences of entities and relationships. The third level concerns the fuzzy values of attributes of special entities and relationships. The generic rules for mapping the fuzzy ER schema to the fuzzy relational databases were introduced in [9]. Also, based on the three levels of fuzziness in the ER model, the fuzzy extension of several major EER concepts was introduced in [10] using fuzzy set theory, including superclass/subclass, generalization/specialization, category, and the subclass with multiple superclasses. The graphical representations of the fuzzy EER model and the design methodologies for the fuzzy object-oriented databases were developed in [19]. In addition, IFO data model proposed in [1], which is a conceptual data model that incorporates the fundamental principles of semantic database modeling within a graph-based representational framework, was extended to deal with fuzzy information in [31, 32]. In [31], several types of imprecision and uncertainty such as the values without semantic representation, the values with semantic representation and disjunctive meaning, the values with semantic representation and conjunctive meaning, and the representation of uncertain information were incorporated into the attribute domain of the object-based data model. In [32], a mapping process to transform the conceptual schemas of the 2 ExIFO (Extended IFO) model into the similarity-based fuzzy extended NF relations [33] including uncertain properties that are represented in both models was also described in [32].
3 Fuzzy database modeling In this section, we concentrate on fuzzy database modeling at conceptual and logical levels. First we introduce some notions and notations of the fuzzy ER/EER model proposed in [10, 19, 38] and then we give a complete extension of the extended possibility-based fuzzy nested relational databases. 3.1 Fuzzy extended entity-relationship (EER) model Attribute. In the fuzzy ER/EER model, the following levels of fuzziness can be found in attributes. The first level of fuzziness means that attributes may be fuzzy corresponding to the data model. In other words, they have their membership degree. The third levels of fuzziness means that attributes may take fuzzy values. The membership degree here means uncertainty. The membership degree connected with an attribute means that the attribute may or may not belong to the corresponding entity. These two levels of fuzziness can appear in single-valued and multivalued attributes, and should be identified from crisp
128
Zongmin M. Ma
attributes in the data model. A composite attribute consists of any kind of the attribute types mentioned above. The graphical representations of these types of the attributes are shown in Figure 3.1. Here, for the attribute with a membership degree, we place membership degree inside the diagrams of attribute in the fuzzy ER model. For example, let A be an attribute type and µ (A) be its membership degree in the model. Then "µ (A)/A" is enclosed in the rectangle. If µ (A) = 1, "1/A" is usually denoted "A" simply. In a similar way, entity types and relationship types with membership degrees can be represented. Entity and Relationship. The following levels of fuzziness can be found in entities of the fuzzy ER/EER model: the first level of fuzziness means that entities may be fuzzy corresponding to the data model, i.e., they have their membership degree; the second level of fuzziness is related with the instances of special entities and relationships, which means that an entity instance belongs to the corresponding entity and relationship fuzzily, respectively. If entities are connected with the attributes with the third level of fuzziness, we say such entities are ones with the third level of fuzziness. These three levels of fuzziness can also be found in the relationships of the fuzzy ER/EER model. Here the membership degree connected with an entity (or relationship) means that the entity (or relationship) may or may not belong to the corresponding data model. It should be noted that an entity type can be a weak entity type and a relationship type may be an ownership relationship.
(a) µ
(e)
(b)
(c)
(d)
µ
µ (A)/A
µ
(f)
(g)
(h)
(a) Single-valued attribute type (b) Multivalued attribute type (c) Disjunctive fuzzy value attribute type (d) Conjunctive fuzzy value attribute type (e) Single-valued attribute type with membership degree (f) Multivalued attribute type with membership degree (g) Disjunctive fuzzy value attribute type with membership degree (h) Conjunctive fuzzy value attribute type with membership degree
Fig. 3.1 Attributes in Fuzzy ER/EER Model
Modelling Fuzzy Information in the EER
129
The graphical representations of the entities and relationship in the fuzzy ER/EER model are shown in Figure 3.2. Generalization/Specialization. Consider two entity types E and S on the universe of discourse U and they are fuzzy sets with membership functions µE and µS, respectively. Then S is a fuzzy subclass of E and E is a fuzzy superclass of S if and only if the following is true. (∀ e) (e ∈ U ∧ µS (e) ≤ µE (e))
µ (E)/E
E
µ(R)/R
R
(a)
(b)
(c)
(d)
(a) Entity with membership degree (b) Entity with fuzzy instances
(c) Relationship with membership degree (d) Relationship with fuzzy instances
Fig. 3.2 Entities and Relationships in Fuzzy ER/EER Model A generalization defines a superclass from several entity types, generally being with some common features, while a specialization defines several subclasses from an entity type according to a certain feature. Considering a fuzzy superclass E and its fuzzy subclasses S1, S2, …, Sn with membership functions µE, µS1, µS2, ..., and µSn, respectively, Then the following relationship is true. (∀ e) (∀ S) (e ∈ U ∧ S ∈ {S1, S2, …, Sn} ∧ µS (e) ≤ µE (e)) S1, S2, …, Sn are a fuzzy total specialization of E if (∀ e) (∃ S) (e ∈ E ∧ S ∈ {S1, S2, …, Sn} ∧ µS (e) > 0 ∧ µS (e) ≤ µE (e)) S1, S2, …, Sn are a fuzzy partial specialization of E if (∃ e) (∀ S) (e ∈ E ∧ S ∈ {S1, S2, …, Sn} ∧ µE (e) > 0 ∧ µS (e) = 0) In addition, S1, S2, …, Sn are disjoint if
130
Zongmin M. Ma (∃/ e) (∀ S) (∀ S’) (e ∈ U ∧ S ∈ {S1, S2, …, Sn} ∧ S’ ∈ {S1, S2, …, Sn} ∧ min (µS (e), µS’ (e)) > 0)
S1, S2, …, Sn are overlapping if (∃ e) (∀ S) (∀ S’) (e ∈ U ∧ S ∈ {S1, S2, …, Sn} ∧ S’ ∈ {S1, S2, …, Sn} ∧ min (µS (e), µS’ (e)) > 0) Generalization is the inverse process of specialization. Therefore, we have fuzzy total generalization, fuzzy disjoint generalization, and fuzzy overlapping generalization according to the discussion above. It should be noted that there is no fuzzy partial generalization, which is the same as the situation in the classical EER model. Category. Let S1, S2, …, Sn, and E be fuzzy entity sets with membership functions µS1, µS2, ..., µSn, and µE, respectively. Then E be a fuzzy category of S1, S2, …, Sn if (∀ e) (∃ S) (e ∈ E ∧ S ∈ {S1, S2, …, Sn} ∧ µE (e) > 0 ∧ µS (e) ≥ µE (e)) Note the difference between the fuzzy category and the fuzzy subclass with more than one fuzzy superclass. Let E be a fuzzy subclass and S1, S2, …, Sn be its fuzzy superclasses, which membership functions are respectively µE, µS1, µS2, ..., and µSn. Then (∀ e) (∀ S) (e ∈ E ∧ S ∈ {S1, S2, …, Sn} ∧ µE (e) > 0 ∧ µS (e) ≥ µE (e)) Aggregation. Let E be an fuzzy aggregation of fuzzy entity sets S1, S2, …, and Sn, which membership functions are µE, µS1, µS2, ..., and µSn, respectively. Then (∃ e) (∃ e1) (∃ e2) ... (∃ en) (e ∈ E ∧ e1 ∈ S1 ∧ e2 ∈ S2 ∧ … ∧ en ∈ Sn ∧ µE (e) = µS1 (e1) × µS2 (e2) × ... × µSn (en)) The diagrammatic notations in Figure 3.3 are used to represent fuzzy specialization, category and aggregation in the fuzzy EER model, respectively.
Modelling Fuzzy Information in the EER
131
d
o
d
o
∧
∨
×
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(a) Fuzzy total & disjoint specialization (b) Fuzzy total & overlapping specialization (c) Fuzzy partial & disjoint specialization (d) Fuzzy partial & overlapping specialization (e) Fuzzy subclass with fuzzy multi-superclasses (f) Fuzzy category ( )F i
Fig. 3.3 Specialization, Category and Aggregation in Fuzzy EER Model
3.2 Extended possibility-based fuzzy nested relational databases Model structure 2
A fuzzy NF relational schema is a set of attributes (A1, A2, ..., An, pM) and their domains are D1, D2, ..., Dn, D0, respectively, where Di (1 ≤ i ≤ n) can be one of the following: (1) The set of atomic values. For each element ai ∈ Di, it is a typical simple crisp attribute value. (2) The set of null values, denoted ndom, where null values may be unk, inap, nin, and onul. (3) The set of fuzzy subset. The corresponding attribute value is an extended possibility-based fuzzy data. (4) The power set of the set in (1). The corresponding attribute value, say ai, is multivalued one with the form of {ai1, ai2, ..., aik}. (5) The set of relation values. The corresponding attribute value, say ai, is a tuple of the form which is an element of Di1 × Di2 × ... × Dim (m > 1 and 1 ≤ i ≤ n), where each Dij (1 ≤ j ≤ m) may be a domain in (1), (2), (3), and (4) and even the set of relation values. The domain D0 is a set of atomic values and each value is a crisp one from the range [0, 1], representing the possibility degree that the corresponding tuple is 2 true in the NF relation. We assume that the possibilities of all tuples are precisely one in the chapter. Then for an attribute Ai ∈ R (1 ≤ i ≤ n), its attribute domain is formally represented as follows: τi = dom | ndom | fdom | sdom | where B1, B2, …, Bm are attributes.
132
Zongmin M. Ma
A relational instance r over the fuzzy NF schema (A1 : τ1, A2 : τ2, ..., An : τn) is a subset of Cartesian product τ1 × τ2 × ... × τn. A tuple in r with the form of consists of n components. Each component ai (1 ≤ i ≤ n) may be an atomic value, null value, set value, fuzzy value, or another tuple. 2 An example of the fuzzy NF relation is shown in Table 3.1. It can be seen that Tank_Id and Start_data are crisp atomic-valued attributes, Tank_body is a relation-valued attribute, and Responsibility is a set-valued attribute. In the attribute Tank_body, two component attributes Volume and Capacity are fuzzy ones. 2
Table 3.1. Pressured air tank relation Tank _Id
TA1
Body _Id BO01
TA2
BO02
Tank_body Material Volume
Alloy Steel
about 2.5e +03 about 2.5e +04
Capacity
about 1.0e +06 about 1.0e +07
Start_ Date
Responsibility
01/12/99
John
28/03/00
{Tom, Mary}
Assessment of data redundancies and removal 2
In order to manipulate fuzzy data in the fuzzy NF relational databases, it is necessary to evaluate the semantic measure of fuzzy data. On the basis, data redundancies can be assessed and removed. For two extended possibility-based fuzzy data, say πA and π B, there exist four types of semantic relationships between them, which are equivalence, inclusion, intersection, and irrelevancy. The notion of semantic inclusion degree is proposed in order to assess these semantic relationships in [19]. Definition 3.1. Let U = {u1, u2, …, un} be an universe of discourse. Let πA and πB be two fuzzy data on U based on possibility distribution. The semantic inclusion degree of π A and π B SID ( π A, π B), which means π A semantically includes πB, is then defined as follows: n
n
SID (πA, πB) = ∑ min (π B (u i ), π A (u i )) / ∑ π B (u i ) i =1 ui ∈U
i =1
Following the definition 3.1 and a given threshold β, we have (1) πA semantically includes πB iff SID (πA, πB) ≥ β.
Modelling Fuzzy Information in the EER
133
πA and πB are semantically equivalent to each other iff SID (πA, πB) ≥ β and SID (πB, πA) ≥ β. (3) πA and πB are semantically irrelevant to each other iff SID (πA, πB) < β and SID (πB, πA) < β. Here, we only focus on if πA and πB are semantically equivalent to each other.
(2)
Therefore, the notion of equivalence degree for two fuzzy data is given as follows. Definition 3.2. Let πA and πB be two fuzzy data and SID (πA, πB) be the degree that πA semantically includes πB. The semantic equivalent degree of πA and πB SE (πA, πB), denoting the degree that πA and πB are equivalent to each other, is defined as follows. SE (πA, πB) = min (SID (πA, πB), SID (πB, πA)) Two fuzzy data πA and πB are considered β-redundant if and only if SE (πA, πB) ≥ β. For two crisp data, atomic or set-valued, their equivalent degree is one if they are equal to each other, where the same set-valued data are considered equal. Assume that πA and πB are β-redundant to each other, the elimination of duplicate could be achieved by merging πA and πB and producing a new fuzzy data πC, where πA, πB, and πC be three fuzzy data on U = {u1, u2, …, un} based on possibility distribution and there is a resemblance relation on U. Then we define the following merging operation according to Zadeh’s extension principle [35]. πC = πA ∪f πB = {πC (w) / w | (∃ πA (ui) / ui) (∃ πB (vj) / vj) (πC (w) = Max (πA (ui), πB (vj)) ∧ ( w = ui | πC(w) = πA(ui) ∨ w = vj | πC(w) = πB(vj)) ∧ Res (ui, vj) ≥ α ∧ ui, vj ∈ U ∧ 1 ≤ i, j ≤ n) ∨ (∃ πA(ui) / ui) (∀ πB (vj) / vj) (πC (w) = πA(ui) ∧ w = ui ∧ Res (ui, vj) < α ∧ ui, vj ∈ U ∧ 1 ≤ i, j ≤ n) ∨ (∃ πB (vj) / vj) (∀ πA (ui) / ui) (πC (w) = πB (vj) ∧ w = vj ∧ Res (ui, vj) < α ∧ ui, vj ∈ U ∧ 1 ≤ i, j ≤ n)} 2
Now let us concentrate on the redundancies of tuples in a fuzzy NF relation. First, let us look at two values on a structured attribute aj = (Aj1: πAj1, Aj2: πAj2, …, Ajm: πAjm) and aj’ = (Aj1: πAj1’, Aj2: πAj2’, …, Ajm: πAjm’), which consist of simple attribute values, crisp (atomic and set-valued) or fuzzy, on the schema R (Aj1, Aj2, ..., Ajm). There is a resemblance relation on each attribute domain Djk (1 ≤ k ≤ m) and αjk ∈ [0, 1] (1 ≤ k ≤ m) is the threshold on the resemblance relation.
134
Zongmin M. Ma
Let β ∈ [0, 1] be a given threshold. aj and aj’ are α-β-redundant if and only if for k = 1, 2, …, m, min (SEαjk (πAjk, πAjk’)) ≥ β holds true. min (SEαjk (πAjk, πAjk’)) (1 ≤ k ≤ m) is called the equivalence degree of structured attribute values. Consequently, the notion of equivalence degree of structured attribute values can be extended for the tuples in the fuzzy nested relations to assess tuple redundancies. Informally, any two tuples in a nested relation are redundant, if, for pair of the corresponding attribute values, the equivalence degree is greater than or equal to the threshold value. If the pair of the corresponding attribute values is simple, the equivalence degree is one for two values. For two values of structured attributes, however, the equivalence degree is one for structured attributes. Two redundant tuples t and t’ are written t ≡ t’. Fuzzy nested relational operations Union and Difference. Let r and s be two union-compatible fuzzy nested relations. Then r ∪ s = min ({t | t ∈ r ∨ t ∈ s}) and r − s = {t | t ∈ r ∧ (∀v ∈ s) (t ≡/ v)} Here, the operation min () means to remove the fuzzy redundant tuples in r and s. Of course, the threshold value should be provided for the purpose. Cartesian Product. Let r and s be two fuzzy nested relations on schemas R and S, respectively. Then r × s is a fuzzy nested relation with the schema R ∪ S. The formal definition of Cartesian product operation is as follows: r × s = {t | t (R) ∈ r ∧ t (S) ∈ s} Projection. Let r be a fuzzy nested relation on the schema R and S ⊂ R. Then the projection of r on the schema S is formally defined as follows: ΠS (r) = min ({t | (∀ v ∈ r) (t = v (S)}) Here, an attribute in S may be of the form B.C, in which B is a structured attribute and C is its component attribute. Being the same as union operation, projection operation also needs to remove fuzzy redundant tuples in the result relation after the operation. Selection. In classical relational databases, the selection condition is of the form X θ Y, where X is an attribute, Y is an attribute or a constant value, and θ ∈ {=, ≠, >, ≥, <, ≤}. In order to implement fuzzy query for fuzzy relational databases, “θ” should be fuzzy, denoted ≈, ≈/ , , ≺ , , and ≺ . In addition, X
Modelling Fuzzy Information in the EER
135
is only a simple attribute or the simple attribute of a structured attribute but Y may be one of the following. (a) A constant, crisp or fuzzy one; (b) A simple attribute; (c) The simple component attribute of a structured attribute, having the form A. B, where A is a structured attribute and B is its simple component attribute. Assume that there is a resemblance relation on the universe of discourse and α is the threshold on it. Then the fuzzy comparison operations are defined as follows (1) X ≈ Y iff SEα (X, Y) ≥ β, where β is a selected cut (the followings are the same). (2) X ≈/ Y iff SEα (X, Y) < β. (3) X Y iff X ≈/ Y and min (Supp (X)) > min (Supp (Y)). (4) X Y iff X ≈ Y or X Y. (5) X ≺ Y iff X ≈/ Y and min (Supp (X)) < min (Supp (Y)). (6) X ≺ Y iff X ≈ Y or X ≺ Y. Depending on Y, the following situations can be identified for the selection condition X θ Y. Let X be the attribute Ai: τi in a fuzzy nested relation. (1) Ai θ c, where c is a crisp constant. According to τi, the definition of Ai θ c is as follows: • if τi is dom, Ai θ c is a traditional comparison and θ ∈ {=, ≠, >, <, ≥, ≤}, • if τi is fdom, Ai θ c is a fuzzy comparison and θ ∈ {≈, ≈/ , , ≺ , , ≺ }, • if τi is ndom, Ai θ c is a null comparison and regarded as the special fuzzy comparison, • if τi is sdom, Ai θ c is a element-set comparison. Then Ai θ c if c and any element in the value of Ai of a tuple satisfy the “θ”. (2) Ai θ f, where f is a fuzzy value. • if τi is dom, fdom, or ndom, Ai θ f is a fuzzy comparison and θ ∈ {≈, ≈/ , ≺ , , ≺ }, • if τi is sdom, Ai θ f is a fuzzy set comparison. Then Ai θ f if f and any element in the value of Ai of a tuple satisfy the fuzzy “θ”, where θ ∈ {≈, ≈/ , , ≺ , , ≺ }. (3) Ai θ Aj, where Aj: τj is a simple attribute and i ≠ j. • if τi and τj are all dom, Ai θ Aj is a traditional comparison, • if τi and τj are dom and fdom, fdom and fdom, or ndom and fdom, Ai θ Aj is a fuzzy comparison, • if τi and τj are dom and ndom, Ai θ Aj is a null comparison, • if τi and τj are dom and sdom, Ai θ Aj is a element-set comparison, • if τi and τj are fdom and sdom, Ai θ Aj is a fuzzy set comparison,
,
136
Zongmin M. Ma • if τi and τj are all ndom, Ai θ Aj is a null-null comparison. Then Ai θ Aj if they have the same null values on the same universe of discourse, • if τi and τj are ndom and sdom, Ai θ Aj is a null-set comparison and regarded as the special element-set comparison, • if τi and τj are sdom and sdom, Ai θ Aj is a set-set comparison and regarded as the special element-set comparison.
(4) Ai θ Aj. B, where Aj is a structured attribute (i ≠ j) and B is a simple attribute. The situations are the same as those in above case (3). In the fuzzy nested relational databases, the selection condition is similar to the selection condition in the fuzzy relational databases except that the attribute may be of the form B.C, where B is a structured attribute and C is its component attribute. Let Q be a predicate denoting the selection condition. The selection operation for a fuzzy nested relation r is defined as follows: σQ (r) = {t | t ∈ r ∧ Q (t)} In addition to some traditional relational operations, two restructuring, called Nest and Unnest (called Pack and Unpack, Merge and Unmerge also in literature), are also crucial in the fuzzy nested relational databases. The Nest operator can gain the nested relation with structured attributes. The Unnest operator is used to flatten the nested relation. That is, it takes a nested relation on a set of attributes and desegregates it, creating a "flatter'' structure. Nest Operation. Let r be a fuzzy nested relation with the schema R = {A1, A2, …, Ai, …, Ak, …, An}, where 1 ≤ i, k ≤ n. Now Y = {Ai, …, Ak} is merged into a structured attribute B and a new fuzzy nested relation s is formed, which schema is S = {A1, A2, …, Ai-1, B, Ak+1, …, An}. The following notation is used to represent the Nest operation above: s (S) = ΓY → B (r (R)) = {ω [(R − Y) ∪ B] | (∃u) (∀v) (u ∈ r ∧ v ∈ r ∧ SE (u [R − Y], v [R − Y]) < β ∧ ω [R − Y] = u [R − Y] ∧ ω [B] = u [Y]) ∨ (∀u) (∀v) (u ∈ r ∧ v ∈ r ∧ SE (u [R − Y], v [R − Y]) ≥ β ∧ ω [R − Y] = u [R − Y] ∪f v [R − Y] ∧ ω [B] = u [Y] ∪ v [Y])} It can be seen that in the process of the Nest operation on attribute sets Y to B, multiple tuples in r which are fuzzily equivalent on the attribute set R − Y are merged to form a tuples of s. Such merging operation is respectively completed on attribute sets R − Y and Y. On the R − Y, fuzzy union ∪f is used and for an attribute C ∈ R − Y, the value of C of the created tuple is an atomic value, crisp
Modelling Fuzzy Information in the EER
137
or fuzzy. The value of an attribute B. C ∈ Y of the created tuple, however, is a set value and the common union is used. Another restructuring operation, called Unnest, is an inverse of Nest under certain conditions. In a classical nested relation, this condition is that the nested relation is in Partitioned Normal Form (PNF). A relation is in PNF if and only if (a) all of a subset of the simple attributes forms a relation key and (b) every sub-relation is in PNF. Unnest Operation. Let s be a fuzzy nested relation with the schema S = {A1, A2, …, Ai-1, B, Ak+1, …, An}, where B is a structured attribute and B : {Ai, …, Ak}. Unnest operation products a new fuzzy nested relation r, which schema is R = {A1, A2, …, Ai-1, Ai, …, Ak, Ak+1, …, An}, i.e., R = S − B ∪ {Ai, …, Ak}. The following notation is used to represent the Unnest operation above: r (R) = Ξ B (s (S)) = {t [(R − B) ∪ { Ai, …, Ak }] | (∀u) (u ∈ s ∧ t [R − B] = u [R − B] ∧ t [Ai … Ak] ∈ u [B])}
4
Mapping a fuzzy EER model to a fuzzy nested relational schema
After constructing the fuzzy EER (FEER) model, we can map it to the fuzzy nested relational database (FNRDB) model. For this purpose, first of all, we should know how the fuzzy nested relational databases support the fuzzy EER model. Three levels of fuzziness can be found in the fuzzy EER model, in which the first level of fuzziness occurs in metadata. The fuzzy nested relational databases do not support this level of fuzziness, which can only model the fuzziness that occurs in the schema/instance and instance levels. In addition, an entity in the EER model generally corresponds to a relation in the nested relational databases and the attributes of an entity correspond to the attributes of the relation. The relationships in the EER model can be mapped into the attributes of the relations that are created by the entities connected with the relationships. The details of the formal mapping are illustrated as follows. 4.1 Transformation of entities Generally speaking, an entity in the FEER model can be transformed to a relation in the FNRDB and the attributes of the entity may be considered as the attributes of the mapped relation directly. Note that the multivalued attributes of the FEER model are mapped into set-valued attributes in the FNRDB.
138
Zongmin M. Ma
According to the discussion above, we can distinguish four kinds of entities in the FEER model as follows: (a) entities without any fuzziness at the three levels, (b) entities with the fuzziness only at the third level, (c) entities with the fuzziness at the second level, and (d) entities with the fuzziness at the first level. For the entities in (a) and (b), we can directly transform them to the relations. For the entities in (c), however, an additional attribute should be added to each relation transformed from the corresponding entity, which is used to denote the possibility that the tuples belong to the relation. It should be noted that for the entities with membership degree and the entities whose attributes have membership degree, the fuzzy nested relational model cannot represent the first level of fuzziness and the transformations from the entities to the relations and from the attributes of the entities to the attributes of the relations can not be completed. Figure 4.1 shows the transformation of the entities. Good Partner
Good Partner
Number
Number
Site
Site
Sate
pD
Sate
Fig. 4.1 Transformations of Entities Note that the above-mentioned entities are not subclass entities, which mapping will be discussed below. In addition, a weak entity in the FEER model depends on its owner entity. A weak entity in the FEER model can be mapped to a relation-valued attribute. 4.2 Transformation of relationships A relationship in the EER model should be mapped into an association relation, which attributes serve as a group of pointer to combine an explicit reference from one tuple to another tuple. Considering the constraint of cardinality, such attributes in two associated tuples can be single-valued or multivalued attributes. Let entity E1 with attributes {K1, A1, …, Am} and entity E2 with attributes {K1, B1, …, Bn} be connected with relationship R {X1, …, Xk}, where K1 and K2 are key attributes of E1 and E2, respectively, and R may be an one-to-one, one-tomany, or many-to-many relationship. In addition to the transformation process
Modelling Fuzzy Information in the EER
139
of the entity transformation given above, the relationship R is mapped into a relational schema with attribute set {K1, K2, X1, …, Xk}, where K1 and K2 are key attributes. If the constraint of cardinality is a one-to-many relationship, i.e., R is a one-to-many relationship from E1 to E2, K2 is a set-valued attribute of r1 created by E1, and K1 is a single-valued attribute of r2 created by E2. Considering the fuzziness in the FEER model, we can distinguish four kinds of relationships as follows: (a) relationships without any fuzziness at the three levels, (b) relationships with the fuzziness only at the third level (it means that the attributes of relationships may take fuzzy values), (c) relationships with the fuzziness at the second level, and (d) relationships with the fuzziness at the first level. For the relationship R in (a) or (b), the transformation of the relationships can be conducted according to the rules given above. For the relationship R in (c), an additional attribute denoting the possibility that the relationship instance belongs to the relation should be added into the relation r created by R. It should be noted that for the relationships with membership degree and the relationships whose attributes have membership degree, the fuzzy nested relational model cannot represent the first level of fuzziness and the transformation of such relationships can not be completed. Figure 4.2 shows the transformation of the relationships.
r1 K1
E1
A1
K1
A1
pD
r2 X
K2
R
B1
pD
r K1 K2
E2
K2
X
pD
B1
Fig. 4.2 Transformations of Relationships 4.3 Transformation of generalizations Let E1 with attributes {K1, A1, A2, …, Ak} and E2 with attributes {K2, B1, B2, …, Bm} be generated to form supertype S. Assume {A1, A2, …, Ak} ∩ {B1, B2, …, Bm} = {C1, C2, …, Cn}. Generally speaking, E1 and E2 are mapped into schemas
140
Zongmin M. Ma
{K1, A1, A2, …, Ak} − {C1, C2, …, Cn} and {K2, B1, B2, …, Bm} − {C1, C2, …, Cn}, respectively. As to the transformation of S, depending on K1 and K2, we distinguish the following two cases. (a) K1 and K2 are identical. Then S is mapped into the schema {K, C1, C2, …, Cn}, where K is the same as K1/K2. (b) K1 and K2 are different. Then S is mapped into the relational schema {K, C1, C2, …, Cn}, where K is the surrogate key created by K1 and K2 [32]. Considering the fuzziness in entities, the following cases for the transformation of the generalization are distinguished: (a) E1 and E2 are both crisp. Then E1 and E2 are transformed to relations r1 and r2 with attributes {K1, A1, A2, …, Ak} − {C1, C2, …, Cn} and {K2, B1, B2, …, Bm} − {C1, C2, …, Cn}, respectively. S is transformed to a relation r with attributes {K, C1, C2, …, Cn}. (b) When there is the fuzziness of instance/schema level in E1 and (or) E2, being similar to case (a) also, relation r, as well as relations r1 and r2, are formed. Note that r, r1 and (or) r2 created by E1 and (or) E2 with the instance/schema level of fuzziness should include attribute pD. (c) When there is the fuzziness of schema level in E1 and (or) E2, relation r, as well as relations r1 and r2, are formed. But the fuzziness at this level cannot be represented in the created relations. Figure 4.3 shows the transformations of the generalization. Owner
Address
d
Person
Owner Address
pD
Number
Person Sex
pD
Address
Company
Number
Number
Sex
Number
Number
Company BossName
BossName
Fig. 4.3 Transformation of Generalization
Modelling Fuzzy Information in the EER
141
4.4 Transformation of specializations Let S be an entity type with attributes {K, A1, A2, …, An}, where K is the key. Let entity type S1 with attributes {A11, A12, …, A1k} and entity type S2 with attributes {A21, A22, …, A2m} be the subclasses of S. Since S1 and S2 are subclasses of S, there are no keys in S1 and S2. At this point, S is mapped into the relational schema {K, A1, A2, …, An}, and S1 and S2 are mapped into schemas {K, A11, A12, …, A1k} and {K, A21, A22, …, A2m}, respectively. Figure 4.4 shows the transformation of specialization. 4.5 Transformation of categorizations The categorization in the EER model is concerned with the issue of selective inheritance. Essentially, the categorization shows the uncertainty that which entity in the categorization will take place in the schema is unknown currently. The entities, fuzzy or not, in the categorization can respectively be mapped into the relations following the methods given above. The categorization entity also follows the same transformation. Some additional attributes, however, should be added into the corresponding relation. These attributes are a set of all attributes of the entities in the categorization. Engine
Number d
Number
Plane
Model
Plane Engine Name Number Usage
Car Engine
Rate
Usage Name
Engine Model
Number
Car Engine Designer
Rate
Designer
Fig. 4.4 Transformation of Specialization 4.6 Transformation of aggregations Each aggregation in the fuzzy EER model can be mapped into a relation of the fuzzy nested relational schema with relation-valued attributes. Depending on the
142
Zongmin M. Ma
component entities, the aggregation entity may be crisp or fuzzy. As we know, there are four kinds of entities in the fuzzy EER model. The fuzziness of the component entities only on attribute values does not influence the relation mapped from the aggregation entity. If there is the fuzziness of the component entities at instance/schema level, namely, the fuzziness at the second level, however, an additional attribute must be added to the relation mapped from the aggregation entity, indicating the aggregation degree of the tuples. The fuzziness of the component entities at schema level, i.e., the fuzziness at the first level, however, cannot be modeled in the relation mapped from the aggregation entity. Figure 4.5 shows the transformations of aggregation.
5 Conclusion The incorporation of imprecise and uncertain information in database model has been an important topic of database research because such information extensively exists in data and knowledge intensive applications such as expert system, decision-making, and CAD/CAM etc. Besides, that there are complex object structures is another characteristic of these systems. The classical relational database model and its extension of imprecision and uncertainty do not satisfy the need of modeling complex objects with imprecise and uncertain information. In this chapter, we have presented the fuzzy extended entity-relationship (FEER) model to cope with imperfect as well as complex objects in the real world at a conceptual level. Also a nested relational database model has completely been extended for modeling the extended possibility-based fuzzy data. In particular, we have provided the formal approach to mapping a fuzzy EER model to a fuzzy nested relational database schema.
Acknowledgements The authors wish to thank the anonymous referees for their valuable comments and suggestions, which improved the technical content and the presentation of the paper. Work supported in part by the MOE Funds for Doctoral Programs (20050145024).
Modelling Fuzzy Information in the EER
143
Car
CarID
ChassisID
Model
Name
×
Chassis
EngineID
Size
Engine
Car Chassis
CarID
ChassisID
Engine
Model
EngineID
Size
Name
pD
Fig. 4.5 Transformation of Aggregation
References [1] Abiteboul, S. and Hull, R. (1987) IFO: A Formal Semantic Database
Model, ACM Transactions on Database Systems, 12(4): 525-565. [2] Bordogna, G., Pasi, G. and Lucarella, D. (1999) A Fuzzy Object-Oriented
[3]
[4]
[5] [6]
[7]
Data Model for Managing Vague and Uncertain Information, International Journal of Intelligent Systems, 14: 623-651. Bordogna, G. and Pasi, G. (2001) Graph-Based Interaction in a Fuzzy Object Oriented Database, International Journal of Intelligent Systems, 16(7): 821-841. Bosc, P. and Prade, H. (1997) An Introduction to Fuzzy Set and Possibility Theory-based Approaches to the Treatment of Uncertainty and Imprecision in Data Base Management Systems. In Uncertainty Management in Information Systems: from Needs to Solutions (A. Motro and P. Smets, (Eds), pp. 285-324, Kluwer Academic Publishers. Bosc, P. and Pivert, O. (1995) SQLf: A Relational Database Language for Fuzzy Querying, IEEE Transactions on Fuzzy Systems, 3(1): 1-17. Bosc, P. and Pivert, O. (2003) On the Impact of Regular Functional Dependencies When Moving to a Possibilistic Database Framework, Fuzzy Sets and Systems, 140(1): 207-227. Bosc, P., Dubois, D. and Prade, H. (1998) Fuzzy Functional Dependencies and Redundancy Elimination, Journal of the American Society for Information Science, 49(3): 217-235.
144
Zongmin M. Ma
[8] Buckles, B. P. and Petry, F. E. (1982) A Fuzzy Representation of Data for [9]
[10]
[11] [12]
[13] [14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
Relational Database, Fuzzy Sets and Systems, 7(3): 213-226. Chaudhry, N. A., Moyne, J. R. and Rundensteiner, E. A. (1999) An Extended Database Design Methodology for Uncertain Data Management, Information Sciences, 121(1-2): 83-112. Chen, G. Q. and Kerre, E. E. (1998) Extending ER/EER Concepts towards Fuzzy Conceptual Data Modeling, In: Proc. of the 1998 IEEE International Conference on Fuzzy Systems, 2: 1320-1325. Chen, P. P. (1976) The Entity-Relationship Model: Toward a Unified View of Data, ACM Transactions on Database Systems, 1(1) 9-36. Cross, V., Caluwe, R. and Vangyseghem, N. (1997) A Perspective from the Fuzzy Object Data Management Group (FODMG), In: Proc. of the 1997 IEEE International Conference on Fuzzy Systems, 2: 721-728. Cross, V. and Firat, A. (2000) Fuzzy Objects for Geographical Information Systems, Fuzzy Sets and Systems, 113: 19-36. De Tré, G. and De Caluwe, R. (2003) Level-2 Fuzzy Sets and Their Usefulness in Object-Oriented Database Modelling, Fuzzy Sets and Systems, 140(1): 29-49. Dubois, D., Prade, H. and Rossazza, J. P. (1991) Vagueness, Typicality, and Uncertainty in Class Hierarchies, International Journal of Intelligent Systems, 6: 167-183. George, R., Srikanth, R., Petry, F. E. and Buckles, B. P. (1996) Uncertainty Management Issues in the Object-Oriented Data Model, IEEE Transactions on Fuzzy Systems, 4(2): 179-192. Gyseghem, N. V. and Caluwe, R. D. (1998) Imprecision and Uncertainty in UFO Database Model, Journal of the American Society for Information Science, 49(3): 236-252. Ma, Z. M., Zhang, W. J. and Ma, W. Y. (2000) Semantic Measure of Fuzzy Data in Extended Possibility-Based Fuzzy Relational Databases, International Journal of Intelligent Systems, 15(8): 705-716. Ma, Z. M., Zhang, W. J., Ma, W. Y. and Chen, G. Q. (2001) Conceptual Design of Fuzzy Object-Oriented Databases Using Extended EntityRelationship Model, International Journal of Intelligent Systems, 16: 697711. Ma, Z. M. and Mili, F. (2002) An Extended Possibility-Based Fuzzy Nested Relational Database Model and Algebra, IFIP International Federation for Information Processing (KLUWER Academic Publishers), 221: 285-288. Ma, Z. M. and Mili, F. (2002) Handling Fuzzy Information in Extended Possibility-Based Fuzzy Relational Databases, International Journal of Intelligent Systems, 17(10): 925-942.
Modelling Fuzzy Information in the EER
145
[22] Marín, N., Pons, O. and Vila, M. A. (2001) A Strategy for Adding Fuzzy
[23] [24]
[25]
[26]
[27]
[28]
[29]
[30]
[31]
[32]
[33]
[34] [35] [36]
Types to an Object-Oriented Database System, International Journal of Intelligent Systems, 16(7): 863-880. Naiburg, E., (2000) Database Modeling and Design Using Rational Rose, http://www.therationaledge.com/rosearchitect/mag/current/spring00/f5.html. Poncelet, P., Teisseire, M., Cicchetti, R. and Lakhal, L. (1993) Towards a Formal Approach for Object Database Design, In: Proc. of the 19th Very Large Data Bases Conference, 278-289. Prade, H. and Testemale, C. (1984) Generalizing Database Relational Algebra for the Treatment of Incomplete or Uncertain Information and Vague Queries, Information Sciences, 34: 115-143. Raju, K. V. S. V. N. and Majumdar, A. K. (1988) Fuzzy Functional Dependencies and Lossless Join Decomposition of Fuzzy Relational Database System, ACM Transactions on Database Systems, 13(2) 129-166. Sözat, M. I. and Yazici, A. (2001) A Complete Axiomatization for Fuzzy Functional and Multivalued Dependencies in Fuzzy Database Relations, Fuzzy Sets and Systems, 117(2): 161-181. Takahashi, Y. (1993) Fuzzy Database Query Languages and Their Relational Completeness Theorem, IEEE Transactions on Knowledge and Data Engineering, 5(1): 122-125. Teorey, T. J., Yang, D. Q. and Fry, J. P. (1986) A Logical Design Methodology for Relational Databases Using the Extended EntityRelationship Model, ACM Computing Surveys, 18 (2): 197-222. Umano, M. and Fukami, S. (1994) Fuzzy Relational Algebra for Possibility-Distribution-Fuzzy-Relational Model of Fuzzy Data, Journal of Intelligent Information Systems, 3: 7-27. Vila, M. A., Cubero, J. C., Medina, J. M. and Pons, O. (1996) A Conceptual Approach for Deal with Imprecision and Uncertainty in Object-Based Data Models, International Journal of Intelligent Systems, 11: 791-806. Yazici, A., Buckles, B. P. and Petry, F. E. (1999) Handling Complex and Uncertain Information in the ExIFO and NF2 Data Models, IEEE Transactions on Fuzzy Systems, 7(6): 659–676. Yazici, A., Soysal, A., Buckles, B. P. and Petry, F. E. (1999) Uncertainty in a Nested Relational Database Model, Data & Knowledge Engineering, 30: 275–301. Zadeh, L. A. (1965) Fuzzy Sets, Information and Control, 8(3): 338-353. Zadeh, L. A. (1978) Fuzzy Sets as a Basis for a Theory of Possibility, Fuzzy Sets and Systems, 1(1): 3-28. Zemankova M. and Kandel A. (1985) Implementing Imprecision in Information Systems, Information Sciences, 37(1-3): 107-141.
146
Zongmin M. Ma
[37] Zicari, R. and Milano, P. (1990) Incomplete Information in Object-Oriented
Databases, ACM SIGMOD Record, 19(3): 5-16. [38] Zvieli, A. and Chen, P. P. (1986) Entity-Relationship Modeling and Fuzzy
Databases, In: Proc. of the 1986 IEEE International Conference on Data Engineering, 320-327.
Querying Contradictory Databases by Taking into Account Their Reliability and Their Number Laurence Cholvy ONERA Centre de Toulouse, 2 av Ed. Belin,31055 Toulouse, France [email protected]
1 Introduction Databases integration, federated databases, multidatabases, databases merging ([3], [1], [2], [17], [5], [12], [14], [10], [4]) are close problems which aim to query several independent databases viewed as a single one. A first difficulty is to define a global schema which provides the user with a unified view of the different databases and mappings between this global schema and the different local schemas. This problem is intensively studied in the Database Community and leads to two main paradigms: the “Localas-view” paradigm (which describes each data source as a view of the global schema) and the “Global-as-view” paradigm (which defines global relations as views of the local ones). See for instance [10] and [4] for more details. This present work assumes that this first difficulty has been solved yet and that the different databases share a common data description language, i.e. a common set of relations symbols. The remaining difficulty is that the information stored in the different databases may be contradictory, specially in the presence of some integrity constraints (key constraints, foreign key constraints etc.) that may be common to the different databases or not. The point is then to provide consistent answers to queries. In order to illustrate that point, consider a user facing three databases which store information about flights taking off from Paris Orly airport. Assume that R denotes a relation which stores, for each flight, its number, the name of the company which manages it and the airport terminal it takes off from. The first database stores the facts: R(1, AF, Orly-West), R(2, TWA, OrlyWest) , the second one stores R(1, AF, Orly-South), R(3, AF, Orly-South) and the last one stores R(1, AF, Orly-West), R(2, TWA, Orly-South). These facts respectively mean that flight number 1 is an Air-France flight which takes L. Cholvy: Querying Contradictory Databases by Taking into Account Their Reliability and Their Number, Studfuzz 203, 149–168 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
150
Laurence Cholvy
off from Orly West terminal, flight number 2 is a TWA flight which takes off from Orly West terminal, flight number 1 is an Air-France flight which takes off from Orly South terminal, etc. What is the answer that can be provided to the user if she wants to know which terminal flight number 1 takes off from ? There is not a single way to answer this query. In fact, depending on the information one has about the databases, different queries can be provided. Firstly, we can adopt a cautious attitude which consists in making the disjunction of all the contradictory answers. In the example, that attitude would lead to answer that Flight number 1 takes off from Orly-West terminal or from Orly-South terminal. But this way of answering is questionable since it introduces disjunctions in the answers even if the initial data are not disjunctive. If we know the relative reliability of the databases, then this information can be taken into account (with an ordinal method or a cardinal one) for providing complete answers ([15], [11], [5], [16]...). For instance if we know that the first database is managed by Paris-Orly airport while the others are not, it can be assumed as the most reliable and we can answer that Flight number 1 takes off from Orly-West terminal. A more precise method consists in considering that databases reliability depends on the topics of the data ( [6]). For instance, if we know that the second database is managed by Air-France company, then we can assume that, as far as Air France flight are concerned, it is the most reliable and we can answer that Flight number 1 takes off from Orly-South terminal. When the reliability of the databases is not known or when all the databases are equally reliable, an alternative is to provide an answer which agrees with the biggest number of databases ([9], [13], [7]). In the previous example, it comes to answer that Flight number 1 takes off from Orly-West terminal. Finally, a last method for answering queries is a combination of the previous ones and consists in taking into account the number of the databases which support a piece of information, and their reliability too. More specifically, in such a method, stating that a fact is true requires comparing the reliability degrees of the different databases which support that fact and the reliability degrees of the different databases which support its negation. In this present work, we focus on such a method in which reliability degrees of the databases are integers so that the biggest an integer, the higher the database reliability. We define a query evaluator for consistently answering queries by taking into account the number of the databases that support the answer and their reliability degrees as well. This chapter is organized as follows. Section 2 defines a query evaluator for consistently answering queries in the case of propositional databases. We prove that this query evaluator corresponds to a weighted majority merging operator. Section 3 extends the query evaluator to first order databases. Two
Querying Contradictory Databases
151
examples are given in section 4. We show in section 5 that this present work generalizes previous ones. Finally, section 6 is devoted to a discussion.
2 A query-evaluator in the propositional setting 2.1 Assumptions and vocabulary We consider a propositional language L, and several databases db1 , . . . , dbn called primitive databases whose contents are finite and consistent sets of literals1 of L. If db is a primitive database and if α is a positive integer, then α.db denotes a set called weighted database. (The binary operation . will formally be defined in the following sections). If db1 and db2 are two databases which are not primitive then (db1 ∗ db2 ) denotes a set called a merged database. (The binary operation ∗ will formally be defined in the following sections). For instance, if we face three primitive database db1 , db2 and db3 then 3.db1 denotes a weighted database, (2.db1 ∗1.db2 ), and (3.db1 ∗(1.db2 ∗2.db3 )) denote merged database. Intuitively speaking, the notation (3.db1 ∗(1.db2 ∗2.db3 )) will be used to represent the fact that queries are addressed to the three databases, db1 , db2 and db3 , assuming that their respective reliability degrees are 3, 1 and 2. The query evaluator for answering queries is described as a set of formulas of a logical language, named M L and defined in the following. 2.2 The logical language The meta-language M L defined by: • constants symbols of M L are propositional atoms of L, names of databases, plus a constant symbol denoted nil and constant symbols denoting integers: 1, 2, etc. • a binary function, ., which will be used to denote the weighted databases. • a binary function, ∗, which will be used to denote the merged databases. • a binary function denoted + which is the sum of integers. • the binary meta-predicate symbols are: In, B, neg, = and >. • A ternary meta-predicate symbol is Occur. The intuitive semantics of the predicates is the following: - neg(l, l ) is true if l and l are two atoms of L and l is the negation of l. - In(l, db) is true if l is a literal of L, bd is a primitive database and l belongs to db. 1
A literal is a positive or negative atom.
152
Laurence Cholvy
- Occur(db, l, i) is true if l is a literal of L, i is an integer, db denotes a database (primitive, weighted or merged) and l appears i times in db. - B(db, l) is true if l is a literal of L, db is a database (primitive, weighted or merged) and we can conclude that l is true from db. 2.3 The program If we consider n databases db1 ..dbn , let us call M ET A(db1 , ..., dbn ), be the following set of the M L formulas in which db, db denote primitive, weighted or merged databases, l, l denote literals of L, α, i, j, k, k denote integers. (1) In(l, db) if db is a primitive database and l belongs to db (2) Occur(db, l, k) ∧ k = α.k → Occur(α.db, l, k ) (3) ¬(db = nil) ∧ Occur(db, l, i) ∧ Occur(db , l, j) ∧ (k = i + j) → Occur(db ∗ db , l, k) (4) In(l, db) → Occur(db ∗ nil, l, 1) (5) ¬In(l, db) → Occur(db ∗ nil, l, 0) (6) Occur(db, l, i) ∧ Occur(db, l , j) ∧ neg(l, l ) ∧ (i > j) → B(db, l) (7) neg(l, l ) where l and l are complementary literals of L. (8) k = (r + l) and (r + l) = k for any k in {1...n} for any r in {1...k} and for any l such that k = l + r (9) k = α.k for any integers α ∈ {1...n}, k ∈ {1...n}, and k ∈ {1...n2 } so that k = α.k (10) k > r for any k in {1...n} and for any r in {1...k − 1} such that k > r Axioms of type (1) are used to list the contents of the different primitive databases. According to axiom (2), the number of occurrence of a literal l in the weighted database α.db is α times the number of occurrence of l in db. According to axiom (3), the number of occurrence of a literal l in the merged database db1 ∗ db is the sum of the number of occurrences of l in db1 and the number of occurrences of l in db . Axiom (4) states that if a literal l belongs to the primitive database db then the number of occurrence of l in db is 1. Axiom (5) states that if a literal l does not belong to the primitive database db then the number of occurrence of l in db is 0. Axiom (6) states that we can conclude l from a database db if the number of occurrences of l in db is strictly greater than the number of occurrences of its negation in db. (7) relate any literal of L with its negation. Finally, (8), (9) and (10) define, in extension, the sum of integers, product and comparison of integers needed in the previous axioms. Notice that there is a finite number of them.
Querying Contradictory Databases
153
2.4 Relation with a weighted majority merging operator In [9], Konieczny and Pino-P´erez introduced a majority merging operator as follows2 Let db1 ...dbn be n information sources to be merged. A majority merging operator, denoted ∆Σ , is defined such that the models of the information source which is obtained from merging db1 ... dbn with this operator, is semantically characterized by: M od(∆Σ ([db1 , ..., dbn ])) =
min
≤Σ [db
(W)
1 ...dbn ]
where W denotes the set of all the interpretations of the language L (the propositional language used to describe the contents of the information sources). ≤Σ [db1 ...dbn ] is a total pre-order on W defined by: w ≤Σ [db1 ...dbn ] w iff dΣ (w, [db1 ...dbn ]) ≤ dΣ (w , [db1 ...dbn ])
with dΣ (w, [db1 ...dbn ]) =
n
i=1
min
w ∈M od(dbi )
d(w, w )
where M od(dbi ) is the set models of dbi and d(w, w ) is the Hamming distance. In other words, when merging db1 ...dbn with the operator ∆Σ , the result is semantically characterized by the interpretations which are minimal according to the pre-order ≤Σ [db1 ,...,dbn ] . Now, let us consider that the knowledge-bases db1 ...dbn are associated with weights α1 ...αn which are integers. We can extend the previous definitions and 1 ...αn , such that the models of the informadefine a new merging operator, ∆α Σ tion source which is obtained from merging db1 ... dbn with this operator, is semantically characterized by: 1 ...αn ([db1 , ..., dbn ])) = M od(∆α Σ
min
Σ,α1 ...αn 1 ...dbn ]
(W)
≤[db
1 ...αn where ≤Σ,α [db1 ...dbn ] is a total pre-order on W defined by:
1 ...αn w ≤Σ,α [db1 ...dbn ] w iff dΣ,α1 ...αn (w, [db1 ...dbn ]) ≤ dΣ,α1 ...αn (w , [db1 ...dbn ])
with dΣ,α1 ...αn (w, [db1 ...dbn ]) =
n
i=1
2
αi .
min
w ∈M od(dbi )
d(w, w )
One will notice that we slightly change the presentations of these definitions to remain coherent with what has already been presented.
154
Laurence Cholvy
1 ...αn In other words, when merging db1 ...dbn with the operator ∆α , the Σ result is semantically characterized by the interpretations which are minimal α1 ...αn 1 ...αn is thus a weighted majority according to the pre-order ≤Σ,α [db1 ,...,dbn ] . ∆Σ merging operator. The following proposition proves that the query evaluator defined in section 2.3 corresponds to such a merging operator.
Proposition 1. Let db1 ...dbn be primitive databases. Let α1 ....αn be integers and l be a literal of L. Then, PROLOG proves B(α1 .db1 ∗ (... ∗ (αn .dbn ∗ nil)), l) in the meta-program M ET A(db1 , ..., dbn ) 1 ...αn ([db1 ...dbn ]) |= l iff ∆α Σ Proof. Notation In the following, [α1 .db1 ...αn .dbn] will denote the multi-set: [db1 ...(α1 times)...db1 , ...dbn ...(αn times)...dbn ]. (Only if ) Assume that PROLOG proves B(α1 .db1 ∗ (... ∗ (αn .dbn ∗ nil)), l) in the meta-program M ET A(db1 , ..., dbn ). 1 ...αn ([db1 ...dbn ])) such that w1 |= l. Assume that ∃w1 ∈ M od(∆α Σ Let us then consider the interpretation w2 which is identical to w1 except that the valuation of l in w2 is true, i.e, w2 |= l. Since PROLOG proves B(α1 .db1 ∗ (... ∗ (αn .dbn ∗ nil)), l) in the metaprogram M ET A(db1 , ..., dbn ), this implies (axiom (6)) that PROLOG also proves Occur(α1 .db1 ∗ (... ∗ (αn .dbn ∗ nil)), l, i), neg(l, l ), Occur(α1 .db1 ∗ ... ∗ (αn .dbn ∗ nil), l , j) and i > j. I.e, that in α1 .db1 ∗ (... ∗ (αn .dbn ∗ nil)), the number of occurrences of l is strictly greater that the number of occurrences of its negation. Thus, by definition of axioms (1)...(5), this means that, in the multi-set [α1 .db1 ...αn .dbn], the number of occurrences of l is strictly greater that the number of occurrences of its negation. Thus, dsum (w2 , [α1 .db1 ...αn .dbn]) < dsum (w1 , [α1 .db1 ...αn .dbn]) Thus, finally, dΣ,α1 ...αn (w2 , [db1 ...dbn ]) < dΣ,α1 ...αn (w1 , [db1 ...dbn ]). 1 ...αn ([db1 ...dbn ])) and thus leads to a conThis shows that w1 ∈ M od(∆α Σ tradiction. 1 ...αn ([db1 ...dbn ])) =⇒ w |= l. Finally, we conclude that ∀w w ∈ M od(∆α Σ (If ) 1 ...αn ([db1 ...dbn ]) |= l. Let us assume that ∆α Σ This implies that, in [α1 .db1 ...αn .dbn], the number of occurrences of l is strictly greater that the number of occurrence of its negation. Thus, by definition of axioms (1)...(5), this implies that, in α1 .db1 ∗ ... ∗ (αn .dbn ∗nil), the number of occurrences of l is strictly greater that the number of occurrences of its negation. Thus, by axiom 6, this implies that PROLOG proves B(α1 .db1 ∗ (... ∗ (αn .dbn ∗ nil)), l) in the meta-program M ET A(db1 , ..., dbn ). (End of proof )
Querying Contradictory Databases
155
2.5 Other properties The following two propositions show that when answering a query addressed to several databases, these databases have symmetrical roles i.e querying several databases is equivalent to querying any permutation of them. Proposition 2. Let db1 ...dbn be primitive databases. Let α1 ....αn be integers and l be a literal of L. In M ET A(db1 ...dbn ), PROLOG proves B(α1 .db1 ∗ (α2 .db2 ∗ nil), l) iff it also prove B(α2 .db2 ∗ (α1 .db1 ∗ nil), l). Proof This proposition is a corollary of proposition 1. Proposition 3. Let db1 ...dbn be primitive databases. Let α1 ....αn be integers and l be a literal of L. In M ET A(db1 ...dbn ), PROLOG proves B(α1 .db1 ∗ (α2 .db2 ∗(α3 .db3 ∗nil)), l) iff it also proves B(α1 .db1 ∗(α2 .db2 ∗(α3 .db3 ∗nil))), l). Proof This proposition is a corollary of proposition 1. Due to these results, parenthesis can be omitted when designating the different databases we query. And, for instance, 2.db1 ∗ 1.db2 ∗ 2.db3 will denote (2.db1 ∗(1.db2 ∗(2.db3 ∗nil)))) as well as (2.db3 ∗(2.db1 ∗(1.db2 ∗nil)))) Proposition 4. Let db1 ...dbn be primitive databases. Let α1 ....αn be integers and l be a literal of L. The three following cases are exclusive: (1) In M ET A(db1 ...dbn ), PROLOG proves B((α1 .db1 ∗ . . . ∗ αn .dbn ), l) (2) In M ET A(db1 ...dbn ), PROLOG proves B((α1 .db1 ∗. . .∗αn .dbn ), l ) so that it also proves neg(l, l ). (3) In M ET A(db1 ...dbn ), PROLOG don’t prove B((α1 .db1 ∗ . . . ∗ αn .dbn ), l) nor B((α1 .db1 ∗ . . . ∗ αn .dbn ), l ) so that it also proves neg(l, l ). Proof This proposition is a corollary of proposition 1.
3 Application to first-order databases In this section we extend the previous program for specifying a query evaluator which consistently answers queries addressed to several first order databases which can be contradictory. But, in order to export these results to first order databases, we only consider databases which are “equivalent to sets of ground literals” ([5]). Such databases are defined below. 3.1 Databases equivalent to sets of ground literals Let LO be a first order language. Definition 1 . A database is a pair DB =< EDB, IDB >3 such that EDB is a non empty and finite set of positive or negative ground literals of 3
“EDB” stands for “extensional database” and “IDB” stands for “intensional database”.
156
Laurence Cholvy
LO and IDB is a finite and consistent set of clauses of LO written without function symbol. Notice that literals in EDB can be positive or negative. Definition 2. Let DB =< EDB, IDB > a database. Let a1 ...an (resp. P1 ...Pk ) be the constant (resp. predicate) symbols which appear in the formulas of EDB ∪ IDB. The Herbrand base is the set of positive literals written with the Pi and the aj . A Herbrand interpretation of DB is an interpretation whose domain is {a1 , ..., an }. A Herbrand model of DB is a Herbrand interpretation which satisfies EDB ∪ IDB. Definition 3. Let HM1 ...HMn be the Herbrand models of EDB ∪ IDB. Let L = {l : l is a literal of the Herbrand base such that ∃ HMi ∃ HMj HMi |= l and HMj |= ¬l}. The database DB =< EDB, IDB > is equivalent to a set of ground literals iff for any satisfiable conjunction l1 ∧ ... ∧ lm where ∀i ∈ {1...m} such that li ∈ L or ¬li ∈ L, there exists HMi0 such that HMi0 |= l1 ∧ ... ∧ lm . Example. Consider DB1 =< EDB1 , IDB1 > with EDB1 = {p(a)} and IDB1 = {¬p(x) ∨ ¬q(x), p(x) ∨ r(x)}. The Herbrand models of DB1 are4 : {p(a)} and {p(a), r(a)}. We have: L = {r(a)} We can check that ¬r(a) is satisfied in the first Herbrand model and that r(a) is satisfied in the second. So DB1 is equivalent to a set of ground literals. Consider now DB2 =< EDB2 , IDB2 > with EDB2 = {p(a)} and IDB2 = {¬p(x) ∨ q(x) ∨ r(x)}. The Herbrand models of DB2 are {p(a), q(a)}, {p(a), r(a)} and {p(a), q(a), r(a)}. We have L = {r(a), q(a)}. We can check that none of the Herbrand models satisfy ¬q(a) ∧ ¬r(a). Thus DB2 is not equivalent to a set of ground literals. In [5] we have proved the following result: Proposition 5. Let DB =< EDB, IDB > a database which is equivalent to a set of ground literals. Let l1 ...ln be some ground literals of LO such that l1 ∨ ... ∨ ln is not a tautology. Then, EDB ∪ IDB |= l1 ∨ ... ∨ ln iff ∃i0 ∈ {1...n} EDB ∪ IDB |= li0 This result ensures that, in a database equivalent to a set of ground literals, a disjunction of ground literals which is not a tautology is deducible from the database iff one of these literals is deducible from the database. This implies that there is no real disjunctive data deducible from these databases. 3.2 Taking IDB into account In this section, following [5], we extend M ET A(db1 ...dbn ) in order to take the clauses of IDB into account. Let us denote by h the function which associates any clause of IDB with a set of formulas in the following way: h(l1 ∨ ... ∨ ln ) = {(¬l1 ∧ ...¬li−1 ∧ ¬li+1 ... ∧ ¬ln ) → li , i ∈ {1, ...n}} Then, the axiom (1) of M ET A(db1 ...dbn ) is replaced by the following ones: 4
A model is denoted here by the set of its positive facts.
Querying Contradictory Databases
157
(1.1) EDB(db, l) if the ground literal l is in the EDB part of the database db. (1.2) IDB(db, f ) if f ∈ h(c), where c is a clause in the IDB part of db. (1.3) EDB(db, l) → In(l, db) (1.4) IDB(db, (r → l)) ∧ Bconj (db, r) → In(l, db) (1.5) Bconj (db, nil) (1.6) In(l1 , db) ∧ Bconj (db, r1 ) → Bconj (db, l1 ∧ r1 ) We can prove : Proposition 6. Let db =< EDB, IDB > be such that IDB is not recursive. Let l be a ground literal. Then PROLOG proves In(l, db) iff EDB ∪ IDB |= l. (The proof is similar to the one in [5]) This result ensures that, if IDB is not recursive, axiom (1) can be replaced by axioms (1.1)...(1.6). Thus, if IDB is not recursive, the meta-program defined for databases which are sets of propositional literals can be used in the case of first order databases which are equivalent to sets of ground literals. 3.3 Definition of answers Definition 4 (Answers for closed queries). Let db1 ...dbn be n databases, each of them being equivalent to a set of literals. Let α1 ...αn be integers. Let F be a ground literal. The answer to the query “Can it be concluded that F true from the databases db1 ...dbn when assuming that their respective degrees of reliability are α1 ...αn ?” is defined by: answer((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) = Y ES iff PROLOG proves B((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) in M ET A(db1 ...dbn ) answer((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) = N O iff PROLOG proves neg(F, F ) and B((α1 .db1 ∗ . . . ∗ αn .dbn , F ) in M ET A(db1 ...dbn ) answer((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) =?
else
Notice that we are in an open world approach: primitive databases are not necessarily complete. This explains the last case. This corresponds to the case when the two goals B((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) and B((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) finitely fail in PROLOG. Definition 5. (Answers for open queries). Let db1 ...dbn be n databases, α1 ...αn be integers and F (X) be an open literal. Consider the query “What are the X which satisfy F ” when it addressed to databases db1 ...dbn whose respective degrees of reliability are α1 ...αn ? . Its answer is defined by: answer((α1 .db1 ∗ ... ∗ αn .dbn ), F (X)) = {A : tuple of constant symbols such that PROLOG proves B((α1 .db1 ∗ ... ∗ αn .dbn ), F (A))}
158
Laurence Cholvy
3.4 Properties Following these definitions and according to proposition 4, we can ensure that the query evaluator generates consistent answers to any query addressed to several databases even if they are contradictory. Proposition 7 Let db1 ...dbn be n databases, each of them being equivalent to a set of literals. Let α1 ...αn be integers and F be a ground literal which represents a closed query. Then the three following cases are exclusive: (1) answer((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) = Y ES (2) answer((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) = N O (3) answer((α1 .db1 ∗ . . . ∗ αn .dbn ), F ) =? Proof This is a consequence of proposition 4. Proposition 8 Let db1 ...dbn be n databases, each of them being equivalent to a set of literals. Let α1 ...αn be integers and let F (X) be a literal which represents an open query. Let A be a tuple of constants, then the two following cases are exclusive : (1) A ∈ answer((α1 .db1 ∗ . . . ∗ αn .dbn ), F (X)) (1) A ∈ answer((α1 .db1 ∗ . . . ∗ αn .dbn ), ¬F (X)) Proof This is a consequence of proposition 4. These two propositions show that the query-evaluator defined in this work, consistently answers queries, even if the databases are contradictory.
4 Illustrative examples This section illustrates the query-evaluator defined previously on two examples. The two examples differ on the fact that, in the first one, the databases share a common set of rules while it is not the case in the second one. 4.1 The databases share a common set of rules We consider three databases which store information about flights taking off from Paris Orly airport. The first database, Agency, is managed by a travel agency, the second, Orly, is managed by Paris Orly airport, and the third, AF , one by Air-France company. Information we focus on are flight identification numbers, flight destination, departure time, departure terminal and company. Let these databases be denoted Agency =< EDBAgency , IDB > Orly =< EDBOrly , IDB > Air F rance =< EDBAF , IDB > with:
Querying Contradictory Databases
159
EDBAgency = { f light(1, T oulouse, 0825, AF, national), f light(2, London, 1130, AF, international), f light(3, London, 1430, BAW, international), f light(4, N ewY ork, 1945, T W A, international), f light(5, W ashington, 2200, T W A, international)} EDBOrly = { terminal(1, OrlyW ), terminal(2, OrlyW ), terminal(3, OrlyW ), terminal(4, OrlyS), terminal(5, OrlyW )} EDBAF = { f light(1, T oulouse, 0825, AF, national), f light(2, London, 1130, AF, international), terminal(1, OrlyW ), terminal(2, OrlyS)} IDB = { ∀x∀y∀z f light(x, y, z, AF, national) → terminal(x, OrlyW ), ∀x∀y∀zf light(x, y, z, AF, international) → terminal(x, OrlyS), ∀x∀y∀z∀t f light(x, y, z, BAW, t) → terminal(x, OrlyS), ∀x∀y∀z∀t f light(x, y, z, T W A, t) → terminal(x, OrlyS), ∀x¬terminal(x, OrlyS) ∨ ¬terminal(x, OrlyW ) Thus, according to the Agency database, flight number 1 is a national AirFrance flight to Toulouse and its departure time is 8.25; flight number 2 is an international Air-France flight to London and its departure time is 11.30; flight number 3 is an international British AirWay international flight to London and its departure time is 14.30; flight number 4 is a TWA international flight to New York and its departure time is 19.45; flight number 5 is a TWA international flight to Washington and its departure time is 22.00. According to Orly database, departure terminal of flights number 1, 2, 3 and 5 is Orly-West, departure terminal of flight number 4 is Orly-South. According to AF database, flight number 1 is a national AirFrance flight to Toulouse, its departure time is 8.25 am and its departure terminal is OrlyWest; flight number 2 is a international AirFrance flight to London, its departure time is 11.30 am and its departure terminal is Orly-South. Finally, it is known that Orly-West is the departure terminal of national Air-France flights; Orly South is the departure terminal of international AirFrance flights, British AirWays flights and TWA flights. And obviously, OrlySouth and Orly-West are two different terminals. Here are some queries and the answers generated by the evaluator • When querying the three databases and assuming that they are equally reliable, is it true that flight number 2 starts at 11.30 am to London and is an international flight ? answer((1.Agency ∗1.Orly ∗1.AF ), f light(2, London, 1130, AF, international)) = Y ES
Indeed, on the first hand, two databases (Agency and AF ) support the
160
Laurence Cholvy
fact that flight number 2 starts at 11.30 am to London and is an international flight. On the second hand, according to the Orly database, flight number 2 starts from Orly West. Thus, with the rules, Orly database supports the fact that flight number 2 is not an international flight. Finally, since the three databases are supposed to be equally reliable, this means that there are two supports (or two proofs) to the fact that flight number 2 starts at 11.30 am to London and is an international flight and only one support to the contrary. By majority, we conclude that flight number 2 starts at 11.30 am to London and is an international flight. • When querying the three databases and assuming that their respective reliability degrees are 1, 3 and 1, is it true that flight number 2 starts at 11.30 am to London and is an international flight ? answer((1.Agency ∗3.Orly ∗1.AF ), f light(2, London, 1130, AF, international)) = NO
Assuming that the reliability degrees of the three databases are 1, 3 and 1 still implies that there are two supports to the fact that flight number 2 starts at 11.30 am to London and is an international flight. But this also implies that there are three supports to the contrary. Thus, by majority, we conclude that it is not true that flight number 2 starts at 11.30 am to London and is an international flight. • When querying the three databases and assuming that their respective reliability degrees are 1, 3 and 2, is it true that flight number 2 starts at 11.30 am to London and is an international flight ? answer((1.Agency∗3.Orly∗2.AF ), f light(2, london, 1130, AF, international)) = ?
Here, assuming that the reliability degrees of the three databases are 1, 3 and 2 implies that there are three supports to the fact that flight number 2 starts at 11.30 am to London and is an international flight but also implies that there three supports to the contrary. Thus, we cannot decide whether the fact is true or not. • When querying the three databases and assuming that they are equally reliable, what are the number, the destination, the departure time and the company of international flights ? answer((1.Agency ∗ 1.Orly ∗ 1.AF ), f light(x, y, z, t, international)) = {(2 LONDON 1130 AF), (4 NEWYORK 1945 TWA) }
The reasons why flight number 2 belongs to the answer are given previously (see the first query). As for flight number 4, it belongs to the answer because there is no opposition to that. Finally, flight number 3 (resp 5) does not belong to the answer because, according to Agency database, there is one support to the fact that it is an international flight,
Querying Contradictory Databases
161
and according to Orly database (and the rules) there is also one support to the contrary. • When querying the three databases and assuming that their respective reliability degrees are 1, 3 and 1, what are the number, the destination, the departure time and the company of international flights ? answer((1.Agency ∗ 3.Orly ∗ 1.AF ), f light(x, y, z, t, international)) = {(4 NEWYORK 1945 TWA)}
Flight number 2 does not belong to the answer any longer because we can conclude that it is not true that it is an international flight (see the second query). • When querying the three databases and assuming that their respective reliability degrees are 1, 3 and 2, what are the number, the destination, the departure time and the company of international flights ? answer((1.Agency ∗ 3.Orly ∗ 2.AF ), f light(x, y, z, t, international)) = {(4 NEWYORK 1945 TWA)}
Flight number 2 does not belong to the answer because we cannot decide whether it is an international flight or not (see the third query). • When querying Agency database and Orly database and assuming that their respective reliability degrees are 1 and 2, what are the number of the flights whose departure terminal is Orly West ? answer((1.Agency ∗ 2.Orly), terminal(x, OrlyW )) = {1, 2, 3, 5}
Flight number 4 does not belong to the answer because since reliability degrees of Agency and Orly are respectively 1 and 2, there is one support to the fact that flight number 2 starts from Orly West and two supports to the contrary. • When querying the three databases and assuming that their respective reliability degrees are 1, 2 and 3, what are the number of the flights whose departure terminal is Orly West? answer((1.Agency ∗ 2.Orly ∗ 3.AF ), terminal(x, OrlyW )) = {1, 3, 5} Flight number 2 does not belong to the answer any longer because now, even there are still two supports to the fact that its departure terminal is Orly West, there are four supports to the contrary.
162
Laurence Cholvy
4.2 The databases don’t share a common set of rules We consider now that the three databases are: Agency EDBAgency , IDBAgency > Orly =< EDBOrly , IDBOrly > Air F rance =< EDBAF , IDBAF >
=<
with : IDBAgency = { ∀x∀y∀z∀t f light(x, y, z, BAW, t) → terminal(x, OrlyS), ∀x∀y∀z∀t f light(x, y, z, T W A, t) → terminal(x, OrlyS), ∀x¬terminal(x, OrlyS) ∨ ¬terminal(x, OrlyW )} IDBOrly = {∀x∀y∀z f light(x, y, z, AF, national) → terminal(x, OrlyW ), ∀x∀y∀z f light(x, y, z, AF, international) → terminal(x, OrlyS), ∀x¬terminal(x, OrlyS) ∨ ¬terminal(x, OrlyW )} IDBAF = {∀x¬terminal(x, OrlyS) ∨ ¬terminal(x, OrlyW )} Notice that now, ∀x¬terminal(x, OrlyS) ∨ ¬terminal(x, OrlyW ) is the only rule the three databases have in common. The queries for which, the answer has changed from an example to the other are: • When querying the three databases and assuming that they are equally reliable, what are the number, the destination, the departure time and the company of international flights ? answer((1.Agency ∗ 1.Orly ∗ 1.AF ), f light(x, y, z, t, international)) = {(2 LONDON 1130 AF), (3 LONDON 1430 BAW),(4 NEWYORK 1945 TWA), (5 WASHINGTON 2200 TWA)}
Flight number 3 and number 5 now belong to the answer because, even if like before, Agency database supports the fact that they are international flights, Orly database now does not support the contrary. • When querying the three databases and assuming that their respective reliability degrees are 1, 3 and 1, what are the number, the destination, the departure time and the company of international flights ? answer((1.Agency ∗ 3.Orly ∗ 1.AF ), f light(x, y, z, t, international)) = {(3 LONDON 1430 BAW), (4 NEWYORK 1945 TWA),(5 WASHINGTON 2200 TWA)}
Flight number 3 and number 5 now belong to the answer (see previous query) • When querying the three databases and assuming that their respective reliability degrees are 1, 3 and 2, what are the number, the destination, the departure time and the company of international flights ?
Querying Contradictory Databases
163
answer((1.Agency ∗ 3.Orly ∗ 2.AF ), f light(x, y, z, t, international)) = {(3 LONDON 1430 BAW), (4 NEWYORK 1945 TWA),(5 WASHINGTON 2200 TWA)}
Flight number 3 and number 5 now belong to the answer (see previous query) • When querying the three databases and assuming that their respective reliability degrees are 1, 2 and 3, what are the number of the flights whose departure terminal is Orly West? answer((1.Agency ∗ 2.Orly ∗ 3.AF ), terminal(x, OrlyW )) = {1, 2, 3, 5} Flight number 2 now belong to the answer because Orly does not provide support to its contrary any longer.
5 Particular cases In this section, we show that the query-evaluator defined in the present chapter is a general case of two other query evaluators that we have defined in previous work. 5.1 Taking into account the relative reliability of the databases In [5], we have defined a query-evaluator for answering queries addressed to several databases by taking into account their relative reliability. Let us denote it here M ET A1(db1 ...dbn ). Axioms were defined so that PROLOG proves B(dbi1 > dbi2 > ... > dbin , l) iff we can conclude l if we consider that database dbi1 is strictly more reliable than database dbi2 ,itself being strictly more reliable than etc... Detailing M ET A1(db1 ...dbin ) is out of the scope of this chapter, however, let us recall that its main axioms were (A1) In(l, db) → B(l, db) for any primitive database db and any literal l which belongs to db. (A2) B(O, l) → B(O > db, l) for any primitive database db and any order O, on some primitive databases but db. (A3) In(l, db) ∧ neg(l, l ) ∧ ¬B(O, l ) → B(O > db, l) Example . Consider db1 = {a, b}, db2 = {a, ¬c}, db3 = {¬a, c}. In M ET A1(db1 ...dbn ), PROLOG proves B(db1 > db2 > db3 , a), B(db1 > db2 > db3 , b) and B(db1 > db2 > db3 , ¬c). This means that, if we consider that db1 is more reliable than db2 , itself more reliable than db3 , then, we can conclude a and b and ¬c.In M ET A1(db1 ...dbn ), PROLOG can also prove, for instance, B(db3 > db2 > db1 , ¬a) B(db2 > db1 > db3 , b) and B(db2 > db1 > db3 , c). This means that, if we consider that db3 is more reliable than db2 , itself more reliable than db1 , then we can conclude ¬a, b and c.
164
Laurence Cholvy
The following proposition ensures that reasoning with total orders between databases, as it is done in M ET A1(db1 ...dbn ) can similarly be made in M ET A(db1 ...dbn ). Proposition 9. Let db1 ...dbn primitive databases and l be a literal. PROLOG proves B(dbi1 > dbi2 > ... > dbin , l) in M ET A1(db1 ...dbn ), iff PROLOG proves B(2n−1 .dbi1 ∗ 2n−2 .dbi2 ∗ ... ∗ 20 .dbin , l) in M ET A(db1 ...dbn ) . Proof First, notice that PROLOG proves B(2n−1 .dbi1 ∗ 2n−2 .dbi2 ∗ ... ∗ 20 .dbin , l) in M ET A(db1 ...dbn ) iff, in [2n−1 .dbi1 , 2n−2 .dbi2 , ...20 .dbin ], the number of occurrences of l is strictly greater than the number of occurrences of its negation. (If ) Assume that PROLOG proves B(dbi1 > dbi2 > ... > dbin , l) in M ET A1(db1 ...dbn ). Then, ∃j l ∈ dbij and ∀k < j ¬l ∈ dbik . Thus, in [2n−1 .dbi1 , 2n−2 .dbi2 , ...20 .dbin ], the number of occurrences of l is greater than S n−ij and the number of occurrences of ¬l is smaller than 2n−ij −1 + ... + 20 . Since S n−i > 2n−i−1 + ... + 20 , this implies that, in [2n−1 .dbi1 , 2n−2 .dbi2 , ...20 .dbin ], the number of occurrences of l is strictly greater than the number of its negation. Thus, PROLOG proves B(2n−1 .dbi1 ∗ 2n−2 .dbi2 ∗ ... ∗ 20 .dbin , l) in M ET A(db1 ...dbn ), (Only if ) Assume that PROLOG proves B(2n−1 .dbi1 ∗ 2n−2 .dbi2 ∗ ... ∗ 20 .dbin , l) in M ET A(db1 ...dbn ), Then, in [2n−1 .dbi1 , 2n−2 .dbi2 , ...20 .dbin ], the number of occurrences of l is strictly greater than the number of occurrences of its negation. In the list dbi1 , dbi2 , ...dbin let us assume that dbij is the first database l belongs to. Then ∀k < j ¬l ∈ dbik (if it was not the case, then in [2n−1 .dbi1 , 2n−2 .dbi2 , ...20 .dbin ], the number of occurrences of l would not be strictly greater than the number of occurrences of its negation. Thus, (A1) and (A2), this implies that, in M ET A1(db1 ...dbn ), PROLOG proves B(dbij , l). And, by axiom (A3), this implies that in M ET A1(db1 ...dbn ), PROLOG proves B(dbi1 > ... > dbij , l). Thus, by axiom (A2), this implies that in M ET A1(db1 ...dbn ), PROLOG proves B(dbi1 > ... > dbij > ...dbin , l). (End of proof ) 5.2 Taking into account the number of databases In [7], we have defined a query-evaluator for answering queries when addressed to several databases by applying a majority vote. Let us denote it here M ET A2(db1 ...dbn ). Its axioms were defined so that PROLOG proves B(db1 ∗ ... ∗ dbn , l) iff we can conclude l when making a majority vote between the databases db1 ... dbn . Detailing this evaluator is out of the scope of this
Querying Contradictory Databases
165
chapter. However, let us recall that its main axioms were: (B1) Occur(db, l, i) ∧ Occur(db , l, j) → Occur(db ∗ db , l, l) where k = i + j (B2) neg(l, l ) ∧ Occur(db, l, i) ∧ Occur(db, l , j) ∧ (i > j) → B(db, l) (B1) states that if in database db, the number of occurrences of l is i and if in database db , the number of occurrences of l is j, then in database db ∗ db the number of occurrences of l is i + j. (B2) states that if in database db, the number of occurrences of l is strictly greater than the number of occurrences of its negation, then we consider that l is true in db. Example . Consider again db1 = {a, b}, db2 = {a, ¬c}, db3 = {¬a, c}. In M ET A2(db1 ...dbn ), PROLOG proves B(db1 ∗ db2 ∗ db3 , a) and B(db1 ∗ db2 ∗ db3 , b) and does not prove B(db1 ∗ db2 ∗ db3 , c) nor B(db1 ∗ db2 ∗ db3 , ¬c). This means that, from a majority vote between the three databases, we can derive a and b but neither c. nor ¬c. The following proposition proves that M ET A2(db1 ...dbn ) is a particular case of M ET A(db1 ...dbn ). Proposition 10. Let db1 ...dbn primitive databases and l be a literal. In M ET A2(db1 ...dbn ), PROLOG proves B(db1 ∗ ... ∗ dbn , l) iff in M ET A(db1 ...dbn ) it proves B(α.db1 ∗ ... ∗ α.dbn , l), where α is any integer. Proof Assume that in M ET A2(db1 ...dbn ), PROLOG proves B(db1 ∗ ... ∗ dbn , l). This means that (by axioms (B2)) in [db1 , ..., dbn ], the number of occurrences of l is strictly greater than the number of its negation. Thus, for any integer α, it is true that in [α.db1 , ..., α.dbn ], the number of occurrences of l is strictly greater than the number of its negation. This means that in M ET A(db1 ...dbn ), PROLOG proves B(α.db1 ∗ ... ∗ α.dbn , l). (End of proof )
6 Concluding remarks This chapter presented a query-evaluator which consistently answers queries addressed to several databases, in the case when: • The different databases share a common description language. • They may have different sets of constraints. • Each database is equivalent to a a set of ground literals. The method underlying this query evaluator consists in taking into account the number of databases that support a fact and their reliability degrees as
166
Laurence Cholvy
well. This means that, when formulating a query, the user specifies which databases she wants to query and the reliability degrees she assumes they have. It has been proved that, given a query addressed to several databases associated with some reliability degrees, the answer computed by the previous evaluator is the same than the one that would be computed by a classical evaluator if that query was addressed to the database obtained by merging the databases according to a weighted majority attitude. However, here, the merging is never computed and the databases contents are unchanged after the querying. This implies that, for instance, the user may address a query to a group of databases with particular degrees of reliability and later on address a query to the same databases with different degrees of reliability or even to different databases. This query evaluator has been specified by a set of logical formulas. It has been implemented in PROLOG and the previous example has been run. Two main extensions have been made to this query evaluator but they are not detailed due to space restriction. First, following [7], the evaluator has been extended in order to answer more complex queries, i.e queries which are closed or open formulas written under conjunctive normal form. Thus, for instance, coming back to example of section 4.1, the evaluator can answer queries like the following: • When querying the three databases and assuming that their reliability degrees are respectively 3,2 and 1, what are the flights which fly to NewYork or to Washington ? answer( (3.Agency ∗ 2.Orly ∗ 1.AF ), f light(x, N ewY ork, z, t, u) ∨ f light(x, W ashington, z, t, u)) = {(4 N EW Y ORK 1945 T W A IN T ERN AT ION AL), (5 W ASHIN GT ON 2200 T W A IN T ERN AT ION AL)} • When querying the three databases and assuming that their reliability degrees are respectively 3,2 and 1, what are the flights which fly to London and which are not AirFrance flights ? answer( (3.Agency ∗ 2.Orly ∗ 1.AF ), f light(x, London, z, t, u) ∧ ¬f light(x, y, z, AF, u)) = {(3 LON DON 1430 BAW IN T ERN AT ION AL)} Secondly, following [8], we have extended the evaluator in order to provide answers with explanations by distinguishing when the answer is YES (resp, NO) due to the weighted majority vote between the databases (such answer is called “Yes (resp, NO) by weighted majority”); when the answer is YES (resp, NO) because at least one database allows to deduce it and no other database contradicts it (it is called “Yes (resp, NO) unchallenged”); when the answer is “?” because, according to the weighted majority vote, the weight of data which allow to conclude YES is identical to the weight of data which allow to conclude NO (it is called “? balanced inconsistency”); or finally when the
Querying Contradictory Databases
167
answer is “?” because there is no information at all for providing a positive or a negative answer (it is called “Complete lack of information).” For instance, running the example of section 4.1, the answers would be now : • answer((1.Agency∗1.Orly∗1.AF ), f light(2, London, 1130, AF, international))= YES (Weighted majority) • answer((1.Agency∗3.Orly∗1.AF ), f light(2, London, 1130, AF, international))= NO (Weighted majority) • answer((1.Agency∗3.Orly∗2.AF ), f light(2, london, 1130, AF, international))= ? (Balanced inconsistency) • answer((1.Agency ∗ 1.Orly ∗ 1.AF ), f light(x, y, z, t, international)) = {(2 LONDON 1130 AF) (Weighted majority) , (4 NEWYORK 1945 TWA) (Unchallenged) } • answer((1.Agency ∗ 3.Orly ∗ 1.AF ), f light(x, y, z, t, international)) = {(4 NEWYORK 1945 TWA) (Unchallenged)} • answer((1.Agency ∗ 3.Orly ∗ 2.AF ), f light(x, y, z, t, international)) = {(4 NEWYORK 1945 TWA) (Unchallenged)} • answer((1.Agency ∗ 2.Orly), terminal(x, OrlyW )) = { 1 (Unchallenged), 2 (Weighted Majority), 3 (Weighted Majority) , 5 (Weighted Majority)} • answer((1.Agency ∗ 2.Orly ∗ 3.AF ), terminal(x, OrlyW )) = {1 (Unchallenged, 3 (Weighted Majority), 5 (Weighted Majority)}
Acknowledgments This work has been supported by ONERA (grant 1042801F)
References [1] C. Baral, S. Kraus, J. Minker, and V.S. Subrahmanian. Combining multiple knowledge bases. IEEE Trans. on Knowledge and Data Engineering, 3(2), 1991. [2] C. Baral, S. Kraus, J. Minker, and V.S. Subrahmanian. Combining knowledge bases consisting of first order theories. Computational Intelligence, 8(1), 1992. [3] C. Batini, M. Lenzerini, , and S.B. Navathe. A comparative analysis of methodologies for database schema integration. ACM Computer Surveys, 18(4), 1986. [4] L. Bertossi and J. Chomicki. Query answering in inconsistent databases. In Logics for Emerging Applications of Databases, pages 43–83. 2003. [5] L. Cholvy. Reasoning with data provided by federated databases. Journal of Intelligent Information Systems, 10(1), 1998.
168
Laurence Cholvy
[6] L. Cholvy and R. Demolombe. Reasoning with information sources ordered by topics. In Proceedings of the 6th International Conference on Artificial Intelligence : Methodologies, Systems, Applications (AIMSA’94), pages 151–162, Sofia, September 1994. World Scientific. [7] L. Cholvy and Ch. Garion. Answering queries adressed to several databases according to a majority merging approach. Journal of Intelligent Information Systems, 22(2), 2004. [8] L. Cholvy and Ch. Garion. Querying several conflicting databases. Journal of Applied Non-Classical Logics, 14(3), 2004. [9] S. Konieczny and R. Pino-P´erez. On the logic of merging. In Proc. of KR’98L.Bertossi and J. Chomicki., 1998. [10] D. Lembo, M. Lenzerini, , and R. Rosati. Source inconsistency and incompleteness in data integration. In Proceedings of KRDB’02, 2002. [11] J. Lin. Integration of weighted knowledge bases. Artificial Intelligence, 83:363– 378, 1996. [12] J. Lin and A.O. Mendelzon. Merging databases under constraints. International Journal of Cooperative Information Systems, 7(1), 1998. [13] J. Lin and A.O. Mendelzon. Knowledge base merging by majority. In Dynamic Worlds: From the Frame Problem to Knowledge Management. Kluwer Academic Publ., 1999. [14] A. Motro. Multiplex: A forma model for multidatabase and its implementation. In Proc. of NGITS-99, the Fourth Int. Workshop on Next Generation Information Technologies and Systems, Israel, July 1999. Springer-Verlag. [15] F. Sadri. Reliability of answers to queries in relational databases. IEEE Transactions on Knowledge and Data Engineering, 3(2):245–252, 1991. [16] S. S.Benferhat, D. Dubois, J. Lang, H. Prade, A. Saffiotti, and P. Smets. A general approach for inconsistency handling and merging information in prioritized knowledge bases. In Proc. of KR’98, Trento, 1998. [17] V.S. Subrahmanian. Amalgamating knowledge bases. ACM Transactions on Database Systems, 19(2):291–331, 1994.
Database Integrations in Distributed Enterprise Information Systems: A Database Model with Imprecise Information and Query Processing Li Yan Northeastern University, Shenyang, Liaoning 110004, China
1 Introduction To increase product competitiveness, today’s manufacturing enterprises have to deliver their products at reduced cost and high quality in a short time. The change from sellers’ market to buyers’ market results in a steady decrease in the product life cycle time and the demands for tailor made and small-batch products. All these changes require that manufacturing enterprises quickly respond to market changes. Traditional production patterns and manufacturing technologies may find it difficult to satisfy the requirements of current product development. Many types of advanced manufacturing techniques, such as Computer Integrated Manufacturing, Agile Manufacturing, Concurrent Engineering, and Virtual Enterprise based on global manufacturing have been proposed to meet these requirements. One of the foundational supporting strategies is the computer-based information technology. Information systems have become the nerve center of current manufacturing systems. It should be noted that, for various organizational and technological reasons, multiple information systems used in information-based manufacturing enterprises are independently developed, locally administered, and different in logical or physical design [10, 24]. Therefore, a fundamental challenge in heterogeneous and distributed enterprise information management is the sharing of information for enterprise users across organizational boundaries [20]. Databases are an essential part of modern information systems. Therefore, some studies have concentrated on conceptual data modeling of distributed enterprise information systems [18, 25, 43]. But there are few researches on modeling and design of database systems of distributed enterprise information systems. It is especially true for information sharing of such database systems although it is essential. As we know, integrating heterogeneous databases into one system is a major means by which information in distributed information sources can be shared. In L. Yan: Database Integrations in Distributed Enterprise Information Systems: A Database Model with Imprecise Information and Query Processing, Studfuzz 203, 169–192 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
170
Li Yan
the field of databases, the necessity of being able to integrate databases from multiple organizations into a single database has been clearly pointed out by Silberschatz, Stonebraker and Ullman [38]. In the last two decade, heterogeneous database integration has been studied extensively both from a formal and from a practical point of views [2, 14, 17, 21, 29, 37]. More recently, much of the research on integration has focused on integrating conceptual data models and/or object-oriented databases [5, 34] and integrating data sources on the Web [3]. It should be noted that, however, relational databases are still the leading product of commercial database systems so far and most of enterprise information systems are developed based on relational databases. In addition, XML documents for Web-based applications are generally stored, manipulated, and queried in relational databases [4, 39]. Therefore, we focus on multiple relational database integration in this chapter. A core task in database integrations is identification and solutions of a number of conflicts and contradictions in existing component databases. As a result, the integrated databases may contain imprecise information even if component databases are precise [1, 12, 27, 40]. It has been identified that information integration of multidatabases is one reason that databases contain imprecise information [42]. Traditional databases are not able to represent and manipulate imperfect values. So it has been pointed out that the next generation of database systems should address the problems of incompleteness and inconsistency arising from the integration of distributed heterogeneous databases [38]. A number of approaches to querying imprecise data have been investigated under central databases environments [6, 28, 35, 33, 41] or database integration environments [12, 15, 19, 22, 31, 32, 40]. However, only one kind of imprecise information is generally considered in these researches. Furthermore, the methods to query processing are mainly based on qualitative measures. Quantitative computations for query processing are not further investigated although they can provide users with more informatics. Engineering applications are data and knowledge intensive applications. Rich database schemas and data types in component databases may result in several types of imprecise information in integrated databases, including null values, set values, and value intervals, just as shown in the next section. In this chapter, we introduce an extended relational database model that can be applied as the integrated database schema for unified representations of these three types of imprecise data. Based on the extended relational databases with imprecise information, some relational operations for query processing are defined. In particular, we differentiate and develop two kinds of query strategies in this chapter, which are qualitative operations and quantitative operations. The remainder of the chapter is organized as follows. An example scenario is presented in Section 2. We present that virtual enterprise manufacturing systems require information integration for their production activities and the solutions of several kinds of conflicts result in different kinds of imprecise values
Database Integrations in Distributed Enterprise Information Systems
171
simultaneously. In Section 3, an extended relational database model is introduced as the integrated database schema to represent imprecise information. Based on the extended relational databases, in Section 4, some relational operations are defined according to qualitative and quantitative query strategies, respectively. Also, several properties of these relational operations are discussed in Section 4. Section 5 provides a brief overview of related work. Section 6 concludes this chapter.
2 An example scenario Today’s enterprise information systems are generally distributed and heterogeneous. Information sharing for enterprise users across organizational boundaries is essential. In this section, we give virtual enterprise information systems as an example scenario, which typically have the features of today’s enterprise information systems mentioned above. 2.1 Organization structures A virtual enterprise consists of a number of units geographically dispersed but managed as one total unit, although the sub–units may be under separate management [36]. In a virtual enterprise, there is a master company, which may not have its own fabrication facilities and develops products by relying on other manufacturing companies, called partner companies or partners. Moreover, the organizational structure of a virtual enterprise is product–oriented, which implies that the relationship between master and partners is not permanent. In a virtual enterprise, product/configuration design and manufacturing are such a procedure that the master company selects or syntheses the partner companies and monitors their production activities by deriving information concerned. Four types of information have been identified in a virtual enterprise [43], which are product data, part manufacturing data, partner company data, and project progress monitoring data. Partner data can be divided into partner structure data and behavior and performance data. Partner structure data include number and working status of manufacturing equipments, workloads of a partner, etc. Partner behavior and performance data include types of products producible, type of manufactures, lead-time, prices, quality level, and etc. Project progress monitoring data include the person who is responsible for the development of a particular part, the starting and ending dates for the development of a part etc. Among these four types of data, the first one is within the master company and used only by itself and the latter three ones are within the partner company. But the partner company data and project progress monitoring data are used by the master company also. It is clear
172
Li Yan
that organization and production management of a virtual enterprise put an essential requirement on information integration that the master company shares the information from the corresponding partner companies during its engineering activities [20]. 2.2 Database integration and conflict resolutions The master company needs to access and derive data from multiple data sources in partner companies to organize production activities effectively. The data sources in partner companies, however, are independently developed, logically administered, and different in logical or physical designs from the other subsystems. So the features of the information systems in virtual enterprises can be generalized as heterogeneous and distributed. Databases are the major data repositories of information systems and relational databases, being the leading product of commercial database systems so far, are extensively employed in the developments of information systems, including enterprise information systems. Therefore, viewed from databases, multiple information systems integrations are essentially multiple database integrations, which are heterogeneous, distributed and independent. Figure 1 shows the architecture of multiple databases in virtual enterprise information systems, in which the CDB and PDB resided in a partner company are the databases of partner company data and project progress data, respectively. The CDB and PDB resided in master company are obtained through integrating the CDBs and PDBs resided in partner companies, respectively.
PDB
Extranet
……
PDB
Partner Company m
CDB
Partner/Project Data
Partner/ Project Data
Master Company
Partner Company 1
CDB
CDB
PDB
Partner/Project Data
Fig. 1: Architecture of Multiple Databases in Virtual Enterprise Information Systems A core task in database integrations is to identify and resolve a large number of incompatibilities, i.e., semantic conflicts, which exist in different databases
Database Integrations in Distributed Enterprise Information Systems
173
for semantically related data. There is a huge literature in this topic and the following semantic conflicts have extensively been identified. (a) Naming conflicts. This type of conflict can be divided two aspects. One is semantically related data items are named differently and the other is semantically unrelated data items are named equivalently. (b) Data representation conflicts. This occurs when semantically related data items are represented in different data types. (c) Data scaling conflicts. This occurs when semantically related data items are represented in different databases using different units of measure. (d) Missing data. This occurs when the schemas of the integrating databases have different attribute sets. A basic approach for conflict resolution is mapping and conversion. That means that semantically related integrating data items are mapped and conversed into an integrated data item (see figure 2), which corresponds to an attribute in integrated schema (called virtual attribute) and a value on the attribute. There are three mappings for conflict resolutions, which are one-toone mapping, many-to-one mapping, and one-to-many mapping. Using one-toone mapping, naming conflicts can be resolved through mapping semantically related data items (semantically unrelated data items) to a common virtual attribute (different attribute). Also using one-to-one mapping, data representation conflicts can be resolved through mapping semantically related data items to a common virtual attribute. Data scaling conflicts, however, can be resolved through mapping semantically related data items to a common virtual attribute using either many-to-one mapping or one-to-many mapping. As to the conflicts of missing data, it can be resolved using outerjoin operation [16]. 2.3 Information imprecision Conflict resolutions in database integrations generally result in information imprecision in the integrated databases. It has been identified that information integration of multidatabases is one reason that databases contain imprecise information [42]. Let’s focus on one-to-many mapping. It is clear that the attributes in the integrating data items are mapped into range values on the virtual attribute because of on one-to-many mapping. Depending on if all these attributes are discrete or continua, the range values may be set values or value intervals. In addition, when using outerjoin operation for missing data conflicts, the missing attribute values are null values on the virtual attribute in the result databases. Let us look at an example. There are two databases r and s shown in Table 1, which belong to two partner companies in different sites, respectively. They all have the same key attribute Part ID. To provide master company with both sites information about price, lead-time, and transport of the products, these two
174
Li Yan
databases should be integrated. It is clear that attribute values on Price of r and attribute values on Cost of s are semantically related but they are named differently and represented in different data types. Attribute values on Lead-time of r and attribute values on Lead-time of s as well as attribute values on Transport of r and attribute values on Transport of s are represented using different units of measuring. In addition, attribute Material of r is a missing attribute with respect to s. Site 2: Partner Company Component Database s
Site 1: Partner Company Component Database r
Attribute B
Attribute A
b
a
Domain (A)
Mapping: one-to-one many-to-one one-to-many
Integration
Domain (B)
Mapping: one-to-one many-to-one one-to-many
Site 3: Master Company Integrated Database u Attribute K k
Domain (K)
Fig. 2 Strategy of Database Schema Integration Table 1: Databases of partner companies
Site 1: Partner Database r Part ID
Price (Dollars)
Lead-time (Weeks)
Material
Transport
10010 10011 10013
50 55 67
2 3 3
Fe Cu Alloy
Land Air Land
Site 2: Partner Database s Part ID
Cost (Dollars)
Lead-time (Days)
Transport
10010 10012 10013
50.30 58.80 67.47
12 18 21
Train Vessel Truck
Database Integrations in Distributed Enterprise Information Systems
175
Suppose we have two facilities for land transport (train and truck) and one facility for air transport (plane). In addition, lead-time (weeks) is considered to correspond to value intervals. Two weeks, for example, mean more than 7 days but less than or equal to 14 days. Following the integration strategies and approaches above, we have the integrated relation shown in Table 2, where notations [], {}, and ϕ stand for value intervals, set values, and null values, respectively. Table 2: Relational database with imprecise values
Part ID 10010 10011 10012 10013
Price (Dollars)
Lead-time (Days)
Material
Transport
[50, 50] [55, 55] [59, 59] [67, 67]
[8, 14] [15, 21] [18, 18] [15, 21]
{Fe} {Cu}
{Train, Truck} {Plane} {Vessel} {Train, Truck}
ϕ {Alloy}
It has been shown that several types of imprecise information such as null values, set values, and value intervals may occur in integrated relational databases because of conflict resolutions. One of the goals that we integrate multiple databases is to obtain more information from the integrated databases. Traditional databases are not able to represent and manipulate these imperfect values. So, in the following, we present an extended relational database model that can represent these types of imprecise values. Also, we develop qualitative and quantitative operations for information query processing from the integrated databases with imprecise information.
3 An extended relational database model In classical relational databases, a relational schema R is a finite set {A1, A2, ..., An}, denoted by R = {A1, A2, ..., An} or R (A1, A2, ..., An), where Ai (1 ≤ i ≤ n) is called attribute. Each attribute Ai associates with a domain of values, denoted by Dom (Ai) or Di, which is a set of possible values taken by attribute values and may be discrete or continua. A relation instance r on schema R, denoted by r (R), is a set of tuples and for a tuple t ∈ r, it consists of attribute values t [Ai] (1 ≤ i ≤ n), where t [Ai] ∈ Dom (Ai). In classical relational databases, all attribute values of tuples must be atomic. An imprecise value in databases corresponds to a range of values, in which true value is restricted on a specific set of possible values. Viewed from the format of imprecise values, an imprecise value may be a set of possible values
176
Li Yan
(for discrete universe of discourse) or a value interval (for continua universe of discourse). When the value range of an imprecise value is the same as the corresponding attribute domain, it is generally called a null value. Formally imprecise values are defined as follows. Definition 3.1. An imprecise value on an attribute A corresponds to a range of possible values in which exactly one value in the range should be the true value of the imprecise value. Formally it is denoted by {a1, a2, …, am} for discrete attribute domain Dom (A), call a set value, or [a1, an] for continua attribute domain Dom (A), called a value interval. Here, {a1, a2, …, am} (or [a1, an]) ⊆ Dom (A). When an imprecise value corresponds to whole attribute domain, it is essentially a null value, denoted by ϕ. Note that crisp attribute values can be viewed as special cases of imprecise values. A crisp value, for example, p, on a discrete universe of discourse (or a continua universe of discourse) can be represented as {p} (or [p, p]). Moreover, the imprecise value without containing any element is called an empty imprecise value, denoted by ⊥. In fact, the symbol ⊥ means an inapplicable missing data [11]. Definition 3.2. For a set value η = {a1, a2, …, am}, the number of possible values in η is called the modular of η whereas the interval length of a value interval η = [a1, an] is called the modular of η. The modular of an imprecise value η is denoted by ||η||. Considering crisp values and empty imprecise values, we define the modular of an imprecise value as follows. Definition 3.3. Let η be an imprecise value on the universe of discourse U. Then (a) ||η|| = 0 if η = ⊥, i.e., η is an empty imprecise value. (b) ||η|| = 1 if U is a discrete universe of discourse and η = {p}. (c) ||η|| = δ if U is a continua universe of discourse and η = [p, p]. Here, δ is a very small number compared with ||U||. Generally, δ can be viewed to be so close to 0. (d) ||η|| = b − a if η = [a, b] and b ≠ a. (e) ||η|| is equal to the number of possible values in η if η is a set value on U. It is clear that the relational database model that is able to accommodate imprecise values, including set values, value intervals, and null values, must be an extended one. Let R (A1, A2, ..., An) be a relational schema and D1, D2, ..., Dn be the Di corresponding attribute domains. Let 2 denote the power set of Di. Then Di instead of Di, 2 will be the new domain of attribute Ai in extended relational
Database Integrations in Distributed Enterprise Information Systems
177
database model, called extended attribute domain and denoted by Dom* (Ai) to differentiate the traditional attribute domain. Based on the extended relational database model above, a relation instance r (R) is defined as a subset of Cartesian product Dom* (A1) × Dom* (A2) × ... × Dom* (An). For a tuple t ∈ r, it consists of attribute values t [Ai] (1 ≤ i ≤ n), where t [Ai] may be a crisp value, set value/value interval, ϕ, or ⊥. Anyway, t [Ai] ∈ Dom* (Ai). It should be noted that, however, primary key plays a role of the identifier of tuples in a relation, imprecise values are not permitted as the attribute values on the primary key. An example relation with imprecise values based on the extended relational database model is shown in Table 2.
4 Relational operations
4.1 Semantic relationships between imprecise values It is crucial to measure semantic relationships between imprecise values in order to manipulate such imprecise values in the extended relational databases. We introduce two types of semantic measure, namely, qualitative one and quantitative one. Qualitative measures Definition 4.1. Let t and s be two tuples in relation r (R) and A ∈ R. Then (a) t [A] and s [A] are equivalent to each other, denoted by t [A] ≡ s [A], if t [A] = s [A], i.e., they are two common crisp values, (b) t [A] and s [A] are inclusive, denoted by t [A] ≈ s [A], if t [A] ⊆ s [A] or t [A] ⊇ s [A], and (c) t [A] and s [A] are interrelated, denoted by t [A] ~ s [A], if t [A] ∩ s [A] ≠ Φ, in which Φ means an empty set. The above-mentioned semantic relationships between two attribute values are called equivalence, inclusion, and interrelation, respectively. It is clear that, for any an attribute, null values and crisp values as well as set values/value intervals must be inclusive. In addition, it can be seen that the equivalence relationship is reflexivity, symmetry, and transitivity, the inclusion relationship is reflexivity and transitivity, and the interrelation relationship is reflexivity and symmetry. For two imprecise data, say η1 and η2, η1 ≡ η2 ⇒ η1 ≈ η2, and η1 ≈ η2 ⇒ η1 ~ η2, where ⇒ indicates implication. The direction of these inferences is irreversible.
178
Li Yan
Definition 4.2. Let t1 and t2 be two tuples in relation r (R), where R = {A1, A2, ..., An}. If t1 (Ai) ≡ t2 (Ai) (0 ≤ i ≤ n), t1 and t2 are equivalent to each other, denoted by t1 ≡ t2. If t1 (Ai) ≈ t2 (Ai) (0 ≤ i ≤ n), t1 and t2 are inclusive, denoted by t1 ≈ t2. If t1 (Ai) ~ t2 (Ai) (0 ≤ i ≤ n), t1 and t2 are interrelated to each other, denoted by t1 ~ t2. Based on Definition 4.2, two kinds of data redundancies can be identified in the extended relational databases with imprecise values. The first one is due to the tuples that are equivalent to each other and the second one is due to the tuples that are inclusive or interrelated to each other. For the former, we just remove duplicate tuples. But for the latter, all tuples that are redundant by one another should be combined into one tuple which attribute values are the unions of all corresponding attribute values in combining redundant tuples. We call these two kinds of strategies of eliminating redundancies as definite and maybe. Then an operation Reduce (r, type) can be defined to eliminate data redundancies, in which r is a relation, type can be definite and maybe. Quantitative measures Notions of equivalence, inclusion, and interrelation are actually used to differentiate the degrees of information "similarity". Let η1 = [11, 13], η2 = [10, 15], and η3 = [12, 18] be three value intervals. According to Definition 4.1, we have η1 ≈ η2, η1 ~ η3, and η2 ~ η3. Intuitively, η1 is more similar to η2 than to η3. But what are their similar degrees? Especially, η1 and η2 are all similar to η3, but which one is more similar to η3? The notions given above cannot answer these questions because they only qualitatively describe semantic relationship between data items. Therefore, a novel notion equivalence degree is introduced to quantitatively measure the semantic relationship between imprecise values, which stems from the idea of the nearness measure and semantic proximity in fuzzy relational databases [7, 26, 30]. Definition 4.3. Let η1, η2 be two imprecise values on attribute A. The equivalence degree of η1 and η2, denoted by ED (η1, η2), is represented as follows. Here, Dom (A) and ||η|| denote the traditional attribute domain and the modular of imprecise value η, respectively. ED (η1, η2) = ||η1 ∩ η2||/|||η1 ∪ η2|| − ||η1 ∩ η2||/||Dom (A)|| Example 4.1. Let Dom (A) = {a1, a2, …, a20} and so ||Dom (A)|| = 20. Then (a) Suppose η1 = {a11} and η2 = {a11}. Then ED (η1, η2) = 1/1 - 1/20 = 19/20. (b) Suppose η1 = {a12, a13, a14, a15} and η2 = {a16, a17}. Then ED (η1, η2) = 0/6 0/20 = 0.
Database Integrations in Distributed Enterprise Information Systems
179
(c) Suppose η1 = {a14, a15, a16}, η2 = {a14, a15, a16}, η3 = {a14, a15, a16, a17, a18}, and η4 = {a14, a15, a16, a17, a18}. Then ED (η1, η2) = 3/3 - 3/20 = 0.85 and ED (η3, η4) = 5/5 - 5/20 = 0.75. We have ED (η1, η2) > ED (η1, η3). (d) Suppose η1 = {a11, a12, a13}, η2 = {a10, a11, a12, a13, a14, a15}, and η3 = {a12, a13, a14, a15, a16, a17, a18}. Then ED (η1, η2) = 3/6 - 3/20 = 0.35 and ED (η1, η3) = 2/8 2/20 = 0.15. We have ED (η1, η2) > ED (η1, η3). η1 is more similar to η2 than to η3. Intuitively, the more two imprecise values overlap, the more similar they are. Besides, the semantic equivalence degree of two imprecise values is also relevant to their modular. Case (c) in the above example can illustrate this situation well. The notion of semantic equivalence degree of attribute values can be extended to tuples. Let ti and tj be two tuples in r (R), where R = {A1, A2, …, An}. The semantic equivalence degree of tuples ti and tj is defined as ED (ti, tj) = min (ED (ti [A1], tj [A1]), ED (ti [A2], tj [A2]), …, ED (ti [An], tj [An])). Then we can define the notion of duplicate degree of tuples to evaluate and eliminate duplicate tuples quantitatively. Definition 4.4. Let r (R) be a relation over R = (A1, A2, …, An). Let ti = and tj = be two tuples in r. Sr (ti, tj) = max t ∈ r (min (ED (ti, t), ED (ti, t))) is called the duplicate degree of ti and tj, where t may be ti. If a threshold λ (0 ≤ λ ≤ 1) is given and Sr (ti, tj) ≥ λ, we say that ti and tj are duplicate tuples. Based on the notion of duplicate degree of tuples, we can define an operator Reduceλ (r) to eliminate the duplicate tuples in r in which duplicate degrees of tuples are larger than or equal to λ. 4.2 Relational operations for schema integration In multidatabase integrations, there exist two crucial operations. The first one is the operation that can resolve some semantic conflicts such as name conflicts, data representation conflicts, and data scaling conflicts. Mapping operation is introduced for this purpose. On the basis, integration operation can be defined. Mapping Let r (K, A, X) be a source relation, where K is primary key, A is an attribute, and X is a set of attributes. Then the mapping of r (K, A, X) into s (K, V, X), denoted ρA→V (r), is defined, where V is a virtual attribute of the corresponding attribute A.
180
Li Yan
ρA→V (r) = {t|(∃ u) (u ∈ r ∧ t [K] = u [K] ∧ t [V] = f (u [A]) ∧ t [X] = u [X])} Here, f () is a conversion function. For the one-to-one and many-to-one mappings, an atomic value u [A] is converted into another atomic value t [V] uniquely. For the one-to-many mapping, however, an atomic value u [A] is converted into a range value t [V] (an value interval or a set value). Integration Let r (K, A, X) and s (K, A, Y) be two source relations, where K is primary key. Then integration of r and s, denoted r ⊕ s, is defined as follows. r ⊕ s = {t|(∃ u) (∀ v) (u ∈ r ∧ v ∈ s ∧ u [K] ≠ v [K] ∧ t [K] = u [K] ∧ t [A] = u [A] ∧ t [X] = u [X] ∧ t [Y] = ϕ) ∨ (∃ v) (∀ u) (v ∈ s ∧ u ∈ r ∧ v [K] ≠ u [K] ∧ t [K] = v [K] ∧ t [A] = v [A] ∧ t [Y] = v [Y] ∧ t [X] = ϕ) ∨ (∀ u) (∀ v) (u ∈ r ∧ v ∈ s ∧ t [K] = u [K] = v [K] ∧ t [X] = u [X] ∧ t [Y] = v [Y] ∧ (∀ Q) (Q ∈ A ∧ t [Q] = u [Q] ∪ v [Q]))} From the definition above, it can be seen that the result relation r ⊕ s consists of three kinds of tuples. The first kind of tuples in r ⊕ s is the tuples that directly come from r. Note that for such a tuples in r, say u, there is not any tuple in s that has the same key value(s) as u’s key value(s). The second kind of tuples in r ⊕ s is the tuples that directly comes from s. Also for such a tuples in s, say v, there is not any tuple in r that has the same key value(s) as v’s key value(s). The third kind of tuples in r ⊕ s is the tuples that are formed through merging the tuples respectively from r and s because they have the same key value(s). Since there may be inconsistency of schema in r and s, namely, missing data conflict, the result relation of integrating r and s may contain null values.
Database Integrations in Distributed Enterprise Information Systems
181
4.3 Relational operations for query processing
Qualitative operations Traditional two-valued logic (True and False) is not able to represent the results of conditional expression with imprecise information. We adopt three-valued logic (3VL), which proposed firstly by [11] to process the null value in relational database. The 3VL consists of three logical values: true, false, and maybe. Based on the 3VL and the qualitative semantic relationships between imprecise information, several relational operations are defined as follows.
Union. Let r (R) and s (R) be two relations with imprecise values on the same schema. We have two kinds of strategies for the union operations: definite and maybe. Considering the possible data redundancies, we have: r ∪ s = Reduce (r ∪ s, type), type
where ∪ is the common union operator under classical circumstance, and type represents the operation strategies, which may be definite or maybe. Projection. Let r (R) be a relation with imprecise values. Also we have two kinds of strategies for the project operations: definite and maybe. Let X ⊆ R. then we have Π (r) = Reduce (Π (r), type), type
X
X
where Π (r) is the common projection operator under classical circumstance, X
and type represents the operation strategies, which may be definite or maybe. Difference. Difference between two relations r (R) and s (R) is realized by reducing the tuples from r that are the same as that in s. Considering the qualitative semantic relationships between tuples with imprecise information, we have r − s = {t|(∃ u) (∀ v) (u ∈ r ∧ v ∈ s ∧ (t = u) ∧ ((u = v) and (u ≈ v) are all definite
false))} and r − s = {t|(∃ u) (∃ v) (u ∈ r ∧ v ∈ s ∧ ((∀ X) (X ∈ R ∧ t [X] = u [X] – v [X])) maybe
∧ ((u ~ v) ∨ (v ≈ u))}. Natural join. Natural join is a particular kind of join operation. The operation requests that two jointing relations have common attribute subset and only such
Li Yan
182
two tuples that respectively belong to these two relations and which values are the same on the common attribute subset can be jointed together. Let r (R) and s (S) be two relations and X = R ∩ S. Then we have r definite
s = {t|(∃ u) (∃ v) (u ∈ r ∧ v ∈ s ∧ (t [R] = u [R]) ∧ (t [S - X] = v [S - X]) ∧ (u [X] = v [X] is true))} and
r
maybe
s = {t|(∃ u)(∃ v)(u ∈ r ∧ v ∈ s ∧ (t [R - X] = u [R - X]) ∧ (t [S - X] =
v
[S - X]) ∧ (t [X] = u [X] ∩ v [X]) ∧ (u [X] = v [X] is maybe))}. Selection. Let r be a relation with imprecise values and let C be a condition expression. Then the selection with clause C from r is defined as follows. σC (r, definite) = {t|t ∈ r and (C is true for t)} and σC (r, maybe) = {t|t ∈ r and (C is maybe for t)}. Example 4.2. There is a relation with imprecise values shown in Table 2. One makes a query as follows: SELECT Part ID, Price, Lead-time, Transport FROM Table 2 WHERE Leadtime ≤ 20. That means that we’d like to find out all tuples (parts) from Table 2 which values of lead-time is less than 20 days. Here we have two kinds of query strategies: definite and maybe, which correspond to the strategies defined in the relational operations above. Then we have the query answers shown in Table 3 and Table 4, respectively. When the query contain two conditions: (a) Lead-time < 20 and (b) Transport = “Truck”, there is no definite answer for the query. The maybe answer for the query is shown in Table 5. It is clear that there are two kinds of strategies for qualitatively querying databases with imprecise values. In order to obtain the different query answers, SQL should be extended. The following syntax of SQL is used to represent the select sentence.
Part ID 10010 10012
Part ID 10011 10013
Table 3: Definite query answer under single condition Price (Dollars) Transport Lead-time (Days) {Train, Truck} [50, 50] [8, 14] [18, 18] [59, 59] {Vessel}
Table 4: Maybe query answer under single condition Price (Dollars) Transport Lead-time (Days) [55, 55] [15, 21] {Plane} [67, 67] [15, 21] {Train, Truck}
Database Integrations in Distributed Enterprise Information Systems Part ID 10013
183
Table 5: Maybe query answer under complex condition Price Lead-time Transport {Train, Truck} [67, 67] [15, 21]
SELECT FROM FROM <selection condition> WITH Here, a new statement WITH is attached to SQL, in which type may be definite or maybe. Quantitative operations Based on the quantitative semantic relationships between imprecise information and the given threshold, several relational operations are defined as follows. Union. Let r (R) and s (R) be two relations with imprecise values on the same schema. Let λ (0 < λ ≤ 1) be the given threshold. Considering the possible data redundancies under the given threshold, we have r ∪λ s = Reduceλ ({t|t ∈ r ∨ t ∈ s}) It is clear that the union here is λ-union. Projection. Let r (R) be a relation with imprecise values and X ⊆ R. Let λ (0 < λ ≤ 1) be the given threshold. Then the projection is λ-projection and is defined as follows λ
∏X (r) = Reduceλ ({t|(∃ u) (u ∈ r ∧ t = u [X])}) Difference. Let r (R) and s (R) be two relations with imprecise values on the same schema. Let λ (0 < λ ≤ 1) be the given threshold. Then the λ-difference is defined as follows. r −λ s = {t|(t ∈ r ∧ (∀ v) (v ∈ s ∧ S s (t [X], v [X]) < 1 - λ)} Selection. Let r (R) be a relation with imprecise values and λ (0 < λ ≤ 1) be the given threshold. Let X = Y be a condition expression, where X is an attribute and Y is a constant or another attribute. Then the λ-selection is defined as follows.
σ Xλ =Y (r) = {t|t ∈ r ∧ ED (t [X], Y) > λ}
184
Li Yan
Natural join. Natural join of the relations with imprecise values can be defined using Cartesian product and selection. Let r (R) and s (S) be two relations and X = R ∩ S. Let λ (0 < λ ≤ 1) be the given threshold. Then r
λ
s = σ r[ X ]= s[ X ] (r × s) λ
Example 4.3. In order to obtain the query answers with different possibilities, SQL should be extended. A new statement, namely, WITH POSSIBILITY , is attached to SQL. Here 0 < threshold ≤ 1. Note that WITH POSSIBILITY statement can be omitted. The default of is exactly 1. Then one may make a query against the relation shown in Table 2 as follows: SELECT *FROM Table 2 WHERE Lead-time = [9, 16] WITH POSSIBILITY 0.5 Note that this query is a flexible one. Let Dom (Lead-time) = [0, 60]. Following Definition 4.3, we have ED ([8, 14], [9, 16]) = 5/8 - 5/60 = 0.542 > 0.5 ED ([15, 21], [9, 16]) = 1/12 - 1/60 = 0.067 ED ([18, 18], [9, 16]) = 0.0 ED ([15, 21], [9, 16]) = 1/12 – 1/60 = 0.067 Therefore, we have the query answers shown in Table 6.
Part ID 10010
Table 6: Quantitative query result Price (Dollars) Material Lead-time (Days) {Fe} [50, 50] [8, 14]
Transp {Train,
Properties of relational operations In this section, we focus on several properties of the relational operations defined above. Being similar to the classical relational databases, the proposed relational operation is sound. In other words, it is closed. It means that the results of all operations are valid relations. In detail, the result relations produced by the relational operations satisfy the following three criteria: (a) the attribute values must come from an appropriate attribute domain, (b) there are no duplicate tuples in a relation, and (c) the relation must be a finite set of tuples. In the following, we investigate how the relational operations defined above can satisfy these three criteria. Here, the relational operations refer to qualitative and quantitative ones simultaneously. First of all, let us look at the satisfactory situation of the first criterion. Projection and selection all take out a part from the source relation in either column direction or row direction. Because the
Database Integrations in Distributed Enterprise Information Systems
185
attribute values in source relation must belong to the appropriate attribute domain, the attribute values in these two result relations must come from the appropriate attribute domain. Union and difference are conducted under unioncompatible condition, which satisfies the first criterion. In natural join and Cartesian product, the attribute values in result relations come from two source relations, respectively, and they must be within the appropriate attribute domains. For selection, if there are no redundant tuples in ordinary relation, there are no redundant tuples in the result relations. There exist no redundant tuples in the result relations of union, difference, natural join, Cartesian product, and projection. This can be ensured by the definitions of those operations because the removal of redundancies has been considered. Therefore, the second criterion is satisfied. Now let us look at the satisfactory situation of the third criterion. Let r and s be two relations with imprecise values, and let |r| and |s| denote the numbers of tuples in r and s, respectively. Let σP (r) denote the selection operation. It is clear that 0 ≤ |σP (r)| ≤|r| for fuzzy selection. This implies that when no any tuple in r satisfies the selection condition, the tuple number in the result relation is 0, and that when all tuples in r satisfy the selection condition, one obtains |σP (r)| = |r|. When part of the tuples in r satisfy the selection condition, |σP (r)| must be greater than 0 and less than |r|. For the projection denoted ΠX (r), if all tuples in r are redundant after projecting, then |ΠX (r)| = 1. If there is not any redundancy in r after projecting, then |ΠX (r)| = |r|. In the other situations, |ΠX (r)| must be greater than 1 and less than |r|. Additionally, |r ∪ s| must not be s | and |r × s| greater than |r| + |s|, |r – s| must not be greater than |r|, and |r must not be greater than |r| × |s|. Since the number of tuples in the result relation is closely related with the source relations and the source relations are finite, the result relations must be finite. In addition, the set operations defined in relational operations above have similar properties as those of classical set operations. Let r, s, and u be three union-compatible fuzzy relations. Then (a) r ∪ s = s ∪ r and r ∪λ s = s ∪λ r, type
type
(commutativity) (b) r ∪ r = r and r ∪λ r = r, type
(idempotence) (c) (r ∪ s) ∪ u = r ∪ (s ∪ u) and (r ∪λ s) ∪λ u = r ∪λ (s ∪λ u), and type
type
type
type
(associativity) (d) r ∪ s = r ∪ (s − r) and r ∪λ s = r ∪ (s −λ r). type
type
Li Yan
186
The following properties are also held in the relational operations. Let r (R), s (R), and u (Q) be relations with imprecise values. Let P be a selection predicate involving attributes of R. Then (r ∪ s) = (u r) ∪ (u s) and u (r − s) = (a) u type1
(u
type1
r) − (u type 2
(b) u
λ
type1
type2
type1
type1
type2
type1
type 2
s),
(r ∪λ s) = (u
λ
r) ∪λ (u
λ
s) and u
λ
(r −λ s) = (u
λ
r)
−λ (u λ s), (c) σP (r ∪ s, type2) = σP (r, type2) ∪ σP (s, type2) and σP (r − s, type2) = type1
type1
type1
σP (r, type2) − σP (s, type2), type1
λ
λ
λ
λ
(d) σ P (r ∪λ s) = σ P (r) ∪λ σ P (s) and σ P (r −λ s) = (e) σP (u r, type2) = u σP (r, type2), and type1
λ
σ Pλ (r) −λ σ Pλ (s),
type1
λ
(f) σ P (u λ r) = u λ σ P (r). It should be noted that type1 and type2 above may be definite or maybe, respectively. All these properties can be proven by the definitions of relational operations above.
5 Related work Incomplete information query processing was extensively studied in relational databases. Allowing an attribute value to be a set value, Lipski [28] provided two different interpretations of queries, the internal and the external. The internal interpretation ignores all imprecise information, referring only to what is known to the system. In contrast, the external interpretation refers to the real world, which is modeled in an incomplete way by the system and imprecise information is taken into account. Reiter [35] proposed an extended relational theory to formulate relational databases with null values and presented a query evaluation algorithm for such databases. However, due to indefinite information brought in by null values, his algorithm is sound but not complete. The sound and complete query evaluation algorithms for relational databases with null values and disjunctive information were developed by Yuan and Chiang [41]. It should be noted that all these developments are based on deductive databases. Considering several different types of null values (including set values), Candan, Grant and Subrahmanian [6] gave a unified treatment of null values in relational databases using constraints, where each tuple t has an associated constraint Ct.
Database Integrations in Distributed Enterprise Information Systems
187
Based on such conditional tables, null-valued algebraic operations were defined and several properties of their algebra were studied. Allowed any attribute value to a missing value, an inapplicable value, a set value, or value interval, Morrissey [33] investigated uncertain query evaluation and classified two sets of query answers: the set of objects that are known to satisfy with complete certainty and the set that possibly satisfies the query with various degrees of uncertainty. However, relational operations for query were not defined in his paper. In addition, it was noticed by Morrissey [33] that the measures of uncertainty are useful and necessary for ranking objects for presentation to a user. Two methods of estimating this uncertainty, based on information theory, were hereby proposed. Indeed quantitative computations for handling imprecise and uncertain information in databases are receiving considerable attentions. Probability theory and fuzzy set theory, for example, are extensively used to represent, store, and manipulate such information on one hand. On the other hand, some methods based on artificial intelligence were proposed in literatures that estimate null values for approximating answers to queries against incomplete relational databases [8, 9, 13, 23]. In the context of database integrations, data integration problem has received considerable attention. Originally, in [12], set values (called partial values there) were introduced in the integrated relational databases to resolve mismatched domains and a set of extended relational operators was defined based on definite and maybe strategies. From then on, a number of approaches to querying imprecise data are extensively investigated. In [22], the Information Manifold project gave an algorithm for computing answers to queries posed over the global schema. Grahne and Mendelzon [19] considered the uncertainty introduced by multiple sources and defined a set of possible global databases consistent with a collection of sources, in which some of them are sound, some are complete, and some are both sound and complete. In their work, they gave upper and lower approximations to query answers and proved that the answer computed by the Information Manifold algorithm coincided with the certain answer. Tseng, Chen and Yang [40] took a different view for quantitative query answers with degrees of uncertainty. They used a probabilistic relational database model to represent uncertainty arising from data heterogeneity, where probabilities arose at the tuple level and at the attribute level. Based on the probabilistic relational databases, they developed a set of extended relational operations. Also, Florescu, Koller and Levy [15] used probabilistic knowledge in data integration. There, information about the completeness and relative overlap of data sources was used in ordering the accesses to sources in order to maximize the likelihood of obtaining answers early in the evaluation. Since quantitative computing can provide more informatics in handling imprecise and uncertain information, more recently, some computing methods have been proposed for imprecise query answers in data integrations. An approach was
188
Li Yan
presented in [31] for estimating loss of information based on navigation of ontological terms. They defined measures for loss of information based on intensional information as well as on well-established metrics like precision and recall based on extensional information. These measures were used to select results having the desired quality of information. Mendelzon and Mihaila [32] considered the problem of querying collections of sources with incomplete and partially sound data. They provided a method for checking the consistency of a source collection and proposed a probabilistic semantics for query answers.
6 Conclusion Nowadays successful manufacturing systems based on distributed organization principles heavily rely on development and management of distributed information systems. Manufacturing enterprises based on distributed information systems make it possible to organize production activities effectively across organizational boundaries and respond to dynamic changes of the market quickly. An important issue needed to be addressed in distributed enterprise information systems is information sharing. Database integration is one of important means by which information can be shared. In this chapter, we concentrate on the database integrations in the context of distributed enterprise information systems in virtual enterprises. We show that several types of imprecise information, including null values, set values, and value intervals, may arise in integrated databases simultaneously since there exist rich database schemas and data types in component databases under such environment and a number of conflicts are resolved. So we introduce an extended relational database model as the integrated database schema for unified representations of these types of imprecise information. Based the extended relational databases with imprecise information, some relational operations for qualitative and quantitative query processing are defined in the chapter. It should be pointed out, however, that the database integrations considered in the chapter assume that the integrating relational databases have the same keys. In other words, there is no incompatible key conflict in the integrations. It is possible that the component databases developed and maintained independently have different keys. It is especially true for distributed enterprise information systems. How to identify entities for integrations in this situation is hereby very interesting. In addition, the component databases considered in the chapter are all relational databases. However, there are various data sources in current distributed enterprise information systems, including structured relational databases and object-oriented databases as well as semistructured XML files. Integrating heterogeneous information source has been a challenge in the
Database Integrations in Distributed Enterprise Information Systems
189
management of distributed enterprise information systems. Our future work will focus on these issues.
References [1] Altareva, E. and Conrad, S., 2001, The Problem of Uncertainty and
[2]
[3]
[4] [5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
Database Integration, Proc. of the 4th International Workshop on Engineering Federated Information Systems, 92-99. Batini, C., Lenzerini, M. and Navathe, S. B., 1986, A Comparative Analysis of Methodologies for Database Schema Integration, ACM Computing Surveys, 18 (4), 323–364. Bergamaschi, S., Castano, S., Vincini, M. and Beneventano, D., 2001, Semantic integration of heterogeneous information sources, Data & Knowledge Engineering, 36 (3), 215-249. Bourett, R., 2001, XML and Databases, http://www.rpbourret.com/xml/ XMLAndDatabases. htm Bukhres, O. A. and Elmagarmid, A. K., 1996, Object-Oriented Multidatabase Systems: A Solution for Advanced Applications, PrenticeHall 1996. Candan, K. S., Grant, J. and Subrahmanian, V. S., 1997, A Unified Treatment of Null Values Using Constraints, Information Sciences, 98 (1-4), 99-156. Chen, G. Q., Vandenbulcke, J. and Kerre, E. E., 1992, A General Treatment of Data Redundancy in a Fuzzy Relational Data Model, JASIS, 43 (4), 304311. Chen, S. M. and Lee, S. W., 2003, A New Method to Generate Fuzzy Rules From Relational Database Systems for Estimating Null Values, Cybernetics and Systems: An International Journal, 34 (1), 33-57. Chen, S. M. and Yeh, M. S., 1997, Generating Fuzzy Rules From Relational Database Systems for Estimating Null Values, Cybernetics and Systems: An International Journal, 28 (8), 695-723. Cheung, W. M. and Hsu, C., 1996, The Model-Assisted Global Query System for Multiple Databases in Distributed Enterprises, ACM Trans. on Information Systems, 14 (4), 421-470. Codd, E. F., 1987, More Commentary on Missing Information in Relational Databases (Applicable and Inapplicable Information), SIGMOD Record, 16 (1), 42–50. DeMichiel, L. G., 1989, Resolving Database Incompatibility: An Approach to Performing Relational Operations over Mismatched Domains, IEEE Trans. on Knowledge and Data Engineering, 1 (4), 485-493.
190
Li Yan
[13] Dutta, S., 1991, Approximate Reasoning by Analogy to Null Queries, [14]
[15]
[16] [17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
International Journal of Approximate Reasoning, 5, 373–398. Elmagarmid, A. K., Rusinkiewicz, M. and Sheth, A., 1998, Management of Heterogeneous and Autonomous Database Systems, Morgan Kaufmann Publishers, CA. Florescu, D., Koller, D. and Levy, A. Y., 1997, Using Probabilistic Information in Data Integration, Proc. of the 23rd International Conference on Very Large Data Bases, 216-225. Galindo-Legaria, C. A., 1994, Outerjoins as Disjunctions, Proc. of the 1994 ACM SIGMOD International Conference on Management of Data, 348-358. García-Solaco, M., Saltor, F. and Castellanos, M., 1995, A Structure Based Schema Integration Methodology, Proc. of the Eleventh International Conference on Data Engineering, 505-512. Giachetti, R. E., 1999, A Standard Manufacturing Information Model to Support Design for Manufacturing in Virtual Enterprises, Journal of Intelligent Manufacturing, 10, 49-60. Grahne, G. and Mendelzon, A. O., 1999, Tableau Techniques for Querying Information Sources Through Global Schemas, Proc. of 7th International Conference on Database Theory, Lecture Notes in Computer Science, 1540, 332-347. Hardwick, M., Spooner, D. L., Rando, T. and Morris, K. C., 1996, Sharing Manufacturing Information in Virtual Enterprises, Communications of the ACM, 39 (2), 46-54. Hull, R., 1997, Managing Semantic Heterogeneity in Databases: A Theoretical Perspective, Proc. of the Sixteenth ACM SIGACT-SIGMODSIGART Symposium on Principles of Database Systems, 51-61. Kirk, T., Levy, A. Y., Sagiv, Y. and Srivastava, D., 1995, The Information Manifold, Proc. of the AAAI Spring Symposium on Information Gathering in Distributed Heterogeneous Environments. Klein, H. J., 1999, Efficient Algorithms for Approximating Answers to Queries Against Incomplete Relational Databases, Proc.of 6th International Workshop on Knowledge Representation meets Databases, 26-30. Krishnakumar, N. and Sheth, A., 1995, Managing Heterogeneous Multisystem Tasks to Support Enterprise-wide Operations, Distributed and Parallel Databases Journal, 3 (2), 155-186. Li, Q., Zhang, W. J. and Tso, S. K., 2000, Generalization of Strategies for Product Data Modeling with Special Reference to Instance-As-Type Problem, Computers in Industry, 41 (1), 25-34. Liao, S. Y., Wang, H. Q. and Liu, W. Y., 1999, Functional Dependencies with Null Values, Fuzzy Values, and Crisp Values, IEEE Trans. on Fuzzy Systems, 7 (1), 97-103.
Database Integrations in Distributed Enterprise Information Systems
191
[27] Lim, E. P., Srivastava, J. and Shekhar, S., 1994, Resolving Attribute
[28] [29] [30]
[31]
[32]
[33] [34] [35]
[36] [37]
[38]
[39]
[40]
[41]
Incompatibility in Database Integration: An Evidential Reasoning Approach, Proc. of the Tenth International Conference on Data Engineering, 154-163. Lipski, W., 1979, On Semantic Issues Connected with Incomplete Information Databases, ACM Trans.s on Database Systems, 4 (3), 262-296. Litwin, W., Mark, L. and Roussopoulos, N., 1990, Interoperability of Multiple Autonomous Databases, ACM Computing Surveys, 22(3), 267-293. Ma, Z. M., Zhang, W. J. and Ma, W. Y., 2000, Semantic Measure of Fuzzy Data in Extended Possibility-Based Fuzzy Relational Databases, International Journal of Intelligent Systems, 15 (8), 705-716. Mena, E., Kashyap, V., Illarramendi, A. and Sheth A., 2000, Imprecise Answers in Distributed Environments: Estimation of Information Loss for Multi-Ontology Based Query Processing, International Journal of Cooperative Information Systems, 9 (4), 403-425. Mendelzon, A. O. and Mihaila, G. A., 2001, Querying Partially Sound and Complete Data Sources, Proc. of the Fifteenth ACM SIGACT-SIGMODSIGART Symposium on Principles of Database Systems, 162-170. Morrissey, J. M., 1990, Imprecise Information and Uncertainty in Information Systems, ACM Trans. on Information Systems, 8 (2), 157-180. Parent, C. and Spaccapietra, S., 1998, Issues and Approaches of Database Integration, Communications of the ACM, 41 (5): 166-178. Reiter, R., 1986, A Sound and Sometimes Complete Query Evaluation Algorithm for Relational Databases with Null Values, Journal of the Association for Computing Machinery, 33 (2), pp. 349-370. Rolstadas, A., 1995, Enterprise Modeling for Competitive Manufacturing, International Journal of Control Engineering Practice, 3 (1), 43-50. Sheth, A. P. and Larson, J. A., 1990, Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases, ACM Computing Surveys, 22 (3), 183–236 Silberschatz, A., Stonebraker, M. and Ullman, J. D., 1991, Database Systems: Achievements and Opportunities, Communications of the ACM, 34 (10): 110-120. Tatarinov, I., Viglas, S., Beyer, K. S., Shanmugasundaram, J., Shekita, E. J. and Zhang, C., 2002, Storing and Querying Ordered XML Using a Relational Database System, Proc. of the 2002 ACM SIGMOD International Conference on Management of Data, 204 – 215. Tseng, F. S. C., Chen, A. L. P. and Yang, W. P., 1993, Answering Heterogeneous Database Queries with Degrees of Uncertainty, Distributed and Parallel Databases: An International Journal, 1 (3), 281-302. Yuan, L. Y. and Chiang, D. A., 1989, A Sound and Complete Query Evaluation Algorithm for Relational Databases with Disjunctive
192
Li Yan th
Information, Proc. of the 14 ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, 66-74. [42] Zaniolo, C., Faloutsos, S. and Subrahmanian, Z., 1997, Advanced Database Systems, Morgan Kaufmann Publishers. [43] Zhang, W. J. and Li, Q., 1999, Information Modeling for Made-to-Order Virtual Enterprise Manufacturing Systems, Computer Aided Design, 31 (10), 611-619.
Integrating Data Doubtfully Ander de Keijzer Faculty of EEMCS, University of Twente PO Box 217, 7500 AE Enschede, The Netherlands [email protected]
1 Introduction Currently, the number of available information sources is enormous and still growing. Businesses have databases containing product, client, supplier and payment information. Even at home, people have multiple information sources, e.g. address books on PDAs, mobile phones and desktops. All of these information sources contain information on related items, or objects. The user usually only requires the actual information without having to search for it, or be aware that the information is divided over several independent information sources. Integration the information sources would help the user in his or her task, because the information is presented as if it were contained in one information source. As an example, we will use two address books of Mark, who has two address books. One address book is located at home, while the other is located at work. Subsets of these address books are presented in Table 1. Table 1. Subsets of two address books Name Street Number Phone Ed King Fifth Avenue 10 555-1234 Mark Hamburg Central Square 5 555-4321 Joe Rough Broadway 442 555-2341 Name Address Areacode Phone Ed King Fifth Avenue 10 555 1234 Mark Hamburg Central Square 7 555 4321 Stan Choice Amsterdam Avenue 2 555 3412
For Mark, it is not important from which address book an address is taken, as long as it is correct, so the answer
A. de Keijzer: Integrating Data Doubtfully, Studfuzz 203, 193–217 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
194
Ander de Keijzer
Ed King
Fifth Avenue 10
555-1234
could have come from either of the address books. However, it is important that all addresses are available at any given time. 1.1 Data integration Without loss of generality, we assume only two information sources are used as input. There are two different methods to achieve data integration. The existing information sources can be integrated and the result can be stored as a different information source. This method is referred to as materialized integration. After integration, the original sources are no longer needed. Another approach is to integrate the information sources at query time, keeping the original information sources. These sources can be updated independently and can also be used by other applications as autonomous information sources. In this case, the integrated information source is a mediated view over the original sources. In de remainder of this chapter, we will only consider materialized integration. The data in the original information sources can be about the same real world object (rwo). However, this information can conflict. For example, the house number of Mark Hamburg in Table 1 differs in both address books. This conflict has to be resolved, before the information sources can actually be integrated. There are several reasons why the numbers differ. One of the entries could contain a type-error, or even both could contain type-errors. The person in the first address book could be a different one from the person in the second address book, but both with the same name (probably relatives). By just choosing one of possibilities from the address books, we could end up with an error in our end result. 1.2 Uncertain data Instead of choosing one of the possibilities from the original information sources for the resulting information source, we include all possibilities. However, we associate a probability with each of the probabilities to indicate the likelihood they will actually occur in the real world. If the associated probability is 0, the possibility is left out of the resulting information source. As a result, the integrated information source will contain uncertain data, but no data will be lost due to the integration process. 1.3 Outline The remainder of this chapter is organized as follows. In section 2 the concept of possible worlds is described. In section 3 probabilistic relation data is introduced as a means to store uncertain data. This model is extended to probabilistic XML, which is shown in section 4. Querying probabilistic data, is the topic of section 5.
Integrating Data Doubtfully
195
2 Possible worlds A possible world is a collection of all real world objects, where every rwo has one specific appearance. There is, for example, a possible world where Mark Hamburg has house number 5 and another possible world where his house number is 7. Every possible world pW has a probability with which it exists, denoted by P (pW ).
Fig. 1. Possible Worlds derived from different objects.
In the address book example, there is a total of 8 possible worlds, because there are 2 possibilities for Mark Hamburg, Joe Rough and Stan Choice, while there is just one possibility for Ed Kind. This results in 2 × 2 × 2 × 1 = 8 different worlds. The concept of possible worlds is illustrated in Figure 1, here two objects (indicated by squares), with each two possibilities, are represented in a total of four possible worlds (circles). A possible appearance of a real world object is called a possible real world object, or possibility. Note that the definition of possibility in this chapter, is different from other chapters.
3 Probabilistic relational data In [2] we defined real world objects, relations and their probabilistic counterparts as follows. A real world object can be represented by a tuple T ∈ D1 × · · · × Dn , where Di is the domain of attribute i. A set of real world objects can then be described by a relation R ∈ P(D1 × · · · × Dn ). Finally, we represent a real world by a database DB ∈ PP(D1 × · · · × Dn ). Without loss of generality, we assume DB to consist of one relation. A probabilistic attribute is a traditional attribute with an associated prob˜ i = [0, 1] × Di . We can now define a probabilistic ability, i.e., its domain is D ˜ n . pT defines a possible description of a real world ˜1 × · · · × D tuple as pT ∈ D object, also called a possibility. Let Pi (pT ) be the probability part of attribute i in tuple pT and πi (pT ) its value part. ˜1 × · · · × D ˜ n ) be a probabilistic relation. Table 3 shows an Let pR ∈ P(D example of a probabilistic relation. We require pR to have a primary key k (or key for short). Without loss of generality, we assume the key of a probabilistic
196
Ander de Keijzer Table 2. Probabilistic relation addressbook Name Address Ed King [1.0] Fifth Avenue 10 Mark Hamburg [1.0] Central Square 5 Central Square 7 Joe Rough [1.0] Broadway 442 Stan Choice
[1.0] [0.7] [0.3] [0.9] [0.1] [1.0] Amsterdam Avenue 2 [0.9] [0.1]
Phone 555-1234 [1.0] 555-4321 [1.0] 555-2341 [0.9] [0.1] 555-3412 [0.9] [0.1]
relation to consist of one attribute. Key k uniquely identifies the real world object. As a consequence, the associated probability needs to be 1. Observe that the key value itself is not unique in pR in the traditional sense (‘name’ in Table 3). For convenience, we introduce the following notation: Π v (pR) = {pT ∈ pR | πk (pT ) = v} Di pR = {v ∈ Di | (∃pT ∈ pR) • πi (pT ) = v} Π v (pR) is the set of all tuples from pR where the key equals v. Di pR is domain Di restricted to values occurring in pR. Two constraints should hold for a probabilistic relation:
( Pi (pT )) = 1 1. (∀v ∈ Dk pR) pT ∈Π v (pR) 1≤i≤n
2. (∀v ∈ Dk pR)(∀i = k)(∀pT , pT ∈ Π v (pR)) • πi (pT ) = πi (pT ) ⇒ Pi (pT ) = Pi (pT ) The first constraint states that for each real world object, the total probability of all possibilities equals one. This implies that we assume a closed world. The second constraint states that all attributes are independent. 3.1 Possible world model If we consider a normal relation R to be a representation of the real world, then we can consider a probabilistic relation pR to be a representation of several possible worlds. Since the probabilities of the individual attributes are independent, we obtain the probability of a description pT of a real world object (i.e., a possibility) by multiplying the probabilities of the associated attributes: Pi (pT ). 1≤i≤n
The set of tuples representing one possibility for each real world object is called a possible world pW ⊆ pR, defined by
Integrating Data Doubtfully
197
Table 3. Addressbook with distances (a) non-1NF representation name address distance ‘Mark Hamburg’ Central Square 5 [0.7] 50 Central Square 7 [0.3] 60 ‘Joe Rough’ Broadway 442 [0.9] 18 [0.1] 0
[0.7] [0.3] [0.9] [0.1]
(b) 1NF representation name ‘Mark Hamburg’ ‘Mark Hamburg’ ‘Mark Hamburg’ ‘Mark Hamburg’ ‘Joe Rough’ ‘Joe Rough’ ‘Joe Rough’ ‘Joe Rough’
address Central Square Central Square Central Square Central Square Broadway 442 Broadway 442
5 5 7 7
address prob distance 0.7 50 0.7 60 0.3 50 0.3 60 0.9 18 0.9 0 0.1 18 0.1 0
distance prob 0.7 0.3 0.7 0.3 0.9 0.1 0.9 0.1
(c) 3NF representation MAIN name id ‘Mark Hamburg’ 1 ‘Joe Rough’ 2
ROOM id 1 1 2 2
room prob Central Square 5 0.7 Central Square 7 0.3 Broadway 442 0.9 0.1
DISTANCE id 1 1 2 2
distance 50 60 18 0
prob 0.7 0.3 0.9 0.1
1. (∀v ∈ Dk pR) • |Π v (pW )| = 1 2. (∀pT ∈ pR)(∃pT ∈ pW ) • πk (pT ) = πk (pT ) The first constraint ensures that only one possibility is contained in pW for each real world object. The second constraint ensures that each real world object from pR is contained in pW at least once. The set of all possible worlds, also called universe, is denoted by PWR pR = {pW ⊆ pR | pW is a possible world} 3.2 Storage scheme We developed a prototype system on top of an ordinary RDBMS. Probabilistic tables can be stored in several ways corresponding with the normal form to which one likes to adhere. Table 3 shows different representations of the same probabilistic relation addressbook: a representation with non-atomic
198
Ander de Keijzer
attributes (Table 3(a)), the representation brought in first normal form (Table 3(b)), and the representation brought in third normal form (Table 3(c)). The 1NF representation of probabilistic data, however, may lead to massive replication of attribute values, hence, data inconsistency. The 3NF representation, on the other hand, may require many joins, hence, may perform less efficiently. We will discuss both representations. 1NF representation ˜ is not atomic, since it consists of a probability part A probabilistic attribute D ˜1 × and a value part. The representation of a probabilistic relation pR ∈ P(D ˜ n ) can be transformed to be represented only by atomic attributes. We · · ·×D write pR ∈ P([0, 1] × D1 × · · · × [0, 1] × Dn ). pR is now in 1NF. Probability attributes which have only values of 0 or 1 are omitted, since these are not probabilistic, which is the case for at least the key of the relation. If we bring Table 3(a) in 1NF, this results in 2 × 2 + 2 × 2 = 8 rows (see Table 3(b)). Each row has 5 attributes: name, room, room prob, distance and distance prob. 3NF representation The 1NF representation may massively replicate attribute values, hence may lead to data inconsistency if not handled properly. Normalization to 3NF prevents this. We start by observing that the probabilistic relation in Table 3(a) is mod˜ 2 × · · · × PD ˜ n . Each non-atomic attribute can, howelled as pR ∈ Dk × PD ever, be modelled as a separate table. Therefore, we introduce a main table for each probabilistic relation with the key and a unique identifier Dk (to avoid replicating the key value). Furthermore, we introduce a separate table Dk ×Di ×[0, 1] for each non-atomic attribute. The key of these tables consists of the first two attributes, k and i. The main table has functional dependency k → k and the separate tables have functional dependencies k i → p, where p is the probability attribute with domain [0, 1]. These functional dependencies satisfy 3NF. The 3NF representation of table 3 is shown in table 3(c). Dependent attributes The attributes room and distance in table 3 are independent, which means that any given distance can occur with any given address. This becomes extra clear with the 3NF representation where both attributes are stored in different tables. In our example however, the distance is likely to depend on the address. The 3NF representation can model this dependency by storing dependent attributes in the same table. The attributes address and distance in the main table are then replaced by a single attribute called address distance.
Integrating Data Doubtfully
199
4 Probabilistic XML As seen in the previous section, uncertain relational data has the following shortcomings • • • •
All probabilities must be independent Mutual exclusion between attributes is not possible All real world objects must exist in every possible world Ignorance cannot be modelled, i.e. a closed world is assumed.
We introduce uncertainty into the XML data model [8], resulting in probabilistic XML. We will show that most of the issues arising in the relational model can be solved by using probabilistic XML. 4.1 Formalization of probabilistic XML Since order is important in XML, we first introduce some notation for handling sequences. Notational convention 1 Analogous to the powerset notation P , we use a power sequence notation S A to denote the domain of all possible sequences built up of elements of A. We use the notation [a1 , . . . , an ] for a particular sequence. We use set operations for sequences, such as ∪, ∃, ∈, whenever definitions remain unambiguous. We start by defining the notions of tree and subtree as abstractions of an XML document and fragment. We model a tree as a node and a sequence of child subtrees. We abstract from the details of nodes. We assume that they incorporate properties like tag name, node identity, node kind, and, if appropriate, attribute or text value. Equality on nodes is defined as equality on the properties. Definition 1. Let N be the set of nodes. Let Ti be the set of trees with maximum level i inductively defined as follows: T0 = {(n, ∅) | n ∈ N } Ti+1 = Ti ∪ {(n, ST ) | n ∈ N ∧ST ∈ S Ti ∧(∀T ∈ ST • n ∈ N T ) ∧(∀T , T ∈ ST • T = T ⇒ N T ∩ N T = ∅)} where N T = {n} ∪ T ∈ST N T . Let Tfin be the set of finite trees, i.e., T ∈ Tfin ⇔ ∃i ∈ N • T ∈ Ti . In the sequel, we only work with finite trees. We sometimes use n to indicate the root node of a tree.
200
Ander de Keijzer 1
◦ • persons
V .7 kk VVVVV .3 VVVV kkkk k V◦M k ◦ qqq MMMMM q q personq•<< person •
nr • nm • nr • nm • nr • 7 5 7 Mark Mark
Fig. 2. Example probabilistic XML tree.
We obtain a subtree from a tree T by indicating a node n in T which is the root node of the desired subtree. We also define a convenience function child that returns the child nodes of a given node in a tree. Definition 2. Let subtree(T , n) be the subtree within T = (n, ST ) rooted at n. T if n = n • subtree(T , n) = subtree(T , n) otherwise
where T such that (T , ST ) ∈ ST ∧ n ∈ N T . For subtree(T , n) = (n, [(n1 , ST 1 ), . . . , (nm , ST m )]), let child(T , n) [n1 , . . . , nm ].
=
The central notion in this chapter is the probabilistic tree. In an ordinary XML document, all information is certain. When two XML data sources are integrated, they may conflict on information about certain real world objects. Therefore, after data integration, there may exist more than one possibility for a certain text node, or in general, for entire subtrees. We model this uncertainty in a probabilistic tree by introducing two special kinds of nodes: 1. probability nodes depicted as , and 2. possibility nodes depicted as ◦, which have an associated probability. The children of a probability node enumerate all possibilities with a combined probability of 1. Ordinary XML nodes are depicted as •. A probabilistic tree is well-structured, if the children of a probability node are possibility nodes, the children of a possibility node are XML nodes, and the children of XML nodes are probability nodes. In this way, on each level of the tree, you only find one kind of nodes. Figure 2 shows an example of a probabilistic XML tree. The tree represents an XML document with a root node ‘persons’ (which exists with certainty). The root node has either one or two child nodes ‘person’ (with probabilities .7 and .3, respectively). In the one-child case, the name of the person is ‘Mark’
Integrating Data Doubtfully
201
and the house number is either ‘5’ or ‘7’ (with equal probability). In the more unlikely case of two children, the information of both persons is certain, i.e., they both have name ‘Mark’ and one has house number ‘5’ and the other ‘7’. This document is a possible result of two documents having been integrated, one document stating the house number of a person named ‘Mark’ to be ‘5’, and the other stating the telephone number of a person named ‘Mark’ to be ‘7’. It is uncertain if both talk about the same person. A data integration matching rule apparently determined that, with a probability of .7, they represent the same person. Therefore, the combined knowledge of the real world is described accurately by the given tree. A probabilistic tree is defined as a tree, a kind function that assigns node kinds, and a prob function that assigns probabilities to possibility nodes. The root node is defined to always be a probability node. A special type of probabilistic tree is a certain one, which means that all information in it is certain, i.e., there is no more than one possibility for any node. Definition 3. A probabilistic tree PT = (T , kind, prob) is defined as follows • kind ∈ N → {prob, poss, xml} • NkT = {n ∈ N T | kind(n) = k }. • kind(n) = prob where T = (n, ST ) T T ∀n ∈ child(T , n) • n ∈ Nposs • ∀n ∈ Nprob T T • ∀n ∈ Nposs ∀n ∈ child(T , n) • n ∈ Nxml T T • ∀n ∈ Nxml ∀n ∈ child(T , n) • n ∈ Nprob T • prob ∈ Nposs → [0, 1] T • ∀n ∈ Nprob • ( n ∈child(T ,n) prob(n )) = 1. A probabilistic tree PT = (T , kind, prob) is certain iff there is only one posT • sibility node for each probability node, i.e., certain(PT ) ⇔ ∀n ∈ Nprob |child(T , n)| = 1. To clarify definitions, we use b to denote a probability node, s to denote a possibility node, and x to denote an XML node.
Subtrees under probability nodes denote local possibilities. In the oneperson case of Figure 2, there are two local possibilities for the house number, it is either ‘5’ or ‘7’. The other uncertainty in the tree are the possibilities that there are one or two persons. Viewed globally and from the perspective of a device with this data in its database, the real world could either look like • one person with name ‘Mark’ and house number ‘5’ (probability .5 × .7 = .35), • one person with name ‘Mark’ and house number ‘7’ (probability .5 × .7 = .35), or • two persons with name ‘Mark’ and respective house numbers ‘5’ and ‘7’ (probability .3). These are called possible worlds.
202
Ander de Keijzer .8 qqMMM.2 MMM qqq ◦<< << << nm • nr • nm • nr •
q ◦<<
1
◦
• Mark (a) PT 1
1
◦ • 5
1
◦
• Mark
1
◦ • 7
1
◦MMMMM M nm • • nr 1
◦
• Mark
< .2 << ◦ ◦ .8
• 5
• 7
(b) PT 2
Fig. 3. Probabilistic XML tree equivalence.
Definition 4. A certain probabilistic tree PT is a possible world of another probabilistic tree PT , i.e., pw(PT , PT ), with probability pwprob(PT , PT ) iff • PT = (T , kind, prob) ∧ PT = (T , kind , prob ) • T = (n, ST n ) ∧ T = (n, ST n ) • ∃s ∈ child(T , n) • child(T , n) = [s] • X = child(T , s) = child(T , s) • ∀x ∈ X • child(T , x ) = child(T , x ) • B = x ∈X child(T , x ) • ∀b ∈ B • PT b = subtree(PT , b) ∧PT b = subtree(PT , b) ∧pw(PT b , PT b ) • ∀b ∈ B • pb = pwprob(PT b , PTb ) • pwprob(PT , PT ) = prob(s) × b∈B pb The set of all possible worlds of a probabilistic tree PT is PWR pR PT = {PT | pw(PT , PT )}. A probabilistic tree is a compact representation of the set of all possible worlds, but there is more than one possible representation. The optimal representation is the one with the least number of nodes obtained through a process called simplification. Definition 5. Two probabilistic trees PT 1 and PT 2 are equivalent PT iff N 1 < PT = PWR PT . PT is more compact than PT if PWR 1 pR 2 1 2 PTpR N 2 . The transformation of a probabilistic tree to an equivalent more compact one is called simplification. Figure 3 shows an example of two equivalent probabilistic trees. They both denote the set of possible worlds containing trees with • two nodes ‘nm’ and ‘nr’ with child text nodes ‘Mark’ and ‘5’ respectively (probability .8) and
Integrating Data Doubtfully
personq•MMM q
qqq
◦
◦
nm • nm • nr • 5 Mark Marc (a) Independence
person •
MMM < << ◦ ◦ nr • 7
qqMMMMM M qqq q ◦< ◦< <<< <<<
203
< << ◦ ◦ person •
nm • nr • nm • nr • 5 7 Marc Mark (b) Dependence
(c) Uncertainty about existence
Fig. 4. Expressiveness of probabilistic tree model.
• two nodes ‘nm’ and ‘nr’ with child text nodes ‘Mark’ and ‘7’ respectively (probability .2). It is obvious that in Figure 3, PT 2 is a simplification of PT 1 . As mentioned earlier, relational approaches often disallow dependencies among attributes. The higher expressiveness of the probabilistic tree makes such a restriction unnecessary. Figure 4 illustrates three common patterns: independence between attributes (Figure 4(a)): any combination of ‘nm’ and ‘nr’ is possible, dependency between attributes (Figure 4(b)): only the combinations ‘Mark’/‘5’ and ‘Marc’/‘7’ are possible, and uncertainty about the existence of an object (Figure 4(c)): one possibility is empty, i.e., has no subtree. These patterns can occur on any level in the tree, which allows a much larger range of situations to be expressed.
5 Querying uncertain data For relational probabilistic databases, several extensions to relational algebra to support uncertain data have been proposed [1; 7; 4; 6; 5]. Based on [9], we argued in [2] that thinking in terms of possible worlds is powerful in determining a proper semantics of queries. Uncertainty can be treated as having more than one possible instantiation describing a particular real world object. Choosing one possible instantiation, or possibility for short, for each real world object, results in a possible world. Analogous to the notion of parallel universes, all possible worlds co-exist in the database and a query should, therefore, be evaluated in every possible world separately. This approach is not specific to relational databases, so we have adopted it for probabilistic XML as well [8]. 5.1 Querying relational data A probabilistic query should return a different kind of result than a normal relational query. Consider a query such as the one posed below on the addressbook of Table 3.
204
Ander de Keijzer
SELECT name, address FROM addressbook WHERE name=’Mark Hamburg’ Seen from a traditional relational point of view, the answer Mark Hamburg Mark Hamburg
Central Square 5 Central Square 7
555-4321 555-4321
makes no sense: because attribute name is a key, only one tuple containing the name ‘Mark Hamburg’ is expected. From the perspective of probabilistic relations, however, the returned result is a somewhat correct answer, because uncertainty about addresses in the data will inevitably lead to uncertainty in the answer to a question about addresses. The answer only does not specify the probabilities, which it should. This calls for a review of the semantics of relational algebra operators and, as a consequence, their redefinition, in order to support querying on probabilistic data. With the possible world approach, we would expect the answer [0.7] [0.3]
Mark Hamburg Mark Hamburg
Central Square 5 Central Square 7
555-4321 555-4321
which indicates that there is a 70% chance the possibility (‘Mark Hamburg’, ‘Central Square 5’) is correct and a 30% chance the possibility (‘Mark Hamburg’, ‘Central Square 7’) is correct. Operators Select The traditional select operator σc (R) selects tuples from R according to the selection condition c. For a probabilistic select operator, the selection condition c can also refer to probabilities of attributes. We, therefore, introduce a function P (f ) which returns the probability of attribute f . The semantics of the probabilistic select operator σc (pR) should be that it selects possibilities from pR. Note, however, that the result of a selection corresponds to a different universe than the one stored in the database. All probabilities of attributes f in the answer are of the form P (f |c). For example, to obtain data that is highly likely from the addressbook relation, highly likely being a probability more than 60%, we could write σP (address)>0.6 (addressbook). The result of this expression is {(1.0, ‘Mark Hamburg’, ‘Central Square 5’)}, which states that in the universe where a probability of a room is higher than 60%, Mark Hamburg certainly has the address Central Square 5, hence the probability 1.0 in the answer. In other words, the original probabilities have to be normalized in the result. Note that this is a proper probabilistic relation according to Section 3. Furthermore, observe that dependency among attributes may occur.
Integrating Data Doubtfully
205
Project The traditional project operator πf (R) selects all attributes f from all tuples in R leaving out other attributes not in f . The semantics of a probabilistic project operator πf (pR) should be that it selects all attributes f from all possibilities in R. Since all attributes are independent, projection does not affect the associated probabilities. However, if the key of pR is projected out, the individual objects can no longer be identified and the result is meaningless. Therefore, k always has to be part of attribute list f . Table 4. Determining the probabilities associated with a union result by enumerating possible worlds (P˘ = 1 − P ). A e∈A e∈A e∈ /A e∈ /A ⇒
[PA ] [PA ] [P˘A ] [P˘A ]
B e∈B e∈ /B e∈B e∈ /B
A∪B [PB ] e ∈ A ∪ B [P˘B ] e ∈ A ∪ B [PB ] e ∈ A ∪ B [P˘B ] e ∈ / A∪B
[PA PB ] [PA P˘B ] [P˘A PB ] [P˘A P˘B ]
e ∈ A ∪ B [PA + PB − PA PB ] e∈ / A ∪ B [1 − (PA + PB − PA PB )]
Union The probabilistic union operator merges two probabilistic relations possibly containing possibilities for the same real world objects. To properly calculate the probabilities in the answer, it is beneficial to enumerate the possible worlds, i.e., consider each possibility of an element existing or not in the operand sets (see Table 4). The probability of the element occurring in the result is the sum of the three possibilities where the element occurred in either operand. The intersection and difference can be determined analogously. Cartesian product The semantics of a cartesian product is the simultaneous occurrence of two possibilities. Probabilities can be calculated according to this semantics. Aggregate functions Aggregate functions combine the values of a set of attribute values into one value. Examples are the total price of a collection or the average mark of students. Observe that a traditional aggregate operator works in one world
206
Ander de Keijzer
where everything is certain. To come up with a proper semantics for aggregate operators in the context of probabilistic relations, we use the strategy of enumerating possible worlds. For each possible world pW with associated probability P (pW ), we apply the traditional aggregate operator aggr . The results (aggr , P (pW )) together form the resulting probabilistic relation. Table 5. Partial Addressbook with distances Name Address Distance Mark Hamburg [1.0] Central Square 5 [0.7] 50 Central Square 7 [0.3] 60 Joe Rough [1.0] Broadway 442 [0.9] 18 [0.1] 0
[0.7] [0.3] [0.9] [0.1]
Given the partial addressbook relation of Table 5 and the query below: SELECT MAX(distance) FROM addressbook This would traditionally return the number 60. This is, however, only true for the world where ‘Mark Hamburg’ has the address ‘Central Square 7’, but there are three other possible worlds. These possible worlds should contribute to the result. The correct result {(0.63, 50), (0.27, 60), (0.07, 50), (0.03, 60)} = {(0.7, 50), (0.3, 60)} reflects the existence of four possible worlds, two possibilities for both ‘Mark Hamburg’ and ‘Joe Rough’, each with its own maximum distance either 50 or 60. The associated probabilities of worlds with equal maximum values can be combined as shown. In more general terms, the probabilistic aggregate operators (MAX, MIN, SUM, AVG) are defined by aggr f (pR) ∈ P([0, 1] × R) where f indicates a field name. We use the notation aggr f (pW ) for the traditional counterpart of an aggregate operator evaluated in possible world pW . The aggregate function aggr f is defined by aggr f (pR) = {(P (pW ), aggr f (pW )) | pW ∈ PWR pR } A more elaborate explanation of the aggregate functions can be found in [3].
Integrating Data Doubtfully
207
EXP function All aggregate operators take the existence of possible worlds into account. However, the system should be able to predict information about the real world. We, therefore, introduce a new aggregate function, EXP , which returns the expected value of a numerical field. EXP f (pR) ∈ R It is defined by
πf (pT ) × Pf (pT )
pT ∈pR
Pf (pT )
pT ∈pR
The function calculates the weighted average of the field over all possible worlds. In the presence of a GROUP BY clause, we assume pR to represent one group. For example, SELECT EXP(distance) FROM addressbook GROUP BY name returns {53, 16.2}, the expected distances to all persons in our addressbook. EXP can be used in combination with other aggregate functions. The expected maximum distance is obtained by SELECT EXP(MAX(distance)) FROM addressbook 5.2 Querying XML data The parallel universes analogy illustrates that a query on uncertain data may produce a set of possible answers indicating that the answer is uncertain as well. One obtains one possible answer per possible world. The probability of a possible answer is simply the probability of the possible world that gave rise to it. In short, the correct answer to a query on a probabilistic XML document can be obtained as follows: 1. enumerate all n possible worlds for the probabilistic tree, 2. evaluate the query for each possible world according to ordinary semantics, 3. the resulting n possible answers can be simplified by merging (partially) equal possible answers.
208
Ander de Keijzer
S qq SSS.3SSS q.35 q q S◦ ◦q ◦
.7 qqMMM.3 MMM qqq
.35
/.-, ()*+ seq 1
◦
nr • 5
/.-, ()*+ seq
/.-, ()*+ seq < <
1
1
◦
nr • 7
◦
nr • 5
(a) Query answer
1
◦
nr • 7
◦q
◦
/.-, ()*+ seq
/.-, ()*+ seq < <
< .5 << ◦ ◦ .5
nr • 5
nr • 7
1
◦
nr • 5
1
◦
nr • 7
(b) Compact answer
Fig. 5. Query //person[nm="Mark"]/nr.
This approach applies to queries in any query language. In XPath or XQuery, answers to queries are sequences. Queries on uncertain data, hence, result in sets of possible sequences. Assuming that a sequence is represented /.-, ()*+ by a special sequence node seq , a probabilistic tree can be used to represent the set of possible answers and simplification can be used to obtain a compact representation of the answer. Note that this is just a statement about the semantics of a query. An implementation can obviously use a more efficient algorithm for answering queries than by enumerating all possible worlds. Concerning query evaluation, our prototype implementation focuses on obtaining insight into how and to what degree possibilities in the data propagate to possibilities in the answer. It, therefore, simply follows the three-step strategy presented above. Performance of query evaluation is a topic for future research and, hence, beyond the scope of this chapter. We only sketch how the prototype enumerates all possible worlds in Subsection 5.2 below. Figure 5 shows an answer to an example query on the data of Figure 2. We have seen that there are three possible worlds. Evaluating the query in each of these possible worlds gives rise to the answer in Figure 5(a). Intuitively, it states that the answer to house numbers of people named “Mark” is uncertain: it is either “5” or “7” or there are actually two Marks with a house number: one with “5” and one with “7”. Figure 5(b) shows how we can compactly represent this answer by merging the first to possibilities, i.e., making the possibilities local, since in both cases it is certain that the sequence contains one element. Note that to applications, it may be more logical to present the answer to a query as a set of possible answers where each answer is an ordinary XML sequence with ordinary non-probabilistic XML fragments as elements. Such a representation coincides with an enumeration of the possible worlds of the answer. On the other hand, an application prepared for consuming probabilistic answers may benefit in performance by the compact representation. The design of an API is still an open issue.
Integrating Data Doubtfully
209
Enumerating possible worlds Generating all possible worlds from a compact probabilistic XML tree can be done recursively. The approach of our prototype implementation is as follows. Given a node in the tree, the function produces all possible worlds for the subtree rooted at that node. Contrary to the formalization, a possible world is represented by a normal XML tree instead of a certain probabilistic tree, i.e., it contains only XML nodes and no probability and possibility nodes. For producing the set of possible worlds for a certain node, the sets of possible worlds from its children need to be combined. There are two ways to combine sets of possible worlds: = • The product of two sets of possible worlds W1 = {B1 , . . . , Bm } is W1 ⊗ W2 = {A1 , . . . , An } and W2 {A1 B1 , A1 B2 , . . . A1 Bm , A2 B1 , A2 B2 , . . . An Bm }, where AB means a world in which both A and B hold. For example, A and B may correspond to two distinct children of a certain element, hence, a possible world combines one possibility for each child. = • The sum of two sets of possible worlds W1 ⊕ W2 {A1 , . . . , An , B1 , . . . , Bm }. Let W1 , . . . , Wn be the sets of possible worlds for each child of a given node. The resulting set of possible worlds for that node depends on the node kind: • Probability node. The result is W1 ⊕ . . . ⊕ Wn . • Possibility node. The result is W1 ⊗ . . . ⊗ Wn . • XML node. The result is constructed by placing the given XML node as parent (i.e., root) in each world in W1 ⊗ . . . ⊗ Wn .
6 Data integration In this section we will show how probabilistic XML can be used to perform data integration. 6.1 General approach A device’s database is a probabilistic XML document. When data integration with a foreign probabilistic XML document is initiated, the foreign document is considered to be a source of ‘new’ information on real world objects the device either already knows about or not. New information on ‘new’ real world objects is simply added to the database. Any differences in information on ‘existing’ real world objects are regarded as different possibilities for that object. Note that we disregard possibilities concerning order. New information on ‘new’ real world objects is simply considered to come after information on known objects in document order.
210
Ander de Keijzer
Since it is often not possible to determine with certainty that two specific XML elements correspond to the same real world object, we assume the existence of a rule engine that determines the probability of two elements referring to the same real world object. In special cases, this rule engine may obviously decide on a probability of 0 (with certainty not the same real world object) or 1 (with certainty the same real world object). In this chapter, we abstract from the details of this rule engine, but imagine that it uses schema information to rule out possibilities. Or it may, for example, consult a digital street map to declare a certain street name very improbable as there exists no such street in that city. Or it may use Semantic Web techniques to reason away possibilities. In our current prototype, the rule engine is very simple and uses only some basic schema information. It distinguishes two cases: • The schema states that a certain element can appear only once. We assume that this means that the elements of both documents refer to the same real world object, hence, the subtrees are correspondingly merged. For example, if two person elements refer to the same real world person, their descendant elements (e.g., nm and tel) are merged. If two corresponding descendant elements differ, we store this as two possibilities for that element. • The schema states that a certain element can appear multiple times. We assume that this means that the foreign document may contain new elements for this list. For example, the database contains knowledge about two persons “Mark” and “Joe”. The foreign document holds information on a person “Marc”. Note that “Marc” may be the same person as “Mark” only misspelled, or it may refer to a different person. The data integrator will store both possibilities, i.e., one whereby it merges “Mark” and “Marc”, and one whereby it adds a new person element. Each possibility is assigned a probability by the rule engine. For example, it is not unthinkable that “Marc” and “Mark” are actually the same person. On the other hand, it is rather improbable that “Joe” and “Marc” are the same person. Our current prototype is not that clever though, i.e., it assigns the same probability to the likelihood that two person elements refer to the same real-life person as to the likelihood of both referring to different real-life persons. The reason for this is that we need it as a base line to determine the effectiveness of a more clever rule engine. Moreover, we would like to focus on the data integration mechanism first. The minimal set of rules used by our prototype also includes that there can only be one root in an XML document and schema’s of integrated documents are the same, so different tag names are assumed to refer to different real world objects. 6.2 Integrating sequences In general, integrating sequences produces possibilities for all elements referring to either the same or different real world objects. Since we made an
Integrating Data Doubtfully
211
Table 6. Possibilities for merging sequences x = {A, B} and y = {C, D} Referral to real world object resulting sequence A = B = C = D A, B, C, D A = C, B = C = D A/C, B, D A = D, B = C = D A/D, B, C A = C = D, B = C A, B/C, D A = C = D, B = D A, B/D, C A = C, B = D A/C, B/D A = D, B = C A/D, B/C
assumption that the schemas are the same and that elements with different tag names refer to different real world objects, many of those possibilities are ruled out. However, this rule does not limit the possibilities for sequences of elements with the same tag name. Take for example, the integration of address information of people. We are confronted with integrating sequences of person elements. Because we initially chose for a rather dumb rule engine, any two elements, one from each sequence, possibly refer to the same real world object. Therefore, when merging two sequences, X and Y , the resulting number of possibilities can be huge. Let, for example, X = [A, B] and Y = [C, D]. The possibilities to be generated during integration of X and Y are listed in Table 6.2. In the table, A = C indicates that A and C are considered to refer to the same real world object, hence, they should result in a single possibility where A and B are merged: A/B. Since the database already represents all possibilities explicitly, we do not need to consider two elements from one sequence to refer to the same real world object, so A = B and C = D are not valid possibilities. When all elements of X and Y refer to other real world objects, the number of resulting possible worlds is 1. But, when one element from X refers to the same real world object as an element from Y , there are X × Y possible ways how this can be done, since every element from X can in principle be matched with every element from Y . In general, the number of possible ways to merge i elements from X with i elements from Y can be computed as follows. Choose i different elements from X, where the order of choosing the elements is unimportant, but an element x! cannot be chosen more than once. This can be done in (x−i)!i! = xi ways. Then, we choose i elements from Y to merge with those chosen from X. Since the first chosen element from X should be merged with the first element of Y , order is important when choosing elements from Y . The number of ways y! . to choose the i elements from Y is (y−i)! Let x and y be the lengths of X and Y , respectively. The process of merging sequences is commutative, we assume x ≤ y. In determining all possibilities, any i (0 ≤ i ≤ x) elements of X may refer to the same real world object as elements of Y . Therefore, the resulting total number of possibilities for a
212
Ander de Keijzer
merged sequence is
x
x i=0
i
×
y! (y − i)!
We see from this formula, that merged documents can become huge quite rapidly. If we take, for example, x = 5 and y = 5, then the maximum number of possibilities is 1546. The rule engine, however, may rule out certain possibilities. For example, if in the case of Table 6.2, A refers to a person named “Mark” and C to a person named “Joe”, the rule engine may assign probability 0 to the likelihood that A = C. In this way, it rules out two of the seven possibilities. 6.3 Data integration is an equivalence preserving operation An interesting property of the data integration approach described above, is if it preserves equivalence. Let D1 and D2 be two probabilistic XML documents and A = integrate(D1 , D2 ) the result of integrating them. Suppose D1 and D2 are equivalent to D1 and D2 respectively. Is then A = integrate(D1 , D2 ) equivalent to A? A full proof goes beyond the scope of this chapter, but below we try to show that it is plausible. There is a special case for which this property is especially interesting. The set of possible worlds can be represented as a probabilistic tree with one probabilistic node as root and all possible worlds as possibilities directly below it. Figure 3(a) is of this form. Since the above property holds, integrating two probabilistic trees amounts to integrating all combinations of possible worlds of both trees. Below, we first show an example of integrating two certain trees to illustrate the recursive process. We then show the integration of a compact tree with two possibilities with a certain tree. Finally, we show the integration of an equivalent tree in set-of-possible-worlds representation with the same certain tree. The data integration function integrate takes two parameters D1 and D2 . It returns the integration result as a probabilistic XML tree. In the diagrams below, we have omitted probability and possibility nodes whenever there is only one possibility. The example below shows how we can recursively integrate two certain trees. integrate() < << person • • person Mark •
• Joe
After the first integration step, we obtain:
Integrating Data Doubtfully
213
SSSS SSSS kkkk k k k ◦
• Joe
integrate() << <• Joe Mark •
The second integration step integrate(’Mark’, ’Joe’) obviously results in: < << ◦ ◦ Mark •
• Joe
Observe the difference between integrating person elements, which are specified as being part of a sequence, and other elements including text nodes for which there can only be one. The former produces an additional possibility for the case that there exist two persons. In general, text nodes are also part of a sequence (e.g., paragraphs in a text document). Concatenating names of persons, however, does not make sense, so our rule engine decides that, for example, the name of a person can not be “MarkJoe”. Integrating a probabilistic tree and a certain one proceeds as follows: integrate() MMM qq M person person •qq • < << ◦ ◦
• Joe
Mark •Marc • We would first integrate both person elements: kkVVVVVVVV VVV kkkk k k ◦ ◦MMMMM M person person • person • • integrate() MMM qqq M q • << < Joe ◦ ◦ • • Mark Marc where
< << ◦ ◦ • • Mark Marc
• Joe
214
Ander de Keijzer
integrate() MMM qqq M q • << < Joe ◦ ◦ • • Mark Marc intuitively leads to qMMMM M qqq q ◦ ◦ ◦ • • • Mark Marc Joe The entire resulting tree looks like: kkVVVVVVVV VVV kkkk k k ◦ ◦MMM MMM person • person • • person qMMMM M qqq q ◦ ◦ ◦
< << ◦ ◦
• • • Mark Marc Joe
• • Mark Marc
• Joe
The resulting tree can be described as: (Joe ∨ Mark ∨ Marc) ∨ ((Mark ∨ Marc) ∧ Joe)
(1)
Moving the local possibility upwards in the tree, we get an equivalent less compact tree that is in all-possible-worlds representation. The integrate function now behaves as being applied to each possible world separately. MMM hintegrate() M person hhhhh h h • M q MMM q q M q q • ◦ ◦ Joe person • • person • Mark
• Marc
We integrate the person named “Joe” over both possibilities resulting in the following:
Integrating Data Doubtfully
215
hVVVVVV VVVVV hhhhh h h h h ◦h ◦ integrate() < << person • • person
integrate() < << person • • person
• • Mark Joe
• • Marc Joe
The final result is: ffffqMYMYMYMYYYYYYYY fffffqqqq YYYYY f M f f f ◦
• • Marc Joe
person •
person •
< << ◦ ◦
< << ◦ ◦
• • Mark Joe
• • Marc Joe
The boolean representation is ((Mark ∧ Joe) ∨ (Mark ∨ Joe)) ∨ ((Marc ∧ Joe) ∨ (Marc ∨ Joe))
(2)
Note that this is equivalent to the earlier obtained boolean representation. The trees are equivalent. 6.4 Assigning probabilities and confidence scores As explained earlier, the rule engine determines in some way the probability of the various possibilities. Intelligence and the use of information from schema and other data sources can be used to limit the number of possibilities, but also to better assign probabilities. The most simple scheme for assigning probabilities would be the following. Whenever the database conflicts with a foreign document one some element, we assign probability .5 to each resulting possibility. This approach has a severe drawback. For example, when a mobile device has once heard from another device that a person’s name is “Mark” and it meets another device which says its name is “Marc”, the scheme assigns the possibilities “Mark” and “Marc” each a probability of .5. But, when it has already heard from ninetynine different devices, that the name of the person is “Mark”, it should be very suspicious when it meets a device that says this person’s name is “Marc”. Therefore, it should give the possibility “Marc” a very small probability of, say, .01.
216
Ander de Keijzer
The rather basic rule engine of our prototype has a still simple but sufficient scheme for assigning probabilities. It is based on the premise that what you have seen twice, is twice as likely to be correct. In other words, a confidence score should be kept in the data. This factor is an indication of how certain we are about the data, which helps in assigning probabilities when integrating new data. Every time a different device claims a certain possibility is true, the confidence score is increased by one.
7 Conclusions In this chapter we described a probabilistic data model to store relational data. The query language was extended to support the probabilities associated with attributes using the possible world approach. Also a new aggregate function, EXP was introduced. This function calculates the expected value of a numerical field and can be considered a bridge between possible worlds and real world. With this relational model, however, we were unable to model mutual exclusion, dependent probabilities, open worlds or possible worlds with different real world objects. To overcome some of the problems faced in the relational model, the probabilistic model was adapted to be used with XML. With XML mutual exclusion, dependent probabilities and possible worlds with different real world objects can be modelled. Open worlds could easily be modelled using XML, however, querying the tree of an open world is still an open issue.
References [1] Daniel Barbar´ a, Hector Garcia-Molina, and Daryl Porter. A probabilistic relational data model. In Fran¸cois Bancilhon, Costantino Thanos, and Dennis Tsichritzis, editors, Advances in Database Technology - EDBT’90. International Conference on Extending Database Technology, Venice, Italy, March 26-30, 1990, Proceedings, volume 416 of Lecture Notes in Computer Science, pages 60–74. Springer, 1990. [2] Ander de Keijzer and Maurice van Keulen. A possible world approach to uncertain relational data. In Supporting Imprecision and Uncertainty in Flexible Databases, 2004. [3] Ander de Keijzer and Maurice van Keulen. A probabilistic database extension. Technical report, CTIT, TR-04-21, 2004. [4] Debabrata Dey and Sumit Sarkar. A probabilistic relational model and algebra. ACM Trans. Database Syst., 21(3):339–369, 1996. [5] Thomas Eiter, Thomas Lukasiewicz, and Michael Walter. A data model and algebra for probabilistic complex values. Annals of Mathematics and Artificial Intelligence, 33(2-4):205–252, 2001.
Integrating Data Doubtfully
217
[6] Norbert Fuhr and Thomas R¨ olleke. A probabilistic relational algebra for the integration of information retrieval and database systems. ACM Transactions on Information Systems, 15(1):32–66, 1997. [7] Laks V. S. Lakshmanan, Nicola Leone, Robert Ross, and V. S. Subrahmanian. ProbView: a flexible probabilistic database system. ACM Transactions on Database Systems, 22(3):419–469, 1997. [8] M. van Keulen, A. de Keijzer, and W. Alink. A probabilistic XML approach to data integration. In Proceedings of the International Conference on Data Engineering, ICDE 2005, 2005. [9] E Zimanyi and A Pirotte. Imperfect information in relational databases. Uncertainty Management in Information Systems, A. Motro and P. Smets, Eds., 1997.
An Overview of Imperfection Representation in Semistructured Data Matteo Magnani1 and Danilo Montesi2 1
2
Department of Computer Science, University of Bologna, Mura Anteo Zamboni 7, 40127 Bologna, Italy [email protected] Department of Mathematics and Informatics, University of Camerino, Via Madonna delle Carceri 9, 62032 Camerino MC, Italy [email protected]
1 Introduction Today, many important database applications have to deal with data that is both semistructured and either imprecise or uncertain. As an example, scientific databases are likely to contain data affected by some types of imperfection, with a schema that is not well-defined a priori [6, ch.10]. A different area with similar features is that of structured information retrieval, which has become very popular after the spread of XML documents [15]. As a consequence, several models for imperfect semistructured data have been proposed by the scientific community [22; 2; 13; 12; 18]. These models are similar in spirit to the probabilistic extensions of the relational model developed in the Nineties [3; 26; 9; 11; 17]. In particular, probabilities have been attached to nodes instead of tuples or attribute values. As we will show in this chapter, this attitude is useful to model only a few kinds of imperfection, because of the greater complexity of semistructured data. Many kinds of imperfection, such as uncertainty and imprecision, may coexist in the same data instance, and may affect several data representation dimensions. Data representation dimensions are the constituents of a model used to represent data, e.g., nodes, arcs, order, contents and labels in semistructured data. In the first part of the chapter we provide a classification of possible dimensions of imperfection that are likely to affect semistructured data. Dimensions of imperfection are obtained by combining data representation dimensions with relevant kinds of imperfection. For example, the content of a node may be affected by uncertainty, as well as imprecision, labels can be missing, and so on. Afterwards, we use our classification to review existing models for imperfect semistructured data. In particular, we focus on the representation of imperfection, and not on its manipulation. The reader interested in query languages for imperfect XML data will find some useful publications in the area of structured information retrieval (IR). However, these languages M. Magnani and D. Montesi: An Overview of Imperfection Representation in Semistructured Data, Studfuzz 203, 221–239 (2006) c Springer-Verlag Berlin Heidelberg 2006 www.springerlink.com
222
Matteo Magnani and Danilo Montesi
are used to manipulate data that is not imperfect, therefore we do not address them in this survey. The objective of this contribution is not the definition of a new model, but a preliminary analysis that should be performed before the definition of new models. In the aforementioned proposals, a formalism to represent imperfection is chosen a priori, without clarifying what kind of imperfection it is supposed to manage. On the contrary, we claim that we should first understand how data can be affected by imperfection, that is the aim of this chapter, and only subsequently we can choose the most appropriate formalism(s) to manage it. The main motivation of this contribution is the need for a clear understanding of how imperfection can affect semistructured data. To this aim, in Sect. 2 we discuss existing taxonomies of imperfection. Knowledge about different kinds of imperfection is very important, as they usually require the application of different methods and formalisms [29]. Then, we reconsider the identified classes of imperfection in the context of data management. In particular, in Sect. 3 we define a general data structure to specify data representation dimensions for semistructured data. The definition of a data structure is necessary, because we cannot understand how information is missing without defining what information can and cannot be represented by our model. However, it is general enough to consider our results extensively valuable, simple to understand, and its formal definition allows us to clearly identify its data representation dimensions. In Sect. 4 we combine classes of imperfection and data representation dimensions, to identify possible dimensions of imperfection in semistructured data. In particular, in Sect. 4.1 we check which types of imperfection can locally affect each data representation dimension, e.g., labels, nodes, and arcs. Then, in Sect. 4.2, we identify possible relationships between different dimensions of imperfection. We conclude Sect. 4 with an example, explaining why previous work on the relational model has identified only one main dimension of uncertainty, determining a similar approach also on early proposals for semistructured data models. Finally, Sect. 5 is devoted to the comparison of existing models for imperfect semistructured data, using our classification. We conclude the chapter highlighting the results of our investigation. Part of this chapter is an extended version of [19].
2 A taxonomy of imperfect information When we do not know everything we would like or need to know about a real world event, we are ignorant, and our knowledge is imperfect. Ignorance is a state of incomplete knowledge. In particular, ignorance and knowledge bound each other, and information may be viewed as a reduction of ignorance [16]. When we deal with complex systems, ignorance is often unavoidable, and it is sometimes necessary, to reduce the complexity of information. When data
An Overview of Imperfection Representation in Semistructured Data
223
is affected by ignorance, we say it is imperfect. In the following we will use imperfect data and data affected by ignorance as synonyms. Parsons points out that the importance of a taxonomy is not to give a definitive characterization of imperfection — that he calls uncertainty, but to allow distinctions to be drawn [24]. Saying it in another way, a classification is not right or wrong, but it can be more or less useful. Similarly, Smets states that a full inventory of types of ignorance is not practically possible [28]. In the literature, many taxonomies have been proposed, and we can notice a general agreement on some high-level classes [4; 30; 21; 8; 28; 23]. At a first glance, these taxonomies may seem incompatible, but this is mainly due to linguistic ambiguities. For example, we may say that we are certain that Tom is either 12 or 13 years old. However, at the same time we are not certain about Tom’s age: Is he 12 or 13? In fact, the term uncertainty has been used to indicate different kinds of imperfection in different taxonomies. In the presence of such ambiguities, we need a characterization of imperfection that is as independent of natural language semantics as possible. This can be done in many different ways [10], and we will use a set based approach, that is probably the most typical choice. As we have already said, ignorance can be thought as lack of knowledge. Therefore, the way in which knowledge is missing may be used as a principle to classify it. To define knowledge, we must assume the existence of someone (or something) who knows. We call it a cognitive being. A cognitive being can believe in something, that we may call events, ideas, propositions, or in general beliefs, depending on the context. A taxonomy of ignorance depends on the way in which we define what a belief is. Many alternative definitions are possible, and their effectiveness depends on the number and kind of classes it can discriminate. Throughout this chapter we will use the following scenario, illustrated in Fig. 1. Perfect information is defined to be an element of a set U of mutually exclusive elementary events (∀e1 , e2 ∈ U ¬(e1 ∧e2 )). In the following, we will call this element t. For example, if knowledge concerns the color of a monochrome object, U can be the set {black, blue, yellow,. . . } whose elements are all the colors listed in some English dictionary. Notice that we could also use the set of all RGB colors3 — this depends on the application requirements. Cognitive beings do not ever know all possible alternative events. Based on the subset of U known by the cognitive being, that we call KU, we can represent knowledge as an assignment of some quantified belief to a finite set of propositions built over KU. For example, knowledge corresponding to “The object is either blue or yellow” is represented by proposition p1 = {blue, yellow}, and we can believe p1 to be possible, certain, or probable. Other examples of propositions are p2 = {green} and p3 = dark — in this case, p3 can be represented using a fuzzy set. Different kinds of imperfection are characterized by different set3
An RGB color is defined by three integers, representing the amount of red, green and blue.
224
Matteo Magnani and Danilo Montesi
Fig. 1. A scenario for the classification of ignorance. A cognitive being (CB), whose knowledge is described by this scenario, assigns some quantified belief (QB) to a finite set P of propositions, built over a finite set KU (Known Universe) of elementary events. U (Universe) is the set with all possible alternative elementary events.
tings of the constituents of knowledge, i.e., quantified beliefs and propositions, KU and U. Quantified beliefs (Bel) are functions over the finite set of propositions P to an ordered set of values representing ascending degrees of belief. When some belief is assigned to a proposition p, we say that p is a supported proposition, and we call the union of the supported propositions the Core C of a belief function. As an example of beliefs, we may consider the set {impossible, unlikely, probable, sure}, the interval [0, 1], or the interval [0, 100]. The range may vary depending on formalisms and applications, and we will use 0Bel and 1Bel to indicate the smallest and highest values of a function Bel. The kind of quantified belief is defined by constraints on P and on the beliefs that can be assigned to its elements. Depending on the interpretation of these functions, quantified beliefs can represent subjective probability, possibility, necessity, belief, and plausibility, that are real-world modalities of belief and should not be confused with the mathematical theories with the same names [10; 27]. In this chapter we do not deal with these theories, because our analysis is preliminary to their choice. The data models discussed in the second part of the chapter use probability theory, without specifying any interpretation. Therefore, we will focus on a general kind of quantified belief that assigns to each proposition a degree of certainty, ranging from “not believed” to “completely believed, or certain”. One of the most typical kinds of imperfection is characterized by partial beliefs: The cognitive being assigns to some proposition p only part of the maximum amount of belief we can assign to a single proposition, that would indicate certainty. This can be due to several reasons: we may not trust the source of information supporting p, or p may represent a forecast, or p may derive from a generalization of past events (induction), just to mention a
An Overview of Imperfection Representation in Semistructured Data
225
few examples. For instance, we may say that “Perhaps, John is grey-haired”. This kind of imperfection is usually called uncertainty (UN), but this term should be used carefully, because it is often intended in a more general sense. Uncertainty is independent on the kind of proposition to which the belief is assigned, but is related to our trust in it. More formally, we have uncertainty when: ∃p ∈ P | 0Bel < Bel(p) < 1Bel . If no belief is assigned to any proposition, we have absence (ABS). A very unusual and extreme kind of absence corresponds to KU = ∅, where we cannot even formulate hypotheses (define propositions) because we do not know any possible elementary events. Absence is sometimes called incompleteness, but this term is also used with a more general meaning to indicate “incomplete information”, and thus ignorance. A less strict kind of absence corresponds to the situation where we know some events, but we do not believe in any of them — however, we would be able to assign some beliefs. For example, “We do not know the color of John’s hair”. When we know that KU = U, there is no difference in assigning no belief at all, or assigning all the belief to KU. This case is in the middle between absence and non-specificity, that we introduce in the next paragraph, and it is frequently present in databases, where domain constraints may represent a kind of knowledge in practical situations. Formally, we have absence when: p ∈ P | Bel(p) > 0Bel . Uncertainty and absence depend on the way beliefs are assigned to propositions. Other types of imperfection are characterized by the kinds of proposition supported by some belief. In particular, the number of elementary events composing a supported proposition defines the precision of our knowledge. If this number is greater than one, information is imprecise (IMP). There are different varieties of imprecision, depending on the way elementary events are grouped together. When propositions are disjunctions of elementary events, i.e., they are crisp sets, imprecision is usually called non-specificity (NS). For example, assume that U = {black, brown, blonde, white} represents all possible colors of John’s hair. “John has black or brown hair” is a form of non-specificity. In this case, p = {black, brown}. Notice that non-specificity depends on our knowledge, but also on the level of detail of U . We can express non-specificity as follows: ∃p ∈ P, ∃e1 , e2 ∈ KU, e1 = e2 | Bel(p) > 0Bel ∧ e1 , e2 ∈ P . When a set aggregating elementary events is ill-defined, we have vagueness (VAG). Vagueness and non-specificity are respectively the ill-defined and the crisp aspect of the same type of imperfection, i.e., imprecision. For example we may say that “John has dark hair”. If U is the previously defined set, this knowledge is vague. In principle, we may have an ill-defined set of only one
226
Matteo Magnani and Danilo Montesi
element, and in this case vagueness would not be associated to imprecision. However, we cannot identify any practical utility of this case, so we will not consider it. In general vagueness is not associated with a particular formalism. For example, we may represent knowledge “John has dark hair” with the fuzzy set D = {black(1.0), brown(0.8)}, meaning that brown is quite dark (fuzziness). Or we may use the rough set (CD = {black}, CD = {black, brown}), suggesting that sometimes brown hair may not be defined as dark4 [25]. In the following definition, the predicate “ill-defined(p)” returns true when p is a proper fuzzy set, rough set, or any other kind of ill-defined set. ∃p ∈ P | Bel(p) > 0Bel ∧ illdefined(p) . Inconsistency (INC) is characterized by a positive belief assigned to the empty proposition. For example, we can assign some belief to the proposition “John is 13 and 15 years old”. Many theories do not take inconsistency into account, requiring the belief assigned to the empty set to be equal to zero. However, inconsistency is not unusual in real situations. Formally, Bel(∅) > 0Bel . Finally, our knowledge can be wrong. Information is erroneous (ERR) when the true elementary event does not belong to any supported proposition. This can be due to the fact that the true elementary event does not belong to KU, or to the fact that we do not believe in it. Formally, t∈ /C . In Table 1 we have listed the classes of imperfection defined in this section that we will use in the remaining of the chapter. Notice that they are not disjoint. Ignorance can be present in different forms at the same time. When there is absence, we know nothing, so we cannot have uncertainty, vagueness or non-specificity. Otherwise, we may have uncertain but precise, imprecise but certain, or both uncertain and imprecise data. For example, we can believe that“perhaps, John is between 180 and 190 cm. tall”, or “perhaps, John is very tall”, combining imprecision with uncertainty.
3 A reference semistructured data model In the previous section we have defined some classes of imperfection based on quantified beliefs, propositions, and elementary events. However, we have not specified a semantics for the elements of U. In this way, our classes of imperfection are independent on the application context. To apply them to 4
These are the lower and upper approximations of the set of dark colors (D) based on the color (C).
An Overview of Imperfection Representation in Semistructured Data
227
Table 1. The classes of imperfection we consider in this chapter, with abbreviations and significant examples. In the real world, John is 183 cm. tall, and U can be the set {0, 1, . . . , 271, 272}, according to http://www.google.com/search?q=the+tallest+ man Class
Abbr.
Example: John’s tallness
Absence Non-Specificity
ABS NS
Vagueness Uncertainty Inconsistency Error
VAG UN INC ERR
Not known. Between 180 and 190 cm. 183 or 184 or 185 cm. Not very tall. Perhaps, 183 cm. 183 and 184 and 185 cm. 170 cm.
semistructured data modelling, we need to specify what U represents in this context, i.e., what knowledge is about. A data structure defines a projection of the reality on a finite set of data representation dimensions. These dimensions clearly identify everything we can know. As a consequence, they can be used to specify the elements of U. A formal definition of a semistructured data structure is then necessary to identify possible dimensions of imperfection. However, it must be general enough to subsume the ones adopted by existing data models. As it is typical, we use a (directed acyclic) graph extended with additional data [1; 5; 31]. Definition 1. A data graph is a tuple d = (V, E, , λ, τ, δ), where: V = {v1 , . . . , vn } is a finite set of vertices E ⊂ {(vi , vj ) | vi , vj ∈ V } (V, E) is a directed acyclic graph (DAG) is a possibly empty partial order on V λ : V → (L ∪ {null}), where L is a set of names τ : V → T , where T is a set of types ⊕(c(v), ) if outdegree(v) ≥ 1 • ∀v ∈ V, δ(v) = o.w. δ (v) where ⊕ is a parametric concatenation operator, c(v) is the set of v’s children, while δ is a content function defined on leaf nodes.
• • • • • •
The intuition about an edge (vi , vj ) is “vj is part of vi ”. Therefore, inside the model the sub-structure of leaves is not stored, but they are not ontologically different from internal nodes. Nodes may have labels (λ) and they may be ordered (). Every node has a type (τ ) and a content (δ). As nodes represent different levels of detail of the same reality, a content function may be defined for every node, and it must be an aggregation of the content of its children. The function δ is parameterized, because of the generality of semistructured data models. In particular, its recursive definition is based on an elementary function (δ ), which extracts the content of simple-typed nodes
228
Matteo Magnani and Danilo Montesi
(leaves), and a concatenation mechanism. Both these parameters depend on the peculiar application of the model. For example, in the context of an XML application we can define: ⊕ ({v1 , . . . , vm }) = δ(vj1 ) · · . . . · · δ(vjm ) δ (v) = fn:string(v)
(1)
where ∀i, p ∈ [1, m] (i ≤ p ⇒ vji vjp ). is the document order, · is the string concatenation operator, is the blank character and fn:string(v) returns the content of v converted as string. This function has a semantics similar to the one with the same name defined in the XQuery Functions and Operators specification, when applied to leaf nodes [20]. As another example, consider a graph representing overlapping regions, where nodes represent space regions and can be subregions (children) of several other nodes. Here, (1) would lead to an undesired result, because the graph is not ordered, and we do not want to consider the same region many times. A possible instantiation for δ is: ⊕ ({v1 , . . . , vm }) = {o | o ∈ δ(v1 ) ∨ · · · ∨ o ∈ δ(vm )} δ (v) = {λ(v)}
(2)
where λ extracts the name of the regions corresponding to the leaves of the graph. Notice that the computation of δ converges, because data graphs are both acyclic and finite. Data graphs can be organized into collections, folders, or classes, but in this chapter we focus on their internal imperfection. However, when a data graph is embedded in a super-structure, it can be useful to consider uncertainty about whether it belongs to that structure or not. Example 1 (XML document repository). Let x be the following XML document: Hugo Notre-Dame de Paris
We can model it using a data graph containing the following data tree: V = {v0 , v1 , v2 , v3 , v4 , v5 } E = {(v0 , v1 ), (v1 , v2 ), (v1 , v4 ), (v2 , v3 ), (v4 , v5 )} = {(vi , vj ) | i ≤ j} λ(v0 ) = null, λ(v1 ) = book, λ(v2 ) = author, λ(v4 ) = title, λ(v3 ) = λ(v5 ) = null • τ (v0 ) = document, τ (v1 ) = τ (v2 ) = τ (v4 ) = element, τ (v3 ) = τ (v5 ) = text (or other types defined by a DTD/XMLSchema) • δ(v0 ) = δ(v1 ) = ’Hugo Notre-Dame de Paris’, δ(v2 ) = δ(v3 ) = ’Hugo’, δ(v4 ) = δ(v5 ) = ’Notre-Dame de Paris’. • • • •
An Overview of Imperfection Representation in Semistructured Data
229
Here we have used the content aggregation equation (1). Many XML documents may be contained inside a collection, as illustrated in Fig. 2.
book author Hugo
title
Notre-Dame de Paris book
author Manzoni
title I promessi sposi
Fig. 2. An XML document repository. Roots are document nodes.
4 Dimensions of imperfection in semistructured data Each data representation dimension can be affected by different kinds of ignorance. In Sect. 4.1 we discuss which classes of imperfection can be significantly associated with each given data representation dimension, through an analysis of the elements of U. We call the resulting combinations (local) dimensions of imperfection. In Sect. 4.2 we analyze possible dependencies between different dimensions of imperfection. For example, uncertainty on node existence induces uncertainty on the arcs, as the existence of an arc depends on the existence of the nodes it connects. 4.1 Local dimensions of imperfection As we have already said, dimensions of ignorance in a semistructured data context are obtained by combining data representation dimensions (V, E, , λ, τ, δ) and types of uncertainty (ABS, NS, VAG, UN, INC, ERR), but not all combinations are significant: for example, the concept of “vague type”5 is not used in real applications. The following considerations are illustrated in Table 2. A first general consideration is that our knowledge can be wrong, independently on the data representation dimension we are considering. This is 5
“Vague type” can be used to refer to a type containing vague instances. In this chapter, consistently with our definition of knowledge and classes of imperfection, it indicates something that is a type “up to a certain degree”.
230
Matteo Magnani and Danilo Montesi
Table 2. Relevant dimensions of imperfection for semistructured data. The symbol × means that the combination of the corresponding data representation dimension and type of ignorance represents a significant dimension of imperfection. Empty cells indicate that the dimension is not significant ABS V E ≤ λ τ δ
× × × ×
NS × × × × × ×
VAG
UN
NS-UN
×
× × × × × ×
× × × × × ×
VAG-UN
×
INC
ERR
× ×
× × × × × ×
×
the reason why the last column of Table 2 is all checked. Inconsistency is usually due to the complexity of the language used to express propositions. If we know that a proposition corresponds to the empty set, we will not assign any belief to it. Therefore, inconsistency is not typical in nodes (V ), arcs (E) and types (τ ). On the contrary, it is more typical to express contents and labels with complex expressions. Similarly, in expressing the relative order of nodes, it is possible to provide equations with no solutions. However, inconsistency is often easy to identify and correct, and rarely affects labels. Now, we consider the ontological objects of the model, i.e., nodes (V ), arcs (E), and order (). We call them ontological, because they are based on being, or characterized by existence. V represents the existence of a data graph. E and concern the structure of the graph. V is the most important set of ontological objects, because the other data representation dimensions are based on it. In this case, the elements of U are sets of nodes W ⊆ V , where W means that every v ∈ W exists and every v ∈ / W does not exist. Even if a given data instance is composed of a finite number of nodes, we assume not to know V (ontological ignorance). This is due to the nature of semistructured data, where we do not know a priori if new nodes will join an existing instance. Both non-specificity and uncertainty can affect V . In fact, we can say that “a data graph consists of three or four nodes”, as in Fig. 3, or that “we are quite sure that a data graph is composed of two nodes”. On the contrary, absence would be possible in theory, but we do not take it into consideration: if we know nothing about a data graph, we have no reason to consider it. Remember that absence does not mean that we do not know if a particular node exists (this is NS). Additionally, vagueness is meaningless for ontological objects, because existence is a crisp concept, so we do not consider it. Figure 3 shows some possible elements of KU, when we consider vertexes. E and have a behavior similar to V . The only difference is that absence may be significant. Anyway, it is unusual, because absence means that we do not know structural information, and we do not even know if there is a struc-
An Overview of Imperfection Representation in Semistructured Data 1
2
3
1
2
3
231
4
Fig. 3. An example of elements of KU for vertexes. They refer to data graphs with, respectively, three and four nodes
ture. Examples of possible elements of KU for arcs and order are represented in Figs. 4 and 5.
1
1
1
2
2
2
Fig. 4. An example of elements of KU for arcs between nodes.
1 2
1 3
3
2
Fig. 5. An example of elements of KU for the relative order of nodes.
The other dimensions of the model, i.e., λ, τ , and δ, are functional properties with domain V . Possible ranges of these functions have been illustrated in Figs. 6, 7, and 8. In this case, for every function f (v), U represents its range. The types of ignorance that may affect these dimensions depend on the semantics of the range. Absence is not relevant on τ , in the sense that it can be managed outside the model. If we do not know anything about a type, we may specify it using any type (⊥). In principle, vagueness can affect types, but vague types are not used in real applications, and there is not a significant theoretical framework about vague types, to the best of our knowledge. As a consequence, we do not consider them in our taxonomy. On the contrary, non-specificity and uncertainty may affect our knowledge about types. For instance, we may know that a node is either a MSOffice or an Openoffice file (NS). λ is similar to τ , but in this case absence is not usually managed outside the model. It is worth noticing that absence means that we do not know anything about a label, while a null value indicates we know that there is no label. δ is the only dimension that can be reasonably affected by all types of ignorance. In particular, we may know nothing about it (ABS), or we may
232
Matteo Magnani and Danilo Montesi JPEG
GIF
BMP
TIFF
Fig. 6. An example of elements of KU for types.
null
P
H1
BODY
Fig. 7. An example of elements of KU for labels.
know a set of possible values (NS), sometimes ill-defined (VAG), we may not be certain about it (UN), and our knowledge can be partially inconsistent (INC) or completely wrong (ERR). The examples of Table 1 are typical cases of δignorance. Additionally, the definition of δ states that the content of internal nodes may depend on that of leaves. This means that a model for imperfect semistructured data should provide a mechanism to propagate ignorance from leaves to internal nodes. 1
2
...
272
Fig. 8. An example of elements of KU for contents.
4.2 Global imperfection dependencies In the previous sections we have identified several dimensions of imperfection. Sometimes we find that only one of them is present in a specific data instance. However, it may happen that many dimensions co-exist. In particular, some of them depend on others. Figure 9 illustrates this point. In Fig. 9(a) we have represented a semistructured instance with uncertainty on arc (v2 , v3 ). As already pointed out, the content of internal nodes depends on that of leaves. If we are not sure about the existence of arc (v2 , v3 ) we cannot assert that node v5 contributes to the content of v2 , even if we have complete knowledge of δ(v5 ) and δ(v6 ). Therefore, we can say that ignorance on the arcs induces ignorance on δ. If there is ignorance on E and also on δ , the propagation mechanism we mentioned at the end of the previous section should combine them. Moreover, ignorance on every dimension of the model is influenced by ignorance on the set of nodes (V ). In fact, if a node does not exist, we cannot have structural or functional properties on it. Figures 9(b) and 9(c) show two possible interpretations of ignorance on the existence of node v3 . In the first case, non existence of a node implies that all its subparts that are not subparts of other nodes do not exist. Figure 9(c) offers a different interpretation, where arcs are redistributed and transitive relationships of “being subpart of” are maintained.
An Overview of Imperfection Representation in Semistructured Data
233
This example should have clarified that different dimensions of ignorance may influence each other. We can distinguish between two kinds of dependencies: casual and model dependencies. Model dependencies are induced by the definition of the data model, and they are always present. For example, uncertainty on a node always induces uncertainty on the other dimensions, independently on the application or semantics of the data graph. This happens because the definition of the other dimensions is based on V . As a consequence, we may state that I(V ) I(E, ≤, τ, λ, δ)6 and I(E, ≤) I(δ). In addition, there may be casual dependencies. For example, in the context of a particular application the fact that a node has content “male” may determine the absence of a husband sibling. In this case, ignorance on labels or types can influence other dimensions of imperfection. Anyway, this is contextdependent, so we do not address this problem here (even if an investigation of all possible dependencies could be interesting).
v1
}
v2
v1
v2
v4
v4
v5
v6
v6
BB BB BB B!
(a)
(b)
v2
v5
v6
11
11
11
1 v4
111
11
1
v3 B
v11
(c)
Fig. 9. Ignorance interactions.
4.3 The relational case As an example, we show what happens when we restrict our model to the relational one. Here, a relation on attributes (A1 . . . An ) can be represented by a collection of data graphs, where each tuple t = t1 , . . . , tn corresponds to a data graph d = (V, E, , λ, τ, δ) with: • • • • • •
V = {v1 , . . . , vn } E=∅ =∅ λ(vi ) = Ai τ (vi ) = Dom(Ai ) δ(vi ) = ti .
Notice that δ is compatible with our definition of data graph. We also assume that τ and Dom refer to the same set of values. 6
Read, “ignorance on V induces (or leads to) ignorance on . . .
234
Matteo Magnani and Danilo Montesi
In this case, only four dimensions of the model are active: V, λ, τ, and δ. On the contrary, E and are empty. Moreover, additional restrictions further limit the presence of ignorance. The attribute names and the domains (λ and τ ) are not retrieved by typical relational query languages. Additionally, the schema restricts them to be fixed. Therefore, they are not affected by ignorance. The set of nodes (V ) is fixed, due to the strict tabular structure of the relational model, so we assume that all nodes always exist. The only exception is when we use the null value to indicate that an attribute does not exist. Anyway, this interpretation of null as ontological absence is not distinguished by others related to the value of the attributes. For this reason, this case is not treated in a specific way. Finally, we may have ignorance on the values of the nodes (δ), which is the only relevant dimension of ignorance at tuple level, and has been studied in many scientific papers. To sum up, the fact that ignorance in the relational model is monodimensional is due to its simplicity. When the data structure we have introduced to represent semistructured data is used in a more general way, many other dimensions can be worth managing.
5 Existing models for imperfect semistructured data In this section we briefly present the main existing models for imperfect semistructured data, in the light of the dimensions of imperfection identified in Sect. 4. The objective of these models is to extend semistructured data with probabilities, and not to represent specific dimensions of imperfection. Therefore, the fact that some dimensions are not managed by them should not be seen as a limitation. However, the following analysis will be useful to clearly identify possible extensions of these models, and to change or broaden their applicability. Additionally, the adoption of probabilities makes it difficult to state with certainty if some dimensions of imperfection can be modelled or not. Probability theory can be interpreted in several different ways, and the fact that an interpretation has not been specified for these models makes them general, but also difficult to classify. Where possible, we refer to the interpretation implicitly suggested by the motivating examples used by the authors to present their proposals. Table 3 summarizes the following discussion. The fact that the models described in this section are based on probabilities allows us to discuss some features that are common to all of them. First, consider incompatibility and error. Incompatibility cannot be represented by probability theory, because the probability of the empty set must be equal to zero — this is a corollary of Kolmogorov’s axioms. This does not represent a severe limitation in applications where inconsistency is not typical, or is simple to identify and remove. However, using probabilistic models we cannot represent someone’s generic belief about a semistructured data instance. Inconsistency and error are intersecting classes of imperfection. We can have inconsistency without error, if part of the belief is assigned to the
An Overview of Imperfection Representation in Semistructured Data
235
correct event and part to an inconsistent proposition, and we can have erroneous knowledge that is not inconsistent. However, inconsistency can be erroneous, so the models under consideration cannot manage error in general. When erroneous knowledge is consistent, we can represent it as if it was not erroneous, but probability theory does not specify how to combine two descriptions of the same reality where probabilities support disjoint alternatives. As a final general remark, we do not regard probability theory as the best formalism to represent vagueness. While vagueness refers to features that are graded in the real world, probability concerns events that are either true or false. As probability is an axiomatic theory, it can also be interpreted to adapt to vagueness. However, this interpretation has not been discussed in the works we are considering. 5.1 Semistructured probabilistic object data model To the best of our knowledge, the first published proposal of a semistructured probabilistic data model goes back to the year 2001 (Semistructured Probabilistic Object (SPO) data model) [7]. It is considerably different from the other models cited in this section: it is based on relations, where tuples may have a variable number of attributes, and attributes represent random variables. While a probabilistic semistructured database can be represented in XML, the structural features of XML are not directly supported by this model [32]. In particular, the term “semistructured” refers to the variable number of attributes. Therefore, this model can only represent uncertainty on values, and not on structure. As the content of internal XML nodes cannot be represented, uncertain node content is only partially supported. 5.2 ProTDB and TIX The first XML data model extended to manage uncertain information has been TAX [14]. Two relevant extensions of TAX have been proposed. The first defines a probabilistic XML database, ProTDB [22]. On the other hand, TIX, investigates the use of probabilistic information and new algebraic operators for XML information retrieval tasks [2]. As ProTDB, TIX (algebra for querying Text In XML) uses attributes to attach probabilities to XML trees. TIX could also be used to represent probabilistic data. However, we do not consider it in Table 3, because [2] focuses on information retrieval applications, where probabilities represent the relevance of XML fragments: data is perfect, but we do not know if we are interested in it. ProTDB (Probabilistic Tree Data Base) shows how to annotate an XML document with probabilistic information [22]. This can be viewed as a probabilistic model, moving probability annotations from the XML data representation to the underlying model. TAX-like pattern-matching can be applied to this model. Compared with the SPO data model, ProTDB can represent
236
Matteo Magnani and Danilo Montesi
uncertainty on the structure of XML trees. However, the approach used to represent probabilities is based on the assumption that node existence (probability) is independent on other nodes. The authors have tried to overcome this problem by defining ad hoc XML elements indicating specific probability distributions, but the representation of uncertainty on nodes is still partial. The same approach is applied to the content of leaf nodes. As probabilities concern nodes, uncertainty on arcs is not directly managed by ProTDB. 5.3 PXML and PIXML PXML is another model for probabilistic semistructured data [13]. Given a node, it can express any discrete probability distribution of its possible children. Simple distributions may be represented in a compact way, reducing their complexity. PXML distinguishes between leaves and internal nodes, and only leaves have a content. Therefore, the management of uncertainty on content is limited to leaves. This model does not assume node independence, so it has no limitations in representing uncertainty on the structure of XML trees. PIXML (Probabilistic Interval XML) is another version of PXML, where the model is generalized to manage interval probabilities [12]. Using interval probabilities, PIXML can represent more dimensions of imperfection than the models based on simple probabilities. In particular, non-specificity on nodes and arcs can be managed, combined or not with uncertainty. PIXML can also partially represent absence, corresponding to the assignment of a probability interval [0, 1] to every possible alternative instance. However, the set of alternative instances is strictly determined. In fact, both PXML and PIXML are based on a sort of schema, called weak instance, that defines the valid number of children for every node. Therefore, this can be considered a case of non-specificity. Uncertainty on leaves is not managed: this model consistently focuses on uncertain structures. 5.4 A model based on evidence theory We conclude our overview with a model based on Dempster–Shafer’s theory7 [18]. Although this model provides a query algebra to manipulate imperfect trees, from the point of view of imperfection representation it is no more expressive than PIXML. In fact, evidence theory is used to represent interval probabilities, and both models are based on a “possible worlds” semantics.
6 Concluding remarks In this chapter we have studied how imperfection can affect semistructured data. In particular, we have defined a taxonomy of dimensions of imperfection, based on widely accepted classes of ignorance and semistructured data 7
also known as evidence theory.
An Overview of Imperfection Representation in Semistructured Data
237
Table 3. Relevant dimensions of imperfection managed by the models cited in this section. A (−) near the name of a model means that the corresponding dimension of imperfection is only partially supported. The names of the models have been abbreviated as follows: SP (SPO data model), PR (ProTDB), PX (PXML), PI (PIXML), and DS (the model based on Dempster–Shafer’s theory) ABS V E ≤ λ τ δ
NS PI, DS PI, DS
VAG
UN PR(−) , PX, PI, DS PX, PI, DS
NS-UN VAG-UN INC ERR PI, DS PI, DS
SP(−) , PR(−) , PX(−)
representation dimensions. We have also pointed out that several dimensions of imperfection can co-exist inside the same data instance, with possible interactions between them. This has three important consequences. First, it can be necessary to integrate different formalisms to manage different kinds of imperfect data. Existing data models can already deal with more than one dimension of imperfection, but several others have not been considered yet. The second consequence concerns the dependencies between different dimensions of imperfection we have mentioned in Sect. 4.2. Future data models should provide a way to propagate δ-ignorance, and take into account model dependencies. Casual dependencies can probably be managed at schema level, through the application of