Data Warehousing and Knowledge Discovery: 5th International Conference, DaWaK 2003, Prague, Czech Republic, September 3-5,2003, Proceedings

XML for Data Warehousing Chances and Challenges (Extended Abstract) Peter Fankhauser and Thomas Klement Fraunhofer IPSI, Integrated Publication and Information Systems Institute Dolivostr. 15, 64293 Darmstadt, Germany {fankhaus,klement}@fraunhofer.ipsi.de http://www.ipsi.fraunhofer.de

The prospects of XML for data warehousing are staggering. As a primary purpose of data warehouses is to store non-operational data in the long term, i.e., to exchange them over time, the key reasons for the overwhelming success of XML as an exchange format also hold for data warehouses. – Expressive power: XML can represent relational data, EDI messages, report formats, and structured documents directly, without information loss, and with uniform syntax. – Self-describing: XML combines data and metadata. Thereby, heterogeneous and even irregular data can be represented and processed without a fixed schema, which may become obsolete or simply get lost. – Openness: As a text format with full support for Unicode, XML is not tied to a particular hardware or software platform, which makes it ideally suited for future proof long-term archival. But what can we do with an XML data warehouse beyond long term archival? How can we make sense of these data? How can we cleanse them, validate them, aggregate them, and ultimately discover useful patterns in XML data? A natural first step is to bring the power of OLAP to XML. Unfortunately, even though in principle XML is well suited to represent multidimensional data cubes, there does not yet exist a widely agreed upon standard neither for representing data cubes nor for querying them. XQuery 1.0 has resisted standardizing even basic OLAP features. Grouping and aggregation requires nested for-loops, which are difficult to optimize. XSLT 2.0 (XSL Transformations) has introduced basic grouping mechanisms. However, these mechanisms make it difficult to take into account hierarchical dimensions, and accordingly compute derived aggregations at different levels. In the first part of the talk we will introduce a small XML vocabulary for expressing OLAP queries that allow aggregation on different levels of granularity and can fully exploit document order and nested structure of XML. Moreover, we will illustrate the main optimization and processing techniques for such queries. Y. Kambayashi, M. Mohania, W. W¨ oß (Eds.): DaWaK 2003, LNCS 2737, pp. 1–3, 2003. c Springer-Verlag Berlin Heidelberg 2003


2

Peter Fankhauser and Thomas Klement

Data cubes constitute only one possible device to deal with the key challenge of XML data warehouses. XML data are notoriously noisy. They often come without a schema or with highly heterogeneous schemas, they rarely explicate dependencies and therefore are often redundant, and they can contain missing and inconsistent values. Data mining provides a wealth of established methods to deal with this situation. In the second part of the talk, we will illustrate by way of a simple experiment, how data mining techniques can help in combining multiple data sources and bringing them to effective use. We explore to which extent stable XML technology can be used to implement these techniques. The experiment deliberately focuses on data and mining techniques that cannot not be readily represented and realized with standard relational technology. It combines a bilingual dictionary, a thesaurus, and a text corpus (altogether about 150 MB data) in order to support bilingual search and thesaurus based analysis of the text corpus. We proceeded in three steps: None of the data sources was in XML form; therefore they needed to be structurally enriched to XML with a variety of tools. State-of-the-art schema mining combined with an off-the-shelf XML-Schema validator has proven to be very helpful to ensure quality for this initial step by ruling out enrichment errors and spurious structural variations in the initial data. In the next step, the data were cleansed. The thesaurus contained spurious cycles and missing relationships, and the dictionary suffered from incomplete definitions. These inconsistencies significantly impeded further analysis steps. XSLT, extended with appropriate means to efficiently realize fixpoint queries guided by regular path expressions, turned out to be a quick and dirty means for this step. However, even though cleansing did not go very far, the developed stylesheets reached a considerable level of complexity, indicating the need for better models to express and detect such inconsistencies. In the final step, the thesaurus was used to enrich the text corpus with so called lexical chains, which cluster a text into sentence groups that contain words in sufficiently close semantic neighborhood. These chains can be used to understand the role of lexical cohesion for text structure, to deploy this structure for finer grained document retrieval and clustering, and ultimately to enhance the thesaurus with additional relationships. Again, XSLT turned out to be a suitable means to implement the enrichment logic in an ad-hoc fashion, but the lack of higher level abstractions for both, the data structures and the analysis rules, resulted in fairly complex stylesheets. On the other hand, XSLT’s versatility w.r.t. expressing different structural views on XML turned out to be extremely helpful to flexibly visualize lexical chains. The main lessons learned from this small experiment are that state-of-the-art XML technology is mature and scalable enough to realize a fairly challenging text mining application. The main benefits of XML show especially for the early steps of data cleansing and enrichment, and the late steps of interactive analysis. These steps are arguably much harder to realize with traditional data warehouse technology, which requires significantly more data cleansing and restructuring as

XML for Data Warehousing Chances and Challenges

3

a prerequisite. On the other hand, the thesaurus based analysis in Step 3 suffers from the lack of XML-based interfaces to mining methods and tools. Realizing these in XSLT, which has some deficiencies w.r.t. compositionality and expressive power, turns out to be unnecessarily complex.

CPM: A Cube Presentation Model for OLAP Andreas Maniatis1, Panos Vassiliadis2, Spiros Skiadopoulos 1, Yannis Vassiliou1 1

National Technical Univ. of Athens, Dept. of Elec. and Computer Eng., 15780 Athens, Hellas {andreas,spiros,yv}@ dblab.ece.ntua.gr

2 University of Ioannina, Dept. of Computer Science 45110 Ioannina, Hellas [email protected]

Abstract. On-Line Analytical Processing (OLAP) is a trend in database technology, based on the multidimensional view of data. In this paper we introduce the Cube Presentation Model (CPM), a presentational model for OLAP data which, to the best of our knowledge, is the only formal presentational model for OLAP found in the literature until today. First, our proposal extends a previous logical model for cubes, to handle more complex cases. Then, we present a novel presentational model for OLAP screens, intuitively based on the geometrical representation of a cube and its human perception in the space. Moreover, we show how the logical and the presentational models are integrated smoothly. Finally, we describe how typical OLAP operations can be easily mapped to the CPM.

1.

Introduction

In the last years, On-Line Analytical Processing (OLAP) and data warehousing has become a major research area in the database community [1, 2]. An important issue faced by vendors, researchers and - mainly - users of OLAP applications is the visualization of data. Presentational models are not really a part of the classical conceptual-logical-physical hierarchy of database models; nevertheless, since OLAP is a technology facilitating decision-making, the presentation of data is of major importance. Research-wise, data visualization is presently a quickly evolving field and dealing with the presentation of vast amounts of data to the users [3, 4, 5]. In the OLAP field, though, we are aware of only two approaches towards a discrete and autonomous presentation model for OLAP. In the industrial field Microsoft has already issued a commercial standard for multidimensional databases, where the presentational issues form a major part [6]. In this approach, a powerful query language is used to provide the user with complex reports, created from several cubes (or actually subsets of existing cubes). An example is depicted in Fig. 1. The Microsoft standard, however, suffers from several problems, with two of them being the most prominent ones: First, the logical and presentational models are mixed, resulting in a complex language which is difficult to use (although powerful enough).

Y. Kambayashi, M. Mohania, W. W¨oß (Eds.): DaWaK 2003, LNCS 2737, pp. 4-13, 2003. c Springer-Verlag Berlin Heidelberg 2003

CPM: A Cube Presentation Model for OLAP

5

Secondly, the model is formalized but not thoroughly: for instance, to our knowledge, there is no definition for the schema of a multicube. SELECT CROSSJOIN({Venk,Netz},{USA_N.Children,USA_S,Japan}) ON COLUMNS {Qtr1.CHILDREN,Qtr2,Qtr3,Qtr4.CHILDREN} ON ROWS FROM SalesCube WHERE (Sales,[1991],Products.ALL) Year = 1991 Product = ALL

Venk USA USA_N Seattle

Japan USA_S Boston

Netz USA USA_N Seattle

Japan USA_S Boston

Size(city)

R1

R2 R3 R4

Qtr1

Qtr2 Qtr3 Qtr4

Jan Feb Mar

C1

C2

C3

C4

C5

C6

Jan Feb Mar

Fig. 1: Motivating example for the cube model (taken from [6]). Apart from the industrial proposal of Microsoft, an academic approach has also been proposed [5]. However, the proposed Tape model seems to be limited in its expressive power (with respect to the Microsoft proposal) and its formal aspects are not yet publicly available. In this paper we introduce a cube presentation model (CPM). The main idea behind CPM lies in the separation of logical data retrieval (which we encapsulate in the logical layer of CPM) and data presentation (captured from the presentational layer of CPM). The logical layer that we propose is based on an extension of a previous proposal [8] to incorporate more complex cubes. Replacing the logical layer with any other model compatible to classical OLAP notions (like dimensions, hierarchies and cubes) can be easily performed. The presentational layer, at the same time, provides a formal model for OLAP screens. To our knowledge, there is no such result in the related literature. Finally, we show how typical OLAP operations like roll-up and drill down are mapped to simple operations over the underlying presentational model. The remainder of this paper is structured as follows. In Section 2, we present the logical layer underlying CPM. In Section 3, we introduce the presentational layer of the CPM model. In Section 4, we present a mapping from the logical to the presentational model and finally, in Section 5 we conclude our results and present topics for future work. Due to space limitations, we refer the interested reader to a long version of this report for more intuition and rigorous definitions [7].

2.

The logical layer of the Cube Presentation Model

The Cube Presentation Model (CPM) is composed of two parts: (a) a logical layer, which involves the formulation of cubes and (b) a presentational layer that involves the presentation of these cubes (normally, on a 2D screen). In this section, we present

6

Andreas Maniatis et al.

the logical layer of CPM; to this end, we extend a logical model [8] in order to compute more complex cubes. We briefly repeat the basic constructs of the logical model and refer the interested reader to [8] for a detailed presentation of this part of the model. The most basic constructs are: − A dimension is a lattice of dimension levels (L,p), where p is a partial order defined among the levels of L. − A family of monotone, pairwise consistent ancestor functions ancLL is defined, such that for each pair of levels L1 and L2 with L1pL2, the function ancLL maps each element of dom(L1) to an element of dom(L2). − A data set DS over a schema S=[L1,…,Ln,A1,…,Am] is a finite set of tuples over S such that [L1,…,Ln] are levels, the rest of the attributes are measures and [L1,…,Ln] is a primary key. A detailed data set DS0 is a data set where all levels are at the bottom of their hierarchies. − A selection condition φ is a formula involving atoms and the logical connectives ∧, ∨ and ¬. The atoms involve levels, values and ancestor functions, in clause of the form x ∂ y. A detailed selection condition involves levels at the bottom of their hierarchies. − A primary cube c (over the schema [L1,…,Ln,M1,…,Mm]), is an expression of the form c=(DS0,φ,[L1,…,Ln,M1,…,Mm],[agg1(M01),…,aggm(M0m)]), where: DS0 is a detailed data set over the schema S=[L01,…,L0n,M01,…,M0k],m≤k. φ is a detailed selection condition. M1,…,Mm are measures. L0i and Li are levels such that L0ipLi, 1≤i≤n. aggi∈{sum,min,max,count}, 1≤i≤m. The limitations of primary cubes is that, although they model accurately SELECT-FROM-WHERE-GROUPBY queries, they fail to model (a) ordering, (b) computation of values through functions and (c) selection over computed or aggregate values (i.e., the HAVING clause of a SQL query). To compensate this shortcoming, we extend the aforementioned model with the following entities: − Let F be a set of functions mapping sets of attributes to attributes. We distinguish the following major categories of functions: property functions, arithmetic functions and control functions. For example, for the level Day, we can have the property function holiday(Day) indicating whether a day is a holiday or not. An arithmetic function is, for example Profit=(Price-Cost)*Sold_Items. − A secondary selection condition ψ is a formula in disjunctive normal form. An atom of the secondary selection condition is true, false or an expression of the form x θ y, where x and y can be one of the following: (a) an attribute Ai (including RANK), (b) a value l, an expression of the form fi(Ai), where Ai is a set of attributes (levels and measures) and (c) θ is an operator from the set (>, <, =, ≥, ≤, ≠). With this kind of formulae, we can compute relationships between measures (Cost>Price), ranking and range selections (ORDER BY...;STOP after 200, RANK[20:30]), measure selections (sales>3000), property based selection (Color(Product)='Green'). 2 1

2 1

CPM: A Cube Presentation Model for OLAP

7

− Assume a data set DS over the schema [A1,A2,…,Az]. Without loss of generality, suppose a non-empty subset of the schema S=A1,…,Ak,k≤z. Then, there is a set of ordering operations OθS, used to sort the values of the data set, with respect to the set of attributes participating to S. θ belongs to the set {<,>,∅} in order to denote ascending, descending and no order, respectively. An ordering operation is applied over a data set and returns another data set which obligatorily encompasses the measure RANK. − A secondary cube over the schema S=[L1,…,Ln,M1,…,Mm,Am+1,…,Am+p, RANK] is an expression of the form: s=[c,[Am+1:fm+1(Am+1),…,Am+p:fm+p(Am+p)],OθA,ψ] where c=(DS0,φ,[L1,…,Ln,M1,…,Mm],[agg1(M01),…,aggm(M0m)]) is a primary cube, [Am+1,…,Am+p]⊆[L1,…,Ln,M1,…,Mm], A⊆S-{RANK}, fm+1,…,fm+p are functions belonging to F and ψ is a secondary selection condition. With these additions, primary cubes are extended to secondary cubes that incorporate: (a) computation of new attributes (Am+i) through the respective functions (fm+i), (b) ordering (OθA) and (c) the HAVING clause, through the secondary selection condition ψ.

3.

The presentational layer of the Cube Presentation Model

In this section, we present the presentation layer of CPM. First, we will give an intuitive, informal description of the model; then we will present its formal definition. Throughout the paper, we will use the example of Fig. 1, as our reference example. The most important entities of the logical layer of CPM include: − Points: A point over an axis resembles the classical notion of points over axes in mathematics. Still, since we are grouping more than one attribute per axis (in order to make things presentable in a 2D screen), formally, a point is a pair comprising of a set of attribute groups (with one of them acting as primary key) and a set of equality selection conditions for each of the keys. − Axis: An axis can be viewed as a set of points. We introduce two special purpose axes, Invisible and Content. The Invisible axis is a placeholder for the levels of the data set which are not found in the “normal” axis defining the multicube. The Content axis has a more elaborate role: in the case where no measure is found in any axis then the measure which will fill the content of the multicube is placed there. − Multicubes. A multicube is a set of axes, such that (a) all the levels of the same dimensions are found in the same axis, (b) Invisible and Content axes are taken into account, (c) all the measures involved are tagged with an aggregate function and (d) all the dimensions of the underlying data set are present in the multicube definition. In our motivating example, the multicube MC is defined as MC={Rows,Columns,Sections,Invisible,Content}. − 2D-slice: Consider a multicube MC, composed of K axes. A 2D-slice over MC is a set of (K-2) points, each from a separate axis. Intuitively, a 2D-slice pins the axes of

8

Andreas Maniatis et al.

the multicube to specific points, except for 2 axes, which will be presented on the screen (or a printout). In Fig. 2, we depict such a 2D slice over a multicube. − Tape: Consider a 2D-slice SL over a multicube MC, composed of K axes. A tape over SL is a set of (K-1) points, where the (K-2) points are the points of SL. A tape is always parallel to a specific axis: out of the two "free" axis of the 2D-slice, we pin one of them to a specific point which distinguishes the tape from the 2D-slice. − Cross-join: Consider a 2D-slice SL over a multicube MC, composed of K axes and two tapes t1 and t2 which are not parallel to the same axis. A cross-join over t1 and t2 is a set of K points, where the (K-2) points are the points of SL and each of the two remaining points is a point on a different axis of the remaining axes of the slice. The query of Fig. 1 is a 2D-Slice, say SL. In SL one can identify 4 horizontal tapes denoted as R1, R2, R3 and R4 in Fig. 1) and 6 vertical tapes (numbered from C1 to C6). The meaning of the horizontal tapes is straightforward: they represent the Quarter dimension, expressed either as quarters or as months. The meaning of the vertical tapes is somewhat more complex: they represent the combination of the dimensions Salesman and Geography, with the latter expressed in City, Region and Country level. Moreover, two constraints are superimposed over these tapes: the Year dimension is pinned to a specific value and the Product dimension is ignored. In this multidimensional world of 5 axes, the tapes C1 and R1 are defined as: C1 = [(Salesman='Venk'∧ancregion city (city)='USA_N'),(Year='1991'), (ancALL item(Products)='all'),(Sales,sum(Sales))] R1 = [(ancmonth day (Month)='Qtr1'∧Year='1991'),(Year='1991'), (ancALL item(Products)='all'),(Sales,sum(Sales))]

One can also consider the cross-join t1 defined by the common cells of the tapes R1 and C1. Remember that City defines an attribute group along with [Size(City)]. t1=([SalesCube,(Salesman='Venk'∧ancregion city (city)='USA_N ∧ ALL ancmonth day (Month)='Qtr1'∧Year='1991'∧ancitem(Products)='all'), [Salesman,City,Month,Year,Products.ALL,Sales],sum], [Size(City)],true)

In the rest of this section, we will describe the presentation layer of CPM in its formality. First, we extend the notion of dimension to incorporate any kind of attributes (i.e., results of functions, measures, etc.). Consequently, we consider every attribute not already belonging to some dimension, to belong to a single-level dimension (with the same name as the attribute), with no ancestor functions or properties defined over it. We will distinguish between the dimensions comprising levels and functionally dependent attributes through the terms level dimensions and attribute dimensions, wherever necessary. The dimensions involving arithmetic measures will be called measure dimensions. An attribute group AG over a data set DS is a pair [A,DA], where A is a list of attributes belonging to DS (called the key of the group) and DA is a list of attributes dependent on the attributes of A. With the term dependent we mean (a) measures dependent over the respective levels of the data set and (b) function results depending

CPM: A Cube Presentation Model for OLAP

9

on the arguments of the function. One can consider examples of the attribute groups such as ag1=([City],[Size(City)]),ag2=([Sales,Expenses],[Profit]). Invisible

Sections

Year=1992

Year=1991

Content

+

+

Products.ALL = 'all'

Sales, sum(Sales 0), true

ancmonth day (Month)= Qtr1 Quarter = Qtr3 Quarter = Qtr2

Rows ancmonth day (Month)= Qtr4

Salesman='Venk', Region='USA_S' Salesman='Netz', (2) anc region city (City) = (1) 'USA_N' Salesman='Venk', (4) ancregion city (City) = (3) 'USA_N' Salesman='Venk', Country='Japan'

Salesman='Netz', Country='Japan' (6)

(5) Salesman='Netz', Region='USA_S' Columns

Fig. 2: The 2D-Slice SL for the example of Fig. 1.

A dimension group DG over a data set DS is a pair [D,DD], where D is a list of dimensions over DS (called the key of the dimension group) and DD is a list of dimensions dependent on the dimensions of D. With the term dependent we simply extend the respective definition of attribute groups, to cover also the respective dimensions. For reasons of brevity, wherever possible, we will denote an attribute/dimension group comprising only of its key simply by the respective attribute/dimension. An axis schema is a pair [DG,AG], where DG is a list of K dimension groups and AG is an ordered list of K finite ordered lists of attribute groups, where the keys of each (inner) list belong to the same dimension, found in the same position in DG, where K>0. The members of each ordered list are not necessarily different. We denote an axis schema as a pair ASK=([DG1×DG2×…×DGK],[[ag11,ag21,…,agk1 ]×[ag12,ag22 1

,…,agk22]×…×[ag1k,ag2k,…,agkkk])}.

In other words, one can consider an axis schema as the Cartesian product of the respective dimension groups, instantiated at a finite number of attribute groups. For instance, in the example of Fig. 1, we can observe two axes schemata, having the following definitions: Row_S = {[Quarter],[Month,Quarter,Quarter,Month]} Column_S = {[Salesman×Geography], [Salesman]×[[City,Size(City)], Region, Country]} Consider a detailed data set DS. An axis over DS is a pair comprising of an axis schema over K dimension groups, where all the keys of its attribute groups belong to DS, and an ordered list of K finite ordered lists of selection conditions (primary or

secondary), where each member of the inner lists, involves only the respective key of the attribute group. a = (ASK,[φ1,φ2,...,φK]),K≤N or a={[DG1×DG2×…×DGK],[[ag11,ag21,…,agk1 ]×[ag12,ag22,…,agk2 ]×…×[ag1k,ag2k,…,agkk ]], [[φ11,φ21,…,φk1 ]×[φ12,φ22,…,φk2 ]×...×[φ1k,φ2k,…,φkk ]]} 1

1

2

2

k

k

10

Andreas Maniatis et al.

Practically, an axis is a restriction of an axis schema to specific values, through the introduction of specific constraints for each occurrence of a level. In our motivating example, we have two axes: Rows = {Row_S,[ancmonth day (Month)=Qtr1,Quarter=Qtr2, Quarter=Qtr3,ancmonth day (Month)=Qtr4]} Columns = {Column_S,{[Salesman='Venk',Salesman='Netz'], [ancregion city (City)='USA_N', Region='USA_S', Country='Japan']}

We will denote the set of dimension groups of each axis a by dim(a). A point over an axis is a pair comprising of a set of attribute groups and a set of equality selection conditions for each one of their keys. p1=([Salesman,[City,Size(City)]], [Salesman='Venk',ancregion city (City)= 'USA_N'])

An axis can be reduced to a set of points, if one calculates the Cartesian products of the attribute groups and their respective selection conditions. In other words, a=([DG1×DG2×...×DGK],[[p1,p2,…,pl]), l=k1×k2×…×kkk.

Two axes schemata are joinable over a data set if their key dimensions (a) belong to the set of dimensions of the data set and (b) are disjoint. For instance, Rows_S and Columns_S are joinable. A multicube schema over a detailed data set is a finite set of axes schemata fulfilling the following constraints: 1. All the axes schemata are pair-wise joinable over the data set. 2. The key of each dimension group belongs only to one axis. 3. Similarly, from the definition of the axis schema, the attributes belonging to a dimension group are all found in the same axis. 4. Two special purpose axes called Invisible and Content exist. The Content axis can take only measure dimensions. 5. All the measure dimensions of the multicube are found in the same axis. If more than one measure exist, they cannot be found in the Content axis. 6. If no measure is found in any of the "normal" axes, then a single measure must be found in the axis Content. 7. Each key measure is tagged with an aggregate function over a measure of the data set. 8. For each attribute participating in a group, all the members of the group are found in the same axis. 9. All the level dimensions of the data set are found in the union of the axis schemata (if some dimensions are not found in the "normal" axes, they must be found in the Invisible axis). The role of the Invisible axis follows: it is a placeholder for the levels of the data set which are not to be taken into account in the multicube. The Content axis has a more elaborate role: in the case where no measure is found in any axis (like in the example of Fig. 1) then the measure which will fill the content of the multicube is placed there. If more than one measures are found, then they must be placed in the same axis (not Content), as this would cause a problem of presentation on a two-dimensional space. A multicube over a data set is defined as a finite set of axes, whose schemata can define a multicube schema. The following constraints must be met:

CPM: A Cube Presentation Model for OLAP

11

1. Each point from a level dimension, not in the Invisible axis, must have an equality selection condition, returning a finite number of values. 2. The rest of the points can have arbitrary selection conditions (including "true" for the measure dimensions, for example). For example, suppose a detailed data set SalesCube under the schema S = [Quarter.Day, Salesman.Salesman, Geography.City, Time.Day, Product.Item, Sales, PercentChange, BudgetedSales] Suppose also the following axes schemata over DS0 Row_S = {[Quarter],[Month,Quarter,Quarter,Month]} Column_S = {[Salesman×Geography], [Salesman]×[[City,Size(City)], Region, Country]} Section_S = {[Time],[Year]} Invisible_S = {[Product],[Product.ALL]} Content_S = {[Sales],[sum(Sales0)]}

and their respective axes Rows={Row_S,[ancmonth day (Month)=Qtr1,Quarter=Qtr2,Quarter=Qtr3, ancmonth day (Month)=Qtr4]} Columns = {Column_S,{[Salesman='Venk',Salesman='Netz'], [ancregion city (City)='USA_N', Region='USA_S', Country='Japan']} Sections = {Section_S,[Year=1991,Year=1992]} Invisible = {Invisible_S,[ALL='all']} Content_S = {Content_S,[true]} Then, a multicube MC can be defined as MC = {Rows, Columns, Sections, Invisible, Content}

Consider a multicube MC, composed of K axes. A 2D-slice over MC is a set of (K-2) points, each from a separate axis, where the points of the Invisible and the Content axis are comprised within the points of the 2D-slice. Intuitively, a 2D-slice pins the axes of the multicube to specific points, except for 2 axes, which will be presented on a screen (or a printout). Consider a 2D-slice SL over a multicube MC, composed of K axes. A tape over SL is a set of (K-1) points, where the (K-2) points are the points of SL. A tape is always parallel to a specific axis: out of the two "free" axis of the 2D-slice, we pin one of them to a specific point which distinguishes the tape from the 2D-slice. A tape is more restrictively defined with respect to the 2D-slice by a single point: we will call this point the key of the tape with respect to its 2D-slice. Moreover if a 2D-slice has two axes a1,a2 with size(a1) and size(a2) points each, then one can define size(a1)*size(a2) tapes over this 2D-slice. Consider a 2D-slice SL over a multicube MC, composed of K axes. Consider also two tapes t1 and t2 which are not parallel to the same axis. A cross-join over t1 and t2 is a set of K points, where the (K-2) points are the points of SL and each of the two remaining points is a point on a different axis of the remaining axes of the slice. Two tapes are joinable if they can produce a cross-join.

12

4.

Andreas Maniatis et al.

Bridging the presentation and the logical layers of CPM

Cross-joins form the bridge between the logical and the presentational model. In this section we provide a theorem proving that a cross-join is a secondary cube. Then, we show how common OLAP operations can be performed on the basis of our model. The proofs can be found at [7]. Theorem 1. Α cross-join is equivalent to a secondary cube. The only difference between a tape and a cross-join is that the cross-join restricts all of its dimensions with equality constraints, whereas the tape constraints only a subset of them. Moreover, from the definition of the joinable tapes it follows that a 2D-slice contains as many cross-joins as the number of pairs of joinable tapes belonging to this particular slice. This observation also helps us to understand why a tape can also be viewed as a collection of cross-joins (or cubes). Each of this cross-joins is defined from the k-1 points of the tape and one point from all its joinable tapes. This point belongs to the points of the axis the tape is parallel to. Consequently, we are allowed to treat a tape as a set of cubes: t=[c1,…,ck]. Thus we have the following lemma. Lemma 1. A tape is a finite set of secondary cubes. We briefly describe how usual operations of the OLAP tools, such as roll-up, drill down, pivot etc can be mapped to operations over 2D-slices and tapes. − Roll-up. Roll-up is performed over a set of tapes. Initially key points of these tapes are eliminated and replaced by their ancestor values. Then tapes are also eliminated and replaced by tapes defined by the respective keys of these ancestor values. The cross-joins that emerge can be computed through the appropriate aggregation of the underlying data. − Drill-down. Drill down is exactly the opposite of the roll-up operation. The only difference is that normally, the existing tapes are not removed, but rather complemented by the tapes of the lower level values. − Pivot. Pivot means moving one dimension from an axis to another. The contents of the 2D-slice over which pivot is performed are not recomputed, instead they are just reorganized in their presentation. − Selection. A selection condition (primary or secondary) is evaluated against the points of the axes, or the content of the 2D-slice. In every case, the calculation of the new 2D-slice is based on the propagation of the selection to the already computed cubes. − Slice. Slice is a special form of roll-up, where a dimension is rolled up to the level ALL. In other words, the dimension is not taken into account any more in the groupings over the underlying data set. Slicing can also mean the reconstruction of the multicube by moving the sliced dimension to the Invisible axis. − ROLLUP [9]. In the relational context, the ROLLUP operator takes all combination of attributes participating in the grouping of a fact table and produces all the

CPM: A Cube Presentation Model for OLAP

13

possible tables, with these marginal aggregations, out of the original query. In our context, this can be done by producing all combinations of Slice operations over the levels of the underlying data set. One can even go further by combining roll-ups to all the combinations of levels in a hierarchy.

5.

Conclusions and Future Work

In this paper we have introduced the Cube Presentation Model, a presentation model for OLAP data which formalizes previously proposed standards for a presentation layer and which, to the best of our knowledge, is the only formal presentational model for OLAP in the literature. Our contributions can be listed as follows: (a) we have presented an extension of a previous logical model for cubes, to handle more complex cases; (b) we have introduced a novel presentational model for OLAP screens, intuitively based on the geometrical representation of a cube and its human perception in the space; (c) we have discussed how these two models can be smoothly integrated; and (d) we have suggested how typical OLAP operations can be easily mapped to the proposed presentational model. Next steps in our research include the introduction of suitable visualization techniques for CPM, complying with current standards and recommendation as far as usability and user interface design is concerned and its extension to address the specific visualization requirements of mobile devices.

References [1]S. Chaudhuri, U. Dayal: An overview of Data Warehousing and OLAP technology. ACM SIGMOD Record, 26(1), March 1997. [2]P. Vassiliadis, T. Sellis: A Survey of Logical Models for OLAP Databases. SIGMOD Record 28(4), Dec. 1999. [3]D.A. Keim. Visual Data Mining. Tutorials of the 23 rd International Conference on Very Large Data Bases, Athens, Greece, 1997. [4]Alfred Inselberg. Visualization and Knowledge Discovery for High Dimensional Data . 2nd Workshop Proceedings UIDIS, IEEE, 2001. [5]M. Gebhardt, M. Jarke, S. Jacobs: A Toolkit for Negotiation Support Interfaces to MultiDimensional Data. ACM SIGMOD 1997, pp. 348 – 356. [6]Microsoft Corp. OLEDB for OLAP February 1998. Available at: http://www.microsoft.com/data/oledb/olap/. [7]A. Maniatis, P. Vassiliadis, S. Skiadopoulos, Y. Vassiliou. CPM: A Cube Presentation Model. http://www.dblab.ece.ntua.gr/~andreas/publications/CPM_dawak03.pdf (Long Version). [8]Panos Vassiliadis, Spiros Skiadopoulos: Modeling and Optimization Issues for Multidimensional Databases. Proc. of CAiSE-00, Stockholm, Sweden, 2000. [9]J. Gray et al.: Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab and Sub-Totals. Proc. of the ICDE 1996.

Computation of Sparse Data Cubes with Constraints Changqing Chen1, Jianlin Feng2, and Longgang Xiang3 1

School of Software, Huazhong Univ. of Sci. & Tech., Wuhan 430074, Hubei, China [email protected] 2 School of Computer Science, Huazhong Univ. of Sci. & Tech., Wuhan 430074, Hubei, China [email protected] 3 School of Computer Science, Huazhong Univ. of Sci. & Tech., Wuhan 430074, Hubei, China [email protected]

Abstract. For a data cube there are always constraints between dimensions or between attributes in a dimension, such as functional dependencies. We introduce the problem that when there are functional dependencies, how to use them to speed up the computation of sparse data cubes. A new algorithm CFD is presented to satisfy this demand. CFD determines the order of dimensions by considering their cardinalities and functional dependencies between them together. It makes dimensions with functional dependencies adjacent and their codes satisfy monotonic mapping, thus reduces the number of partitions for such dimensions. It also combines partitioning from bottom to up and aggregate computation from top to bottom to speed up the computation further. In addition CFD can efficiently compute a data cube with hierarchies from the smallest granularity to the coarsest one, and at most one attribute in a dimension takes part in the computation each time. The experiments have shown that the performance of CFD has a significant improvement.

1 Introduction OLAP often pre-computes a large number of aggregates to improve the performance of aggregation queries. A new operator CUBE BY [5] was introduced to represent a set of group-by operations
i.e., to compute aggregates for all possible combinations of attributes in the CUBE BY clause. The following example 1 shows a cube computation query on a relation SALES (employee, product, customer, quantity) Example 1: SELECT employee, product, customer, SUM (quantity) FROM SALES CUBE BY employee, product, customer It will compute group-bys for (employee, product, customer), (employee, product), (employee, customer), (product, customer), (employee), (product), (customer) and ALL (no GROUP BY). The attributes in the CUBE BY clause are called dimensions and the attributes aggregated are called measures. For n dimensions, 2 n group-bys are Y. Kambayashi, M. Mohania, W. W¨oß (Eds.): DaWaK 2003, LNCS 2737, pp. 14-23, 2003. c Springer-Verlag Berlin Heidelberg 2003

Computation of Sparse Data Cubes with Constraints

15

computed. The number of distinct values of a dimension is its cardinality. Each combination of attribute values from different dimensions constitutes a cell. If empty cells are a majority of the whole cube, then the cube is sparse. Relational normal forms are hardly suitable for OLAP cubes because of different goals in operational and OLAP databases. The main goal of operational databases is to avoid update anomalies and the relational normal forms are very adaptive for this goal. But for OLAP databases the efficiency of queries is the most important issue. So there are always constraints between dimensions or between attributes in a dimension for a cube, such as functional dependencies. Sparsity clearly depends on actual data. However, functional dependencies between dimensions may imply potential sparsity [4]. A tacit assumption for all algorithms before is that dimensions are independent each other and so all these algorithms did not consider the effect of functional dependencies on computing cubes. Algebraic functions COUNT, SUM, MIN and MAX have the key property that more detailed aggregates (i.e. more dimensions) can be used to compute less detailed aggregates (i.e. fewer dimensions). This property induces a partial ordering (i.e. a lattice) on all group-bys of the CUBE. A group-by is called a child of some parent group-by if the parent can be used to compute the child (and no other group-by is between the parent and the child). The algorithms [1, 2, 3, 6] recognize that group-bys with common attributes can share partitions, sorts, or partial sorts. The difference between them is that how they exploit such properties. In these algorithms, BUC [1] computes from bottom to up, while others compute from top to bottom. This paper addresses full cube computation over sparse data cubes and makes the following contributions: 1.

We introduce the problem of computation of sparse data cubes with constraints, which allows us to use such constraints to speed up the computation. A new algorithm CFD (Computation by Functional Dependencies) is presented to satisfy this demand. CFD determines the partitioning order of dimensions by considering their cardinalities and functional dependencies between them together. Therefore the correlated dimensions can share sorts.

2.

CFD partitions group-bys of a data cube from bottom to up, at the same time it computes aggregate values from top to bottom by summing up return values of smaller partitions. Even if all the dimensions are independent each other, CFD is still faster than BUC to compute full cubes.

3.

Few algorithms deal with hierarchies in dimensions. CFD can compute a sparse data cube with hierarchies in dimensions. In this situation, CFD efficiently computes from the smallest granularity to the coarsest one. The rest of this paper is organized as follows: Section 2 presents the problem of sparse cubes with constraints. Section 3 illustrates how to decide partitioning order of dimensions. Section 4 presents a new algorithm for the computation of sparse cubes called CFD. Our performance analysis is described in section 5. Related work is discussed in section 6. Section 7 contains conclusions.

16

Changqing Chen et al.

2 The Problem Let C= DM be an OLAP cube schema, where D are the set of dimensions and M the set of measures. Two attributes X and Y with one-to-one or many-to-one relation has a functional dependency XY, where X is called a determining attribute and Y is called a depending attribute. Such functional dependency can exist between two dimensions or two attributes in a dimension. The problem is when there are constraints (functional dependencies), how to use them to speed up the computation of sparse cubes. The dependencies considered in CFD are only those that the left and right side always contain a single attribute respectively. Such functional dependencies will help in data pre-processing (see Section 3.2) and partitioning dimensions (see section 4.1). Functional dependencies between dimensions implied the structural sparsity of a cube [4]. With no functional dependencies, the structural sparsity is zero. Considering the cube in Example 1, if we know that one employee sells only one product, we get a functional dependency employeeproduct. Assume we have 6 employees, 4 customers, and 3 different products, then the size of the cube is 72 cells. Further the total number of occupied cells in the whole cube is at most 64=24, thus the structural sparsity is 67%.

3 Data Preprocessing CFD partitions from bottom to up just like BUC, i.e., first partitions on one dimension, then two dimensions, and etc. One difference between CFD and BUC is that CFD chooses the order of dimensions by functional dependencies and cardinalities together. 3.1 Deciding the Order of Dimensions First we build a directed graph by Functional Dependencies between dimensions, called FD graph. The graph ignores all transitive dependencies (i.e., dependencies that can be deduced from other dependencies). A node in the graph is a dimension. Once the graph has been built, we try to classify the nodes. We find the longest path in the graph in order to make the most of dependencies. The nodes in such path form a dependency set and are deleted from the graph. Such process is repeated until the graph is empty. The time complexity of this process is O(n2), where n is the number of dimensions. Example 2: A cube has six dimensions from A to F with the cardinalities descendant and functional dependencies AC, AD, CE, BF. Figure 1 is the corresponding FD graph. From Figure 1, we first get the dependency set {A, C, E} for they have the longest path, then {B, F} and at last {D}. The elements in each set are ordered by dependencies. Although there is a functional dependency between A and D, it is not considered, so the dependency set {D} contains only the dimension D itself. After getting the dependency sets, CFD sorts them descendently by the biggest cardinality of a dimension in each set. Then we merge each set sequentially to determine

Computation of Sparse Data Cubes with Constraints

17

the order of dimensions. By this approach, CFD can make the depending dimension share the sort of the determining dimension because such two dimensions are putted together. If there is no functional dependency, the partitioning order of CFD is just the same as that of BUC. A

C

B

D

F

Tom

towel

towel

Tom

0

0

Bob

soap

towel

Ross

1

0

Smith

soap

soap

Bob

2

1

Smith

3

1

White E

Fig. 1. FD graph

employee product

product employee

employee product

sharver

sort

soap

encode

Louis

soap

soap

Louis

4

1

Ross

towel

shaver

White

5

2

Fig. 2. The encoding of two dimensions with a functional dependency

3.2 Data Encoding Like other algorithms for computing a data cube, CFD assumes that each dimension value is an integer between zero and its cardinality, and that the cardinality is known in advance. A usual data encoding does not consider the correlations between dimensions and simply maps each dimension value between zero and its cardinality. This operation is similar to sorting on the values of a dimension. In order to share shorts, CFD encodes adjacent dimensions with functional dependencies jointly to make their codes satisfy a monotonic mapping. For example, X and Y are two dimensions and f is a functional dependency from X to Y. Assume there are two arbitrary values xi and xj on dimension X, and yi = f(xi) and yj = f(xj) are two values on dimension Y. If xi> xj, we have yi  yj, then y = f(x) is monotonic. Due to the functional dependency between X and Y, the approach of encoding is to sort on dimension Y first, then the values of X and Y can be mapped sequentially to zero and their cardinalities respectively. Figure 2 shows the encoding of two dimensions with a functional dependency: employeeproduct in Example 1. Obviously, if the left or right side of a functional dependency has more than one attribute, it is difficult to encode like that. Note that the mapping relations can be reflected in the fact table for correlated dimensions. But for hierarchies in a dimension the mapping relations should be stored respectively.

4 Algorithm CFD We propose a new algorithm called CFD for the computation of full sparse data cubes with constraints. The idea in CFD is to take advantage of functional dependencies to share partitions and to make use of the property of algebraic functions to reduce aggregation costs. CFD was inspired by BUC algorithm and is similar to a version of algorithm BUC except the aggregation computation and the partition function. After data preprocessing, we can compute a sparse data cube now.

18

Changqing Chen et al.

CFD(input,dim) Inputs: input: The relation to aggregate. dim: The starting dimension to partition. Globals: numdims: the total number of dimensions. dependentable[]: the dependency sets gotten from section 3.1. hierarchy[numdims]: the high of hierarchies in each dimension. cardinality[numDims][]: the cardinality of each dimension. dataCount[numdims]: the size of each partition. aggval[numdims]: sum the results of smaller partitions. 1: if (dim == numdims) aggval[dim]=Aggregate(input); //the result of a thinnest part ition. 2: FOR d = dim; d < numdims; d++ DO 3: FOR h=0; h
The details of CFD are in Figure 3. The first step is to aggregate the entire input to aggval[numdims] when arriving at the smallest partition. For each dimension d between dim and numDims, the input is partitioned on dimension d (line 6). On return from Partition(), dataCount contains the number of records for each distinct value of the d-th dimension. Line 8 iterates through the partitions (i.e. each distinct). The partition becomes the input relation in the next recursive call to CFD, which computes the cube on the partition for dimensions (d+1) to numDims. Upon return from the recursive call, we sum up the aggregation results from smaller partitions (line 14) and continue with the next partition of dimension d. Once all the partitions are processed, we repeat the whole process for the next dimension. We use the same optimization process Write-Ancestors() as that in BUC (line 11). Write-Ancestors() simply outputs each of ancestor group-bys to avoid fruitless partition and computation when the partition contains only one tuple. For a data cube containing a single tuple partition (a1, b1) with dimensions A, B, C, D, Write-Ancestors() directly output (a1, b1), (a1, b1, c), (a1, b1, c, d) and (a1, b1, d).

Computation of Sparse Data Cubes with Constraints

19

4.1 Partitioning CFD partitions from the bottom of the lattice and works its way up towards the larger, less detailed aggregates. When CFD partitions, Partition() determines whether two dimensions are dependent by dependentable[] gotten from section 3.1. 4

A

C

B

D

a1

c1

b1

d1

a1

c1

b1

d2

a1

c1

b2

d2

a2

c1

b1

d2

a3

c2

b3

d1

a4

c3

b4

d2

Fig. 4. Encoding and partitioning of CFD

3

2

ACB

AC

6

1

AB

A

5

ACBD

7

ACD

8

AD

9

10

ABD

CB

13

C

16

B

12

11

CBD

CD

15

14

BD

D

ALL

Fig. 5. CFD processing tree

Example 3: A data cube has four dimensions from A to D with the cardinalities descendant and a dependency: AC. The order of dimensions for is A, C, B and D. Figure 4 illustrates how the input in Example 3 is partitioned during the first calls by CFD. First CFD partitions on dimension A, producing partitions a1 to a4, then it recursively partitions the partition a1 on dimension C, then dimension B. Because of the dependency: AC and the monotonic mapping, CFD will not sort on dimension C. This is one key optimization of CFD. CFD will not affect the efficiency of other optimizations such as finding single tuples. For independent dimensions, CountingSort, QuickSort and InsertSort can be used by CFD just like BUC. 4.2 Data Cube Computation Another key factor in the success of CFD is to take advantage of algebraic functions by summing up the results of the recursive calls from the smallest partitions (line 1 and 14 of Figure 3). By this approach, CFD can reduce half of the time to scan the whole relation, and it is faster than BUC even when dimensions are independent. Figure 5 shows the CFD processing tree for Example 3. The numbers indicate the order in which CFD visits the group-bys and CFD produces the empty group-by ALL last. We find if we directly use the return of the recursive call to aggregate the results, CFD actually runs more slowly than BUC, just as what reported in [1]. This may be the side effect of stack operations. Instead we use the array aggval[numdims] to record and aggregate the results, and it really runs faster than BUC. On the negative side, CFD cannot efficiently compute holistic functions. 4.3 Data Cube with Hierarchies Some approaches have been proposed to reduce structural sparsity risks, i.e. some dimensions with functional dependencies are decomposed to hierarchical dimensions [4, 11]. Hierarchical dimensions, which enable the user to analyze data on different levels of aggregation, are essential for OLAP.

20

Changqing Chen et al.

CFD can also efficiently compute a cube with hierarchies in dimensions (showed in line 3 of Figure 3). Because the attributes in a dimension have functional dependencies from the smaller granularity (the smallest one is the key) to the coarser one, so its computation is similar to that of a cube with constraints between dimensions.

4.4 Memory Requirements The memory requirement of CFD is slightly more than that of BUC. CFD tries to fit a partition of tuples in memory as soon as possible. Say that a partition has X tuples and each tuple requires N bytes. Our implementation also uses pointers to the tuples, and CountingSort requires a second set of pointers for temporary use. Let d0…dn-1 be the cardinalities of each dimension for a data cube, and dmax be the maximum cardinality. n 1

d i counters. In order to aggregate

CountingSort uses dmax counters, and CFD uses i =0

results from the smallest partitions, an n array is needed for an aggregation function. If the counters and pointers are each four bytes and every element in the above array is also four bytes, the total memory requirement in bytes for CFD is: n 1

d i + 4dmax + 4n

( +8) X + 4 i =0

5 Performance Analysis Since MemoryCube is the first algorithm to compute sparse cubes and BUC is faster than MemoryCube, we concentrate on comparing CFD with BUC to compute full cubes. We implemented CFD for main memory only. The implementation of BUC had the same restriction. CFD can be smoothly extended when performing an external partitioning. We did not count the time to read the file and write the results. We only measured running time as the total cost, including partitioning time and aggregating time. We also neglected the cost of determining the order of dimensions. 5.1 Qualitative analysis First we analyze the time of aggregation computation and assume dimensions are independent each other. For a cube with n dimensions, the number of 2 n group-bys need to be computed. BUC needs to scan the whole relation 2n times. CFD uses the results of the smallest partitions to aggregate the results of coarser partitions. This means CFD can save about half of the time of aggregation computation. Due to the single-tuple optimization, CFD can actually save 3-5% of the total cost than BUC when dimensions are independent.

Computation of Sparse Data Cubes with Constraints

21

Next we analyze the partitioning time. For a cube with n dimensions, the order is 0, 1, …n-1, then the number of partitions for each dimension are 20, 21, …2n-1 correspondingly. Say that the partitioning time of each dimension is one. Considering a functional dependency didi+1, it will save 2i+1 times by sharing the sort of the i-th dimension when partitioning the (i+1)-th dimension. If there are k functional dependencies considered and depending dimensions are i1, … ik, then the total time saved is ik m = i1

2 m (k
It seems that if we push the dimensions with dependencies more backward, we will save more partitioning time. But such approach neglects the optimization of finding single tuples as early as possible. Experiments have shown that the performance may decrease if we do like that. 5.2 Full Cube Computation All tests were run on a machine with 128MB of memory and a 500MHz Pentium processor. The data was randomly generated uniformly. The sparse data cube used in our experiments had eight dimensions from A to H, and each dimension had a cardinality of 10. The whole data could be loaded in the memory. Figure 6 compared CFD with BUC to compute a full cube with five sum functions and four functional dependencies: AE, BF, CG, and DH, varying the number of tuples. For 1.5 million tuples, CFD saved the time by about 25% than BUC. The time saved above is caused by two factors: the time for partition and aggregation computation. Figure 7 shows the computation time for one million tuples with dimensions independent varying the number of aggregation functions. For five sum functions, CFD saved 15% than BUC. Figure 8 compared CFD with BUC considering functional dependencies. The data for this test were one million tuples. We considered one dependency in Figure 6 each time. With the position of a dependency moves backwards, the time reduces quickly.

300

160

200 100 0 0.5

200

Time(sec)

Time(sec)

Time ( s ec)

165

250

400

150

150

100 0. 75

1

1. 25

1.5

Number of tuples (million)

BUC

145

1

2

3

4

5

Nu mber of s u m s

CFD

Fig. 6. Computation of a cube

155

BUC

0

1

2

3

4

Po sitio n o f a d epen den cy

CF D

Fig. 7. Aggregation functions

CFD

Fig. 8. Different position

Figure 9 compared CFD with BUC for just one sum function, assuming the dimensions are independent each other. The data varied from 0.5 million tuples to 1.5 million tuples. With the tuples increased, single tuples decreased and CFD saved more time of aggregation computation than BUC. We note that CFD still saved 3% of the total time on the most sparse cube with 0.5 million tuples. This experiment shows that CFD is more adaptive than BUC to compute data cubes with different sparsities.

22

Changqing Chen et al.

250

1500 Ti me ( s ec)

Ti me ( s ec)

200 150 100

1000 500

50 0 0.5

0. 75

1

Number of tuples

BUC

1. 25 1.5 (million)

CFD

Fig. 9. Dimensions independent

0 0.2

0.4

0.6

0.8

1

Number of tuples (million)

CFD & No FD

CFD with FD

Fig. 10. Hierarchies in dimensions

5.3 Hierarchies This experiment explores the computation of a data cube with hierarchies varying the number of tuples. The dimensions were order by cardinality: 500, 200, 100, 80, 60 and 50. The cardinality of coarser granularity was one third of that of smaller granularity adjacent in each dimension. There were three hierarchies in the first two dimensions and two hierarchies in the other dimensions. The results are shown in Figure 10. The bottom line shows the computation considering functional dependencies between attributes in a dimension and the upper line shows that without considering those correlations. The running time increased 30% when functional dependencies not considered. 5.4 Weather Data We compared CFD with BUC on a real nine-dimensional dataset containing weather conditions at various weather stations on land for September 1985 [12]. The dataset contained 1,015,367 tuples. The attributes were ordered by cardinality: station (7037), longitude (352), Solar-altitude (179), latitude (152), present-weather (101), day (30), weather-change-code (10), hour (8), and brightness (2). Some of the attributes were significantly correlated (e.g., only one station was at one (latitude, longitude)). The experiment shows that CFD is effective on real dataset. The running time of CFD had a 3% improvement than BUC considering the functional dependency station latitude. When (latitude, longitude) was treated as hierarchies of the dimension station, CFD had a 5% improvement than BUC that does not consider dependencies.

6 Related Work Since the introduction of the concept of data cube [5], efficient computation of data cubes has been a theme of active research, such as [1, 2, 3, 7, 8, 9, 10]. All algorithms

Computation of Sparse Data Cubes with Constraints

23

before did not consider the effect of constraints on the computation of a data cube. Following Lenher et al., a general principle in OLAP design is that dimensions should be independent, and inside a dimension there should be a single attribute key for the dimension [11]. But it is not always suitable to put all correlated attributes in a dimension to satisfy such principle, just as the weather data in [12] showed. While some work concentrated on fast computing a data cube, some other work dealt with the size problem of a cube [7, 8, 10]. Computing iceberg cubes rather than complete cubes were also proposed by [2, 9].

7

Conclusions

We introduce the problem of computation of sparse data cubes with constraints and present a novel algorithm CFD for this problem. CFD decides the order of dimensions by taking into account cardinalities and functional dependencies together. In the meantime CFD partitions group-bys from bottom to up and aggregates the results of partitions from top to bottom. So CFD combines the advantage of partitioning from bottom to up of BUC and the advantage of aggregation computation from top to bottom of Pipesort and etc. Our approach can efficiently speed up the computation of sparse data cubes as well as those with hierarchies.

References 1.

K.Baeyer, R.Ramakrishnan. Bottom-Up Computation of Sparse and Iceberg CUBEs. SIGMOD’99, pages 359-370 2. K.A.Ross, D.Strivastava. Fast computation of sparse data cubes. In Proc. Of the 23 rd VLDB Conf., pages 116-125,Athens, Green, 1997 3. S. Agarwal, R. Agrawal, P. M. Desgpande, A.Gupta, J. F. Naughton, R. Ramakrishnan, and S.Sarawagi,. On the computation of multidimensional aggregates. In Proc. Of the 22nd VLDB Conf., pages 506-521, 1996 4. T. Niemi, J. Nummenmaa, P. Thanisch, Constructing OLAP Cubes Based on Queries. DOLAP 2001, pages 1-8 5. J.Gray, A.Bosworth, A.Layman and H.pirahesh. Datacube: A relational aggregation operator generalizing group by, cross-tab, and sub-totals, ICDE’96, pages 152-159 6. Y. Zhao, P. M. Desgpande, and J. F. Naughton. An array-based algorithm for simultaneous mutldimensional aggregates. SIGMOD’97, pages 159-170 7. W. Wang, J. Feng, H. Lu, J. X. Yu. Condensed Cube: An Effective Approach to Reducing Data Cube Size. Proc. of the 18 th Int. Conf. on Data Engineering, 2002, pp.155 -165 8. Y. Sismanis, A. Deligiannakis, N. Roussopoulos, Y. Kotidis. Dwarf: Shrinking the PetaCube. SIGMOD’02 9. J. Han, J. Pei, G. Dong and K. Wang. Efficient Computation of Iceberg Cubes with Complex Mea sures. SIGMOD’01. 10. Lakshmanan, L., Pei, J. and Han, J. Quotient Cube: How to Summarize the Semantics of a Data CubeFast. Proc. the 28 rd VLDB Conference, HongKong, China, 2002 11. W. Lenher, J. Albrecht and H. Wedekind. Normal forms for multidimensional databases. SSDBM’98, pages 63-72 12. C. Hahn, S. Warren, and J. London. Edited synoptic cloud reports from ships and land stations over the globe, 1982-1991. http://cdiac.esd.ornl.gov/-cdiac/ndps/ndp026b.h tml, http://cdiac.esd.ornl.gov/ftp/ndp026b/SEP85L.Z, 1994

Answering Joint Queries from Multiple Aggregate OLAP Databases Elaheh Pourabbas1 and Arie Shoshani2 1

Istituto di Analisi dei Sistemi ed Informatica ”Antonio Ruberti” - CNR Viale Manzoni, 30 00185, Rome,Italy [email protected] 2 Lawrence Berkeley National Laboratory Mailstop 50B-3238 1 Cyclotron Road, Berkeley, CA 94720, USA [email protected]

Abstract. Given an OLAP query expressed over multiple source OLAP databases, we study the problem of evaluating the result OLAP target database. The problem arises when it is not possible to derive the result from a single database. The method we use is the linear indirect estimator, commonly used for statistical estimation. We examine two obvious computational methods for computing such a target database, called the ”Full-cross-product” (F) and the ”Pre-aggregation” (P) methods. We study the accuracy and computational complexity of these methods. While the method F provides a more accurate estimate, it is more expensive computationally than P. Our contribution is in proposing a third new method, called the ”Partial-Pre-aggregation” method (PP), which is significantly less expensive than F, but is just as accurate.

1

Introduction

Similar to Statistical Databases that were introduced in the 1980’s [1], OLAP databases have a data model that represents one or more ”measures” over a multidimensional space of ”dimensions”, where each dimension can be defined over a hierarchy of ”category attributes” [3],[7]. In many socio-economic type applications only summarized data is available because the base data for the summaries (called ”microdata”) are not kept or are unavailable for reasons of privacy [7]. We will refer to Statistical Databases or OLAP databases that contain summarized data as ”summary databases”, and the measures associated with them as ”summary measures”. Each summary measure must have a ”summary operator” associated with it, such as ”sum, or ”average”. In this paper, we address the problem of evaluating queries expressed over multiple summary databases [6]. That is, given that the base data is not available and that a query cannot be derived from a single summary database, we examine the process of estimating the desired result from multiple summary databases by a method of interpolation common in statistical estimation, called ”linear indirect estimator”. Essentially, this method takes advantage of the correlation between measures to perform the

Y. Kambayashi, M. Mohania, W. W¨ oß (Eds.): DaWaK 2003, LNCS 2737, pp. 24–34, 2003. c Springer-Verlag Berlin Heidelberg 2003


Answering Joint Queries from Multiple Aggregate OLAP Databases

25

estimation. For example, suppose that we have a summary database of ”totalincome by race, and sex” and another summary database of ”population by state, age, and sex”. If we know that there is a strong correlation between ”population” and ”income” of states, we can infer the result ”total-income by state”. We say that in this case the ”population” was used as a proxy measure to estimate ”total-income by state”. Similarly, the summary database ”population by state, age, and sex” is referred to as the proxy database. The problem we are addressing is to answer the query ”total-income by state” over the two summary databases. Furthermore, the fact that there is a common category attribute to both databases (Sex) can be used to achieve more accurate results, as we’ll show later. Consider the above mentioned databases written in the following notation: ”summary-measure (category-attribute,..., category-attribute)”. One obvious method of estimating this result is to aggregate each of the source databases to the maximum level. We call this the Pre-aggregation (P) method. In this case, we can aggregate Population(State, Sex, Age) over sex and age to produce Population(State)1 and Total-Income(Race,Sex) over race and sex to produce Total-Income(•), where the symbol ”•” indicates aggregation over all the category attributes. Then, we can calculate the proportional estimation using linear indirect estimation (see section 3) to produce Total-Income(State). Another possibility is to produce the full cross product: Total-Income(State,Sex,Race,Age) using Population as a proxy summary measure. Then, from this result we can aggregate over Sex, Race, Age to get Total-Income(State). We call this the Fullcross-product (F) method. This is the most accurate result that can be produced since it performs the linear indirect estimation on the most detailed cells. We use the Average Relative Error (ARE) to measure accuracy (see section 5) of these estimation methods. Using this measure we show that method P is less accurate than method F. Our main result is in proposing a method, called the Partial-pre-aggregation (PP) method that achieves the same accuracy as the F method but at a much lower computation cost. This is achieved by noticing that it is possible to pre-aggregate over all the category attributes that are not in common to the two source databases before we perform the cross product, and still achieve the same accuracy of the full cross product computation. According to this method, in our example, we first aggregate over Race in the TotalIncome(Race,Sex) database to produce Total-Income(Sex) and aggregate over Age in Population(State,Sex,Age) to produce Population(State, Sex). Then, we use the linear indirect estimator to produce Total-Income(State,Sex). Finally, we aggregate over Sex in Total-Income(State,Sex) to produce Total-Income(State). We show that this result is as accurate as F, but by performing the aggregations over the non-common attributes early we minimize the computation needed. The paper is structured as follows. Section 2 introduces syntax for defining joint queries that provides the basis for a formal analysis of the accuracy of the results. Section 3 discusses the underlying methodology for the estimation of query results. Section 4 describes the three methods for estimating the query 1

Population(State) is equivalent to Population(State, ALL, ALL), where ALL indicates the construct introduced in [2]. For the sake of brevity, we use the first notation.

26

Elaheh Pourabbas and Arie Shoshani

results. The accuracy of these methods is investigated in Section 5, whereas, the results of performance evaluation are given in Section 6. Finally, Section 7 concludes.

2

The Joint Query Syntax

We define the syntax of a joint query on two summary databases, in terms of the common and non-common category attributes of the databases. Let M (CMi ), N (CN j ) with 1 ≤ i ≤ p, and 1 ≤ j ≤ q be summary databases, where M and N are summary measures, CMi and CN j are category attributes, and p and q each represents a finite number. A joint query formulated on these summary databases is a triple defined by M T (C C , C C , C T ) where: M T is a selected target summary database that can be one of M or N . Without loss of generality, suppose that M was selected. Then, C C is a set of R common category attributes, where CrC = CMi = CN j , with 1 ≤ r ≤ R. C C is a set of S non-common category attributes, where CsC = CMi or CsC = CN j , with CMi  = CN j , 1 ≤ s ≤ S. C T represents a set of target category attributes, which includes at least one CN i and at least one of the target category attributes is not in the target summary database. According to these distinctions, the summary databases can C C C C T T be represented by M (CM , CM ), N (CN , CN , CN ), where CN are called target category attributes of the joint query, N is called the proxy measure, and M C C T , CN , CN ) is called the proxy database. Note is the target measure (M T ), N (CN that the set of category attributes of summary databases, as well as the category attributes of a joint query can be null.

3

The Linear Indirect Estimator Method

The main idea of such an approach, known in the literature as Small Area Estimation, is to use data from surveys designed to produce estimates of variables of interest at the national or regional level, and to obtain comparable estimates at more geographically disaggregated levels such as counties. This approach is characterized by indirect estimators techniques. An indirect estimator uses values of the variable of interest from available auxiliary (called predictor or proxy) data at the local level that are related to the variable of interest [4],[5]. In this model, the population is partitioned into a large number of domains formed by cross classification of demographic variables such as age, sex, race. Let i denote a small area. For each domain
d, the only available variable of interest Y (e.g., Income) denoted by Y (d) = i Y (i, d) is calculated from the survey data. Furthermore, it is assumed that auxiliary information (e.g., Population) in the form of X(i,
d) is also available. A estimator i is given
ˆ of Y for small area
Y (i, d), where X(d) = (X(i, d)/X(d))Y (d) = by Yˆ (i) = d d i X(i, d), X(i, d)/X(d) represents the proportion of the population of small area
i that
is in each domain d, and i Yˆ (i) equals to the direct estimator Y = d Y (d). The estimate is subject to error. The error is computed using the true values. In

Answering Joint Queries from Multiple Aggregate OLAP Databases

27

our examples, we assume the knowledge of the true values in order to evaluate the error. This provides us with the means of comparing the accuracy of the results using different computational methods. This estimation method applies to any summary database where the assumption is that the characteristics of small areas are sufficiently close to the characteristics of the large areas.

4

Computational Methods for Answering Joint Query

In this section, we propose three methods able to compute joint queries expressed on two summary databases. They are called Full-cross-product, Pre-aggregation, and Partial-Pre-aggregation methods. We first define them, and then we discuss their relative computational complexity. For the definition of these methods, we use the formalism defined for a joint query in Section 2. 4.1

The Full Cross Product Method (F)

The next theorem provides the estimation of the full-cross-product over summary databases that represent different measures. C C C C , CM ) and N (CN , CN ) be two summary databases. The Theorem 1. Let M (CM full-cross-product summary database is obtained as follows: C

C

N (CN , CN ) C C C C C ˆ (CM , CM , CN ) = M (CM , CM )
M C , CC ) N (CN N CC

(1)

N

C C If we assume as proxy the summary database N (CN , CN ), then by linear indirect estimation the proof of the above theorem is straightforward.

Example 1. Table 1 represents the data in the summary databases, Income(Race, Sex), and Population(State, Sex, Age) where by ”Income” we mean ”TotalIncome” in the rest of paper. Let us consider obtaining the full-cross-product ˆ summary database by Eq. 1, Income(State,Sex,Age,Race)=Income(Race, Sex ) Population(State,Age,Sex )
. According to the syntax introduced in SecPopulation(State,Age,Sex ) State,Age

tion 2, Sex is a common category attribute, Race, Age, and State are non-common category attributes. The summary database Population(State, Sex, Age) is the proxy database. For instance, if Income(State) is the target summary database, applying the F method, we first obtain the full cross product summary database shown in Table 2 (for space limitation, only one state is shown) and then we summarize all category attributes except the target attribute. The result is shown in Table 3, third column.

28

4.2

Elaheh Pourabbas and Arie Shoshani

The Pre-aggregation (P) Method

The pre-aggregation method is based on summarizing the summary databases over all common and non-common category attributes before the application of the linear indirect estimator method. C C C C T Definition 1. Let M (CM , CM ) and N (CN , CN , CN ) be summary databases. ˆ T (C T ), is estimated by pre-summarizing all The target summary database M N common and non-common category attributes in the summary databases as fol

C C T C C T , CM ), N (CN ) = C C ,C C N (CN , CN , CN ), and lows: M (•) = C C ,C C M (CM M

M

N

N

T

ˆ T (C T ) = M (•)(
N (CN ) ). then applying the linear indirect estimation: M N N (C T ) CT N

N

Example 2. Consider Table 1. To apply the P method, we first summarize all common and non-common
category attributes in the summary databases Income and Population: Age,Sex Population (State, Age, Sex )=Population (State),
Income(Race, Sex ) = Income(•). Then, by applying linear indirect Race,Sex ˆ estimation, we obtain I ncome(State)=Income(•)
Population(State) . The reState

Population(State)

sult of target summary database ˆI ncome(State) is shown in in Table 3, fourth column. We observe that ˆI ncome(State)F and ˆI ncome(State)P are different. 4.3

The Partial-Pre-aggregation (PP) Method

This method was devised to yield the same accuracy as method F but with a lower computational complexity. The main idea is to summarize the summary databases only over non-common category attributes, and then estimate the target summary database with the common and target category attributes. C C C C T , CM ) and N (CN , CN , CN ) be summary databases. Definition 2. Let M (CM T T ˆ The target summary database M (CN ), is estimated by pre-summarizing all the C non-common category attributes in the summary databases as follows: M (CM )=

C C C T C C T M (C , C ), N (C , C ) = N (C , C , C ), and then estimating C C M M N N N N N CM CN the cross product and summarizing over the common attributes as follows: C T ˆ ( C C , C T ) =
C (M (C C ))(
N (CN ,CNC) T ). ˆ T (C T ) =
C M M N N N M C C N (C ,C ) N

N

C C ,C T N N

N

N

Example 3. Consider again Table 1. First, we summarize
the summary databases over non-common category attributes as follows: Age Population (State, Age,
Sex ) = Population (State, Sex ), Race Income(Race, Sex ) = Income(Sex ). Then,
by applying linear indirect estimation, we obtain ˆI ncome(State)= Sex ˆI ncome ) (State, Sex )=Income(Sex )
Population(State,Sex . We note that the results Population(State,Sex ) State

obtained by ˆI ncome(State)PP are identical to ˆI ncome(State)F shown in Table 3.

Answering Joint Queries from Multiple Aggregate OLAP Databases

29

Table 1. Income(Race,Sex) and Population(State,Sex,Age) Sex

Income Race

Male

White 629907 Black 311121 Hispanic 312800 Non-Hispanic 331254 Total 1585082

Sex

Population

Female

State

330567 241835 192632 272765 1037799

AL CA FL NE TE

Age

Male Female

<65 ≥65 <65 ≥65 <65

years years years years years

6 3 7 3 9

4 3 5 5 6

≥65 <65 ≥65 <65 ≥65

years 5 years 3 years 3 years 12 years 7 58

3 5 7 6 5 49

Total

ˆ Table 2. Income (State,Race,Age,Sex) ˆ Income State AL

5

Sex Race

Age

<65 years ≥65 years Black <65 years ≥65 years Hispanic <65 years ≥65 years Non-Hispanic <65 years ≥65 years ...... ...... White

Male

Female

65162.7931 32581.396552 32184.93103 16092.46552 32358.62069 16179.31035 34267.65517 17133.82759 ......

26985.06122 20238.795918 19741.632653 14806.22449 15725.061224 11793.79592 22266.530612 16699.897959 ......

Accuracy Analysis of Methods

The previous definitions can be used to estimate joint queries with any number of (common, non-common) category attributes. As seen from the previous examples, the application of the F and P methods on the same data yields different results. This difference is formalized by the next theorem.

30

Elaheh Pourabbas and Arie Shoshani

ˆ T (C T ) over M (C C , C C ) and Theorem 2. The estimation of any joint query M N M M C C T , CN ,CN ) using the methods F and P give different results. N (CN Proof. (sketch) We prove this by negation. Suppose methods F and P give the ˆ T (C T ). According to this assumption in Eq.2, (i), (ii) must be same results M N C C equal. CM = CN , but the proportions are not equal. It contradicts the assumption.  
 C C T N (C , C , C )   C N N N C C C   (i) M (CM , CM )
N   C C T C C ,C T N (CN , CN , CN ) C CN CN CM N  

 C C T N (C , C , C ) C   N N N CN CC C C   M (CM , CM )

N (ii)   C , CC , CT ) N (C C T N N N CC C CC C ,C M

CM

N

N

(2)

N

Since the methods F and P yield different results, we need a way of evaluating the accuracy of these results. A common approach to determine the accuracy is based on the calculation of their average relative errors (ARE) from the ”true”
m |ˆvi −vi | 1 base values (v) as defined in [4]: ARE = m . By the linear indirect i=1 vi estimator method, we obtain an estimation of a given target measure for small area i, and then we calculate the ARE from the true value of small area i. In Table 3, the true value for each State as well as the estimated value by methods F (or PP), and P are shown. Note that, the true values are calculated from the base data in order to obtain the error between the true values and the data estimated by each of the F, PP, and P methods. In same table, the relative ARE are shown. As can be seen, while method P is less expensive in terms of computation than method F, method F is more precise w.r.t. method P. An intermediate solution in term of computational complexity of these two methods is method PP. As we saw from the examples 1, and 2 and on the basis of the next theorem the methods F and PP give the same results. ˆ T (C T ) over M (C C , C C ) and Theorem 3. The estimation of any joint query M N M M C C T N (CN , CN ,CN ) using the methods F and PP give the same results. Proof. (sketch) It is easy to show the part F⇒PP by simple algebric manipulation [6]. For the part PP⇒F, we consider a simplified representation of the ˆ T (C T ) obtained by the PP method as follows: (i) M N For any set of category attributes Cp , and Cq with p  = q, the above summary database can be the result of summarization over C p and Cq as follows:


N (CNC ,Cq ,CNT )

Cq C
where the inner term is equivC C ,C ) CN Cp M (CM , Cp ) N (CN q Cq



 N (CNC ,Cq ,CNT ) 

C ˆ (CN , Cp , Cq , M (C = Cp ,Cq M alent to (ii) , C ) p M Cp ,Cq N (C C ) N

T CN ). As can be easily seen, the summarization of the full cross product in (ii) ˆ (C C , C T ) that is equal to the inner term of (i). over Cp , and Cq gives M N N

Answering Joint Queries from Multiple Aggregate OLAP Databases

31

Table 3. Results obtained by methods F (or PP) and P, and their ARE ˆ Income Income F (P P ) AL CA FL NE TE

265039 495302 806255 393694 662591

394218.0000 485085.7143 573222.14286 418128.85714 752226.28571

ˆ Income P 392206.50467 490258.13084 563796.85047 441232.31776 735387.19626

ARE

5.1

ˆ ˆ |Income i(F ) −Incomei | |Incomei(P ) −Incomei | Incomei

Incomei

0.487396195 0.020626377 0.289031209 0.062065607 0.135279963

0.479806763 0.010183422 0.300721421 0.120749409 0.109865960

0.198879870

0.204265395

Pre-aggregation on Category Hierarchies

Let us consider an example of a category hierarchy State → Region → Country. In the case that the result database needs to be evaluated over a higher-level category, such as Income(Region) it is more efficient to aggregate (or roll-up in OLAP terminology) the source database Population(State,Age,Sex) to the region level to produce Population(Region,Age,Sex) before applying the PP method. Intuitively, we expect to get the same level of accuracy. We verified that indeed aggregating (rolling-up) first to the higher category level produced the same accuracy as aggregating after applying the PP method [6]. Thus, we consider aggregation to the desired level of the category hierarchies as a first step of the Partial-Pre-Aggregation (PP) method.

6

Performance Evaluation

In this section, we illustrate the experimental results of performance evaluation of the methods F and PP. We focus our attention on these methods because of their higher accuracy levels w.r.t. method P. The performance evaluation of methods are described over two and then over more than two summary databases through some examples. 6.1

Performance Evaluation over Two Summary Databases

Let the number of cells of a given summary database be defined by XM =  i  C  where |Ci | represents the cardinality of the domain values of the i=1,...,n i-th category attributes. For instance, the total number of cells of the summary database Population(State,Age,Sex) is XP opulation = 20, given that the cardinality of the domain values of the category attributes (State, Age, Sex) are respectively: 5, 2, 2. Note that each cell requires the same space for storing the data value, and therefore the space complexity for each cell is assumed to be the same. Thus counting the number of cells is a good measure of the space required.

32

Elaheh Pourabbas and Arie Shoshani

C C C C T Definition 3. Let M (CM , CM ) and N (CN , CN , CN ) be summary databases, and let M be the target summary database. The total number of cells of the C C C T , CM , CN ,CN ) is defined as follows. In the case target summary database M (CM X X of method F: X M =  M N q , where C C  represents the cardinality of the N

q=1,...,n

|C |

domain values of the category attribute which is common between the target and C (X /|C C |)(XN /|CN |) proxy summary databases. In the case of method PP: X M = M M . q N

q=1,...,n

|C |

Each cell in the target summary database shows the cost in terms of the number of arithmetic operations performed (time) and the number of bytes (space) needed. The space cost is important in the intermediate steps of processing when methods F and PP are applied to achieve the final result. For instance, let us consider the summary databases shown in Table 1. Applying the F method, the ˆ total number of cells of the summary database Income (State,Race,Age,Sex) is 80, while by the method PP, the total number of cells of the summary database ˆ Income (State,Sex) is 10. Concerning the arithmetic operations, we assume that the multiplication and division operations in each cell need some fixed amount of time, such as 3µs. Therefore, in our example, the time cost for estimating the ˆ target summary database Income(State) by F is 255µs, while by PP it is 45µs. 6.2

Performance Evaluation over Three Summary Databases

The evaluation of a joint query over more than two summary databases is performed by selecting the target summary measure of one database and using all the other databases as proxy databases. Consequently, there are many combinations for estimating the result of such a joint query, depending on the order of evaluation. As examples of our methodology to performance evaluation, we use the summary databases Population(State,Age,Sex), Income(Race,Sex,Profession), P opulationE (Race, Sex, Education-level), which are labelled by a, b, and c in Figure 1. The joint query is Income(State,Educationlevel). In this case, the target summary measure is from b, and the target category attributes belong to a and c. In Figure 1, for example, cb indicates that between b and c, the former is the target database and the latter is the proxy database. For answering this query, we use the methods F and PP. Note that for all the cases reported in Figure 1, the solutions yield the same total number of cells for the target summary database using the methods F and PP. ˆ The target summary database Income (State,Education-level) is the result of: (a) Using method F, the aggregation over Race, Sex, Age, Profession in ˆI ncome(State, Race, Sex , Age, Profession, Education − level ), which has 320 cells; (a) Using method PP, the aggregation over Race and Sex in ˆ Income(State, Race, Sex, Education − level), which has 80 cells. Table 4 shows the cost of processing the intermediate steps in each solution. As can be seen from the table, the minimum space cost is achieved with solution B for both methods F and PP. The same table shows the total time cost (in µs) for estimating the target summary database. Overall, solution B with method PP provides the best performance (least amount of space and least amount of computation).

Answering Joint Queries from Multiple Aggregate OLAP Databases

a

c

b

b

a

c

b a b a,c

b a

b a,c (C)

(B)

b

a

c b c

a

b

a b

b a,c

c b c

a

b

c c b

b a

b a,c

(D)

c

b c a

b a,c

(A)

Fig. 1.

a a c

b c

33

b a,c (F)

(E)

Solutions of joint query on multiple summary databases

Table 4. Cost of Space and Time of intermediate summary databases for each solution Space(byte) Methods Time(µs) Methods Solutions A B C D,E,F

7

F PP 160 32 160 192

40 16 80 40

Solutions

F

PP

A B C D,E,F

1440 1056 1440 1536

360 288 480 360

Conclusions

In this paper, we proposed a method, called the Partial-Pre-aggregation method, for estimating the results of a joint query over two source databases. This method is based on partitioning the category attributes of the source databases into ”common”, ”non-common”, and ”target” attributes. By summarizing on the non-common attributes first, we reduce the computational and space complexity of applying the linear indirect estimator method. We have shown that the PartialPre-aggregation method is, in general, significantly more efficient that the Fullcross-product method commonly used by statistical software. Furthermore, we provided a way to evaluate the optimal order of pairing databases for queries over more than two source summary databases.

References [1] Chan, P., Shoshani, A.: SUBJECT: A Directory Driven System for Organizing and Accessing Large Statistical Databases. Conference on Very Large Data Bases, (1981) 553–563 24

34

Elaheh Pourabbas and Arie Shoshani [2] Gray, J., Bosworth, A., Layman,A.,Pirahesh, H.:Data cube: a Relational Aggregation Operator Generalizing Group-by, Cross-tabs and Subtotals. 12th IEEE Int. Conf. on Data Engineering, (1996) 152–159 25 [3] Codd, E. F., Codd, S. B., Salley, C. T.: Providing OLAP (On-Line Analytical Processing) to User-Analysts: An IT mandate. Technical report (1993) 24 [4] Ghosh, M., Rao, J. N. K. : Small Area Estimation: An Appraisal. Journal of Statistical Science. 9 (1994) 55–93 26, 30 [5] Pfeffermann, D.: Samll Area Estimation - New Developments and Directions. International Statistical Review, 70 (2002) 26 [6] Pourabbas, E., Shoshani, A.: Joint Queries Estimation from Aggregate OLAP Databases. LBNL Technical Report, (2001) LBNL-48750 24, 30, 31 [7] Shoshani, A.: OLAP and Statistical Databases: Similarities and Differences. 16th ACM Symposium on Principles of Database Systems, (1997) 185–196 24

An Approach to Enabling Spatial OLAP by Aggregating on Spatial Hierarchy Long Zhang, Ying Li, Fangyan Rao, Xiulan Yu, Ying Chen, and Dong Liu IBM China Research Laboratory, Beijing 100085, P. R. China {longzh,lying,raofy,yuxl,yingch,liudong}@cn.ibm.com

Abstract. Investigation shows that a huge number of spatial data exists in current business databases. Traditional data warehousing and OLAP, however, could not exploit the spatial information to get deep insight into the business data in decision making. In this paper, we propose a novel approach to enabling spatial OLAP by aggregating on the spatial hierarchy. A spatial index mechanism is employed to derive the spatial hierarchy for pre-aggregation and materialization, which in turn are leveraged by the OLAP system to efficiently answer spatial OLAP queries. Our prototype system shows that the proposed approach could be integrated easily into the existing data warehouse and OLAP systems to support spatial analysis. Preliminary experiment results are also presented.

1

Introduction

OLAP system provides architectures and tools for knowledge workers (executives, managers, analysts) to systematically organize, understand, and use their data to make strategic decisions. It is claimed that 80% of the overall information stored in computers is geo-spatial related, either explicitly or implicitly[4]. Currently a large number of spatial data have been accumulated in business information systems. How to analyze such business data associated with the spatial information presents the challenges to data warehousing and OLAP systems. For example, the business data, such as the growth of surrounding neighborhood, proximate competitors and geographical data, such as the distance to the nearest highways, can be used to choose a location for a new store. To address the above issues, two methods are commonly used to support management of business data with spatial information: (1) GIS + DBMS. In this approach, geographical information system (GIS) is used to model, manipulate and analyze the spatial data, and DBMS is used to handle business data. The disadvantage of this method is that the business data and spatial data are maintained separately and it is hard to provide a uniform view for users; (2) DBMS with spatial extensions. Some commercial database vendors, such as IBM and Oracle, provide spatial extensions in their database systems[3, 11]. These spatial extensions provide methods to execute spatial query, namely, Y. Kambayashi, M. Mohania, W. W¨oß (Eds.): DaWaK 2003, LNCS 2737, pp. 35-44, 2003. c Springer-Verlag Berlin Heidelberg 2003

36

Long Zhang et al.

to select spatial objects in specified areas. However they do not provide spatial data analysis functionality in the OLAP tools. The data in the warehouse are often modelled as multidimensional cubes[5] which allow the data to be organized around natural business concepts that are called measures and dimensions. We adopt Han’s definition of spatial-tospatial dimension as the spatial dimension, whose primitive level and all of its high-level generalized data are spatial[9]. The spatial dimension differentiates the non-spatial dimension in that there is little a priori knowledge about the grouping hierarchy[12, 13]. In addition to some predefined regions, the user may request groupings based on a map which are computed on the fly, or may be arbitrarily created (e.g. an arbitrary grid in a selected window). Thus the wellknown pre-aggregation methods which are used to enhance performance can not be applied. In this paper, we present an approach to enabling spatial data manipulation into OLAP systems. We extend the cube query by introducing spatial predicates and functions which explicitly express the spatial relationships among data in fact tables and dimension tables. A spatial index mechanism is employed to derive the spatial hierarchy for pre-aggregation and materialization, which in turn is leveraged by OLAP system to efficiently answer spatial OLAP queries. Our prototype system shows that the proposed approach could be integrated easily into existing data warehousing and OLAP systems to support spatial analysis. Preliminary studies on the performance of the system are presented. The remainder of this paper is organized as follows. A motivating example is given in Section 2. The spatial index mechanism is described in Section 3. The system architecture is depicts in Section 4. Section 5 gives a detailed explanation of query processing. Preliminary experiments are given in Section 6. Related works are given in Section 7 and conclusion is drawn in Section 8.

2

Motivating Example

In our work, the star schema is used to map multi-dimensional data on a relational database. To focus on spatial dimensions, we employ a simple data warehouse of sales from thousands of gas stations as the motivating example. The sales concerning different gas stations, gas types, customers at different time are analyzed. The schema of dimension tables station, gas, customer, time and fact table transaction is as follows: station (station ID, location) gas (gas ID, unit price) customer (customer ID, type) time (time ID, day, month, year) transaction (station ID, gas ID, customer ID, time ID, sales, quantity) Each tuple in the fact table is a transaction item indicating sales, customer ID, gas type, gas station, and time involved in the transaction. Here, station is a spatial dimension. Its attribute location gives the spatial location of a gas station. A location is a point such as “(332, 5587)”. The typical OLAP query

An Approach to Enabling Spatial OLAP by Aggregating on Spatial Hierarchy

37

may be: “What are the total sales of each type of gas for ‘taxi’ customers at each gas station in Oct. 2002?” While an OLAP query involving spatial information may be “What are the total sales of each type of gas for customer ‘taxi’ at gas stations within the query window in Oct. 2002?” Here, a query window is a rectangle that user draws on the map.

3

The Spatial Hierarchy and Pre-Aggregation

In OLAP system, concept hierarchy is the basis for two commonly used OLAP operators: drill-down and roll-up[8]. A concept hierarchy defines a sequence of mappings from a set of low level concepts to higher level, more general concepts. For example, the hierarchy on time dimension is day, month and year. Traditionally, users, field experts and knowledge engineers manually define the concept hierarchies of dimensions. But for the spatial dimension, there is little a priori knowledge available on how to define an appropriate hierarchy in advance. Spatial indexing has been one of the active focus areas in recent database research. There have been many data structures proposed for indexing spatial objects[6], such as Quadtree, R-tree (and its derivations such as R*-tree and R+-tree), and grid file. Among them, those indexes with tree structures provide nesting relationship between high-level nodes and low-level nodes. This relationship might be inspiring candidates for building the spatial hierarchy. Each node in R-tree stores a minimum bounding rectangle (MBR) and the MBR of higher level node encloses all MBRs of its descendants. Thus R-tree provides a naturally nested hierarchy based on the spatial shapes and placement of the indexed objects. This R-tree derived spatial hierarchy plays the same role as traditional concept hierarchies. It can be used for pre-aggregation on spatial dimension. Other tree-like indexes can also be used to derive the spatial hierarchy. Fig. 1 indicates a part of a sample R-tree for the motivating example and the spatial layout of the objects. The leaf entries are gas station IDs. Each intermediate R-tree entry has an associated MBR indicating its spatial scope. r2 r5

r1 r3 s2

r6

r3

r2

r9

s3

r4

s1

s4

r5

r6

r7

r8

r9

r10

r11 r7

...... ..................

...... r11

s1 s2 s3

r8

s4

s5

r4

s5 r10

Fig. 1. A part of sample R-tree and spatial layout of the objects

Assuming that a user draws a query window indicated as bold rectangle in Fig. 1, our Spatial Index Engine (will be introduced in Section 4) uses R-tree to compute the query. It must return exact objects within the query window. Index searching algorithm tries to find higher level nodes in the tree that satisfy spatial

38

Long Zhang et al.

query predicates, more specifically, the within predicate. In some cases, some gas stations are contained in the query window, but MBRs of their ascendant nodes are not completely enclosed by the query window. Thus these gas stations should be fetched instead of their ascendant nodes. According to this, for the query window in Fig. 1, intermediate R-tree entries {r2, r7 } and leaf entries {s2, s3, s4 } are returned. With the spatial hierarchy, pre-aggregation and materialization can be built to answer OLAP queries. Usually, the result of materialization is stored in a summary table. For our motivating example, the summary table concerning spatial dimensions is as follows: spatial_sum (nid, customer, gas, month, year, sales) NIDs are IDs of R-tree nodes. Currently, we materialize the whole cube; that is, all nodes in spatial hierarchy (R-tree) will be computed and the result will be inserted into summary table. The summary table is built up traversing aggregating paths. For example, in Fig. 1, gas stations s1, s2 and s3 are grouped into node r9, and in turn, r7, r8 and r9 are grouped into r3 and so on. For each intermediate node in the spatial hierarchy, all other non-spatial dimensions will be aggregated and materialized. DB2 provides OLAP extensions[2] to SQL and we use Group By Cube to aggregate and materialize non-spatial dimensions. For example, the corresponding pre-aggregation and materialization SQL statement for r9 is: INSERT INTO spatial_sum SELECT "r9" AS nid, c.type AS customer, g.gas_id AS gas, t.month, t. year, sum(tr.sales) AS sales FROM station s, customer c, gas g, transaction tr, time t WHERE tr.station_id in ("s1", "s2", "s3") GROUP BY CUBE (customer, gas, month, year) Using this method, all nodes in the spatial hierarchy are aggregated and materialized into table spatial sum.

4

System Architecture

Our prototype system executes spatial OLAP queries, accessing multidimensional data cubes stored in a relational database and presents the result in a multidimensional format. Fig. 2 depicts the architecture of our system. Warehouse Builder extracts, transforms and loads raw data from operational database systems (ODS) to build fact table and dimension tables. Before summary tables are created, Spatial Index Engine uses dimension tables containing spatial data to build spatial hierarchy. The hierarchy then is used by Warehouse Builder when generating summary tables. User query is generated in Graphic User Interface (GUI) on client/browser side, encoded in XML and then transferred to server side. Spatial OLAP Query Processor extracts all spatial predicates from the query and dispatches them to Spatial Index Engine. By investigating Spatial Index, the engine fetches all

An Approach to Enabling Spatial OLAP by Aggregating on Spatial Hierarchy

39

Graphic User Interface

Clinet/Browser side query (XML)

result (XML)

Server side spatial predicate

Spatial OLAP Query Processor

result set

query

Spatial Index Engine

summary info.

Java OLAP API

Spatial Index

ROLAP Engine

ODS Fact Table

summary table

Warehouse Builder

Dimension Table

ODS

ODS

Fig. 2. System architecture

intermediate R-tree nodes whose MBRs satisfy the predicates and individual leaf nodes (i.e. gas station IDs in our motivating example) whose spatial locations satisfy the predicates. These nodes are returned to Spatial OLAP Query Processor. With these data, the processor rewrites the original query and sends the rewritten query to ROLAP Engine, which is a relational OLAP engine. ROLAP Engine processes the query using summary tables and returns query results to the query processor. Query results are further reconstructed as XML document and sent to client/browser side. The architecture takes the advantage of independent Spatial Index Engine. Thus the system could employ other index structures, such as Quadtree, with little modifications on other modules. The interface between query processor and ROLAP Engine is a Java OLAP API referring to JOLAP API specification[10]. With this API, any OLAP engines complying with it can be easily adopted.

5

Spatial OLAP Query Processing

We use the common query region shape - a rectangle drawn on a map to analyze spatial OLAP query processing. There are two types of queries: – Summary query (SQ). It requests summary information of all selected spatial objects as a whole, e.g. “what are the total sales of each customer at ALL gas stations within my indicated query window during each month in 2002?” – Individual query (IQ). It asks for individual information based on each selected spatial object, e.g. “what are the total sales of each customer at EACH gas station within my indicated query window during each month in 2002?” Obviously, SQ queries investigate and take the advantage of the data already summarized on spatial dimension while IQ queries can not exploit the summary information and must be processed as traditional OLAP queries. To deal with these two types separately, we built different cubes for them. The schema of summary table concerning spatial information is introduced in Section 3. We established the following traditional cube that is based on individual gas stations for IQ queries:

40

Long Zhang et al.

nonspatial_sum (station_id, customer, gas, month, year, sales) Taking a query as an example, “what are the total sales of each customer at all gas stations within my indicated query window during each month in 2002?”. The indicated rectangle query window with upper left corner (122, 220) and lower right corner (500, 768) is depicted in Fig. 1. The SQL statement is: Q1: SELECT c.type as customer, month, sum(sales) FROM customer c, gas g, time t, station s, transaction tr WHERE s.location WITHIN query_window(122,220,500,768) AND t.year= 2002 AND tr.customer_id=c.customer_id AND tr.gas_id=g.gas_id AND tr.station_id=s.station_id AND tr.time_id=t.time_id GROUP BY customer, month When the query is being processed, all the gas stations located within the window will be selected and their sales information will contribute to the query result. After the query is generated in GUI and transferred to the server side, the Spatial OLAP Query Processor extracts the spatial predicates in the WHERE clause. Then query processor dispatches each predicate to Spatial Index Engine. The engine deals with the predicate and returns an NID list containing IDs of all R-tree nodes which satisfy the predicate and a gas station ID list containing all IDs of separate gas stations. For example, for predicate query window (122, 220, 500, 768), Spatial Index Engine returns two list: one NID list containing all intermediate R-tree nodes whose MBRs are within the specified query window (122, 220, 500, 768); the other list of gas station IDs containing all gas stations whose locations are also within the query window, but MBRs of their ascendants are not completely enclosed by the query window. Let the NID list be {23, 179, 255, 88} and the gas station ID list be {868, 3234, 843, 65, 7665}. Then Q1 can be rewritten into: Q2: WITH summary AS (SELECT nid AS id, customer, month, sum(sales) FROM spatial_sum WHERE nid IN (23,179,255,88) AND year=2002 AND gas is null AND year is null UNION ALL SELECT station_id AS id, customer, month, sum(sales) FROM nonspatial_sum WHERE station_id IN (868,3234,843,65,7665) AND year=2002 AND gas is null AND year is null) SELECT customer, month, sales FROM summary GROUP BY customer, month The processing of IQ queries will not investigate the summary data based on spatial information. The user is not interested in the total information on

An Approach to Enabling Spatial OLAP by Aggregating on Spatial Hierarchy

41

all gas stations as one, but each individual gas station in the query window; that is, “what are the total sales of each customer at EACH gas station within my indicated query window during each month in 2002?” Spatial Index Engine will only return one list containing all individual gas stations whose locations confirm to the specified spatial predicate. The corresponding query is: Q3: SELECT c.type as customer, tr.station_id as station, t.month, sum(sales) FROM customer c, station s, time t, transaction tr WHERE s.location WITHIN query_window(300,220,990,882) AND t.year=2002 AND tr.station_id=s.station_id AND tr.time_id=t.time_id AND tr.customer_id=c.customer_id GROUP BY customer, station, month Assuming the gas station IDs returned by the index engine are {868, 3234, 843, 65, 7665}, Q3 can be rewritten as Q4, where non-spatial summary table is used. Q4: SELECT customer, station, month, sum(sales) FROM nonspatial_sum WHERE station IN (868, 3234, 843, 65, 7665) AND year=2002 AND gas is null

6

Experiments

Preliminary experiments are conducted on the simplest shape: point. Our aim is to find whether our approach is effective for spatial OLAP queries and more complex shapes such as line and polygon will be discussed in further work. Two data sets were generated according to the motivating example. The locations of gas stations for each data set comply with Gaussian distribution. By this, we attempted to reflect the fact that gas stations tend to be clustered around areas with dense road network. Table 1 gives the statistic information of these data sets: Table 1. The statistic data of base tables set # of gas stations # of customers # of transactions 1 2

1,000 10,000

200 800

8,663 34,685

Spatial index was built on each data set. We computed and materialized the whole cube by using the spatial hierarchy. The materializing result is stored in table spatial sum. Materialized cube for other non-spatial dimensions is stored in table nonspatial sum. The statistic data are given in Table 2. DB2 UDB and Spatial Extender were used as the database server to support storing spatial objects. Our system runs on a desktop machine with Pentium III 750M CPU and 512M memory.

42

Long Zhang et al. Table 2. The statistic data of spatial index and summary tables Set R-tree size R-tree depth Size of nonspatial sum Size of spatial sum 1 2

1,424 14,182

6 9

25,120 126,176

30,490 195,616

We compare our approach with the one in which spatial dimension is not aggregated. For simplification, our approach is called spatial aggregation (SA) approach and the other one non-spatial aggregation (NSA) approach. In order to restrict discussion to the main concerns, we use the following typical query in the following experiments. Find out the total sales for every gas type in every month for all gas stations located in the specified query window. Obviously, with spatial hierarchy not aggregated, to answer the spatial query, base table station must be searched in order to fetch all IDs of gas stations located among the query window. The query for NSA is SELECT gas, month, sum(sales) FROM nonspatial_sum WHERE gas is not null AND month is not null AND station_id IN (SELECT station_id FROM station WHERE location..xmin>$left AND location..xmin<$right AND location..ymin>$bottom and location..ymin<$top) Here, $left, $right, $bottom and $top indicate the border of the specified query window. It should be mentioned here that if the objects have more complex shapes, such as lines and polygons, searching objects whose location are within a query window is a time-consuming task. In our approach, the better part of this task has been done when building spatial index: the objects have been grouped into the nodes of spatial hierarchy. The searching cost is reduced dramatically. 1200

12000 SA NSA

1000

10000

800

8000 Number of tuple access

Number of tuple access

SA NSA

600

6000

400

4000

200

2000

0

0

10

20

30

40

50 Area%

60

70

80

90

0

10

20

30

40

50 Area%

60

70

80

90

(b) 10,000 gas stations (a) 1,000 gas stations Fig. 3. Aggregation degree

In order to investigate the aggregation ratio of our spatial hierarchy, for each data set, we generated 9 or more query sets, each containing 100 queries. The query windows in every query set have equal area and the centers of the windows

An Approach to Enabling Spatial OLAP by Aggregating on Spatial Hierarchy

43

comply with Gaussian distribution. The proportions of window area to the whole space are predefined as 10, 20, 30, ..., 90% for these query sets. Fig. 3 depicts the average numbers of tuple-access by NSA and SA methods on each set. It can be derived from Fig. 3 that with the area of query window growing, the number of tuple-access by SA approach drops dramatically, while that by NSA approach increases quickly. At any area ratio, the number of accessed tuples by SA is much less than that by NSA, usually more than an order of magnitude. Fig. 4 displays the performance comparison for NSA and SA on above query sets. It can be seen from the figure that with the window area groups, SA always performs much better than NSA. This result also complies with the trend of aggregation degree presented in Fig. 3. 1

0.14

0.9

SA NSA

0.12

SA NSA

0.8

0.7

Elapse time (s)

Elapse time (s)

0.1

0.08

0.06

0.6

0.5

0.4

0.3

0.04

0.2 0.02

0.1 0

10

20

30

40

50 Area%

60

70

80

90

0

0

10

20

30

40

50 Area%

60

70

80

90

(a) 1,000 gas stations (b) 10,000 gas stations Fig. 4. The elapse time of SA and NSA

7

Related Works

Although advances in OLAP technology have led to successful industrial decision supporting systems, the integration of spatial data into data warehouse for supporting OLAP has only recently become a topic of active research. Han et al. [9, 14] are the first ones to propose a framework for spatial data warehouses. They proposed an extension of the star-schema and focus on the spatial measures and propose a method for selecting spatial objects for materialization. It was differed from our work that they focused on spatial measures while we concentrated on the spatial relationship and the spatial query processing. Papadias et al. [12, 13] gave an approach to deal with the problem of providing OLAP operations in spatial data warehouse, i.e. to allow user to execute aggregation queries in groups, based on the position of objects in space. Both Han’s and Papadias’ approaches modified the classical star schema in the existing data warehouses. The work in this paper managed to reserve the star schema while representing the spatial data. The pre-aggregation and materialization is the traditional method to enhance the performance of OLAP systems [1, 7]. Due to the lack of a priori knowledge about the spatial hierarchy, these well-known methods can not be applied directly. Papadias’ method stores aggregation results in the index. In our method, the results are simply stored in relational tables which are separated from the spatial index.

44

8

Long Zhang et al.

Conclusion

Currently a large number of spatial data have been stored in business information systems. In order to analyze these data, traditional OLAP systems must be adjusted to handle them. From a user’s perspective, it is not convenient to process spatial data separately before OLAP analyzing as that current OLAP systems deal with spatial information. In this paper, we discussed the feasibility of enabling spatial OLAP in traditional OLAP systems by aggregating on spatial hierarchy. In our proposed approach, spatial hierarchy can be automatically established. User issues queries involving spatial predicates and the system will compute these predicates automatically using spatial index. Pre-aggregation and materialization techniques are engaged under the help of the spatial hierarchy.

References 1. Baralis, E., Paraboschi, S., Teniente, E.: Materialized View Selection in a Multidimensional Database. Proceedings of VLDB Conference, 1997. 2. Colossi, N., Malloy, W., Reinwald, B.: Relational extensions for OLAP. IBM SYSTEMS JOURNAL, VOL 41, NO 4, 2002. 3. David W. Adler. DB2 Spatial Extender - Spatial data within the RDBMS. Proceedings of VLDB Conference, pp 687-690, 2001. 4. Daratech: Geographic Information Systems Markets and Opportunities. Daratech, Inc. 2000. 5. Gray, J., Chaudhuri, S., Bosworth, A., et al.: Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals. Data Mining and Knowledge Discovery Journal, Vol 1, 29-53 ,1997. 6. Gaede, V., Gnther, O.: Multidimensional Access Methods. ACM Computing Surveys, 1997. 7. Gupta, H.: Selection of Views to Materialize in a Data Warehouse. Proceedings of International Conference on Database Theory , 1997. 8. Han, J., Kamber, M.: Data Mining: Concepts and Techniques. Morgan Kaufmann Publisher, Inc. 2001. 9. Han, J., Stefanovic, N., Koperski, K.: Selective Materialization: An Efficient Method for Spatial Data Cube Construction. Proceedings of PAKDD Conference, 1998. 10. Java OLAP Interface (JOLAP), version 0.85. Java Community Process, May 2002. Available at http://jcp.org/en/jsr/detail?id=069. 11. Kothui, R.K.V., Ravada, S., Abugov, D.: Quadtree and R-tree Indexes in Oracle Spatial: A Comparison using GIS Data. ACM SIGMOD Conference, 2002. 12. Papadias, D., Kalnis, P., Zhang, J., Tao, Y.: Efficient OLAP operations in spatial data warehouses. Technical Report HKUST-CS01-01, Jan. 2001. 13. Papadias, D., Tao, Y., Kalnis, P., Zhang, J.: Indexing Spatio-Temporal Data Warehouses. Proceedings of International Conference on Data Engineering, 2002. 14. Stefanovic, N., Han, J., Koperski, K.: Object-Based Selective Materialization for Efficient Implementation of Spatial Data Cubes. TKDE 12(6): 938-958 (2000)

A Multidimensional Aggregation Object (MAO) Framework For Computing Distributive Aggregations Meng-Feng Tsai and Wesley Chu Computer Science Deptartment University of California, Los Angeles, 90095 [email protected], [email protected]

Multidimensional aggregation plays an important role in decisionmaking systems. A conceptual Multidimensional Aggregation Object (MAO), which consists of measures, scopes and aggregation function, is introduced to represent relationships among aggregators on addressable subsets of data. In the MAO model, aggregations of low-level (intermediate) data can be reused for aggregations on high-level data along the same dimension. Experimental results show that caching intermediate aggregated data can signi cantly improve performance. Incremental compensating and full recomputing cache-updating approaches are proposed. Execution plans for deriving the aggregations from MAOs are presented. The proposed data aggregation technique can be applied to data-warehousing, OLAP, and data mining tasks. Abstract.

1

Introduction

In multidimensional DB, current research focuses on accessing raw data and performing simple data aggregations. In [Venk 96] and [SAgr 96], lattice structure and data caching algorithms are studied but they can only support few aggregators such as MAX, MIN, SUM, AVG. [Mumi 97] studied intermediate data maintenance for these few aggregators. The multidimensional structure model was presented in [RAgr 97]. Research about derivability in multidimensional environments was presented by [Albr 99], but the focus was on predicates of queries and did not support general aggregators. Commercial products are available for performing simple summarization for OLAP systems. Decision support systems, however, require complex aggregate functions such as variance to analyze and summarize data, which is the topic of this paper. Consider an airline enterprise needs to evaluate the eÆciency of its operations. For example, to detect abnormal operations, we need to evaluate the variances of fuel consumption, arrival and departure delays, and the number of passengers for dierent dates, aircrafts and ights, which requires the summarization of data values from dierent aspects and at dierent granularity levels. Due to limitations of the conventional aggregation techniques, many useful functions such as variance, high orders of moments, and a combination of Y. Kambayashi, M. Mohania, W. W¨oß (Eds.): DaWaK 2003, LNCS 2737, pp. 45-54, 2003. c Springer-Verlag Berlin Heidelberg 2003

46

Meng-Feng Tsai and Wesley Chu

several simple aggregations are unable to be supported. To remedy this shortcoming, we propose a multidimensional aggregation methodology that supports the dimensional hierarchies, and also systematically exploits relationships among aggregation functions, therefore allowing the customization of aggregators. This paper is organized as follows. Section 2 proposes the Multidimensional Aggregation Object (MAO) to represent the inter-relationships among aggregations at dierent levels. Section 3 discusses the derivation relationships among MAOs. Section 4 introduces caching methods to enhance the system performance, and also includes experiments to show performance improvements. Section 5 presents methods to synchronize cached data for updates, and section 6 provides a language to specify metadata of execution plans for processing MAOs. 2

Multidimensional Aggregation Object (MAO)

In multidimensional environments, data can be referred to by dierent dimensions at dierent levels. \Scope" is the entity representing the entire addressable data set at dierent levels. In the airline enterprise example, suppose there are three dimensions: a date dimension at the date and the weekday level, an aircraft dimension with only one aircraft level, and a ight dimension classi es data at the ight number and the time block levels. A scope is a combination of dierent dimensions from dierent levels. In the airline enterprise example, the scope (W D; AC; T B ) (in Figure 1) is from date, aircraft, and ight dimensions to weekday, aircraft, and time-block levels. The generalization of \scopes" from top (most generalized) to bottom (most speci ed) is rst proposed by [SAgr 96]. A lattice structure of scopes is shown in gure 1, where (W D; AC; f #) can derive (W D; AC; T B ) since time block (TB) is a generalization of ight number (f#) in the ights dimension. Aggregation functions in multidimensional environments can be categorized into three types [SAgr 96]: Distributive, Algebraic, and Holistic. We will focus on distributive and algebraic functions. Let the input data set be S with size n. Each entry in S is referred to as si . For distributive aggregations, an aggregate function is distributive if the results can be derived incrementally by repeatedly applying an associative and commutative binary operation Op(s1; s2), where s1 is the ongoing accumulating result, initially an empty value, and s2 is a newly selected entry from the remaining data set. As a result, distributive aggregation is capable of processing upper dimension level data from lower level data and the aggregated results are invariant to the sequence of those binary operations. For algebraic aggregators, an algebraic function maps from a nite group of values into a single value. This kind of function can consist of any constants or outputs from other aggregate functions. To present relationships among complex aggregators, a conceptual \object" can be used to encapsulate the necessary information. A multidimensional aggregate object (MAO) is de ned as a tuple with three elements M AO = (M; S; F ) where M = (M1 ; :::; Mm ) is a set of entries of measures; S = (l1 ; :::ln ) is a scope, com-

A Multidimensional Aggregation Object (MAO) Framework

47

/ => roots; WD=>weekdays, Date=>day in 1 month; AC=>aircrafts; TB=>time block; f#=> flight number /, /, /

WD,/ ,/ [7]

/, AC, / [14]

/, /, TB [4]

Date,/ ,/ [30]

WD, AC, / [98]

WD,/ ,TB [28]

/, AC, TB [56]

/, /, f# [53]

Date,AC, / [420]

Date,/ ,TB [120]

WD,AC,TB [392]

WD,/ ,f# [371]

/ ,AC, f# [742]

Date,AC, TB [1680]

Date,/ ,f# [1590]

WD, AC, f# [5194]

Date, AC, f# [ ] = number of possible entries for each scope.

Fig. 1.

Using scopes to present dierent levels of granularity of data.

posed by combinations of dierent dimension levels Di ; and F is the aggregate function that applies to M and S . MAO is the key identity for the derivation relationship that exists between the scopes and aggregation functions. A portion of the MAO derivation relationships for the airline enterprise example is shown in Figure 2.

/ => roots; WD=>weekdays, Date=>day in 1 month; AC=>aircrafts; TB=>time block; f#=> flight number "number of passengers" => #p (measure) #p,(WD,AC,/), Var

#p, (WD,AC,/), Sum

#p, (WD,AC,/),{SqrSum, Sum}

#p, (Date,AC,/), Var

#p, (Date,AC,/), Sum

#p, (Date,AC,/),{SqrSum,Sum}

#p, (WD,AC,TB), Var

#p, (WD,AC,TB),Sum

#p, (WD,AC,TB), {SqrSum,Sum}

note: Variance(Var) can be derived by SqrSum and Sum. (assuming that the function "count" is given) ->

Fig. 2.

ple.

Variance(X) := Sum [sqr(xi - Avg(X)) ] := Sum [sqr(xi)] - Sum(xi)Avg(X).

A portion of the MAO derivation relationships for the airline enterprise exam-

48

3

Meng-Feng Tsai and Wesley Chu

Invariance in MAO Derivation

If a given MAO Mi operates on a given scope with a certain measure and can be derived from another MAO Mj , then we say Mi is derivable from Mj , denoted as Mj ! Mi . We will prove that any MAO derived from multiple lower level MAOs will generate the identical aggregated results. Algebraic aggregation serves as a mapping function among MAOs with the same scopes. If two MAOs are related through an algebraic aggregate, then the implementation of a correct mapping function is essential. For distributive aggregations, we will rst prove that the results from aggregating partitions of the data set are identical despite their dierent partitioned data sets. Let the aggregated results of MAO Mj be Agj and each entry in Mi be ix . If Mi ! Mj through distributive aggregation f , and there are k partitions formed by the dimension hierarchy, then Agj = ff (i1 ; i2 ; :::; ip1 ); f (ip1+1 ; :::; ip2 ); :::; f (ip(k 1)+1 ; :::; ipk )g.

Distributive aggregations using an associative binary operator can be derived by aggregating from dierent partitions on a given data set and yield the identical aggregated result. Lemma 1

Proof: Note that the distributive aggregation uses an associative and commutative binary operator. Let n be an arbitrary positive integer, and let a1 ; a2 ; :::; an be elements from the set S , if associative property holds for S . Then for any partition on S , say on the r-th element, we have the generalized associative property: a1  a2  :::  an = (a1  a2  :::  ar )  (ar +1  :::  an ) where  is an associative binary operator. Repeatedly apply the generalized associative property to two arbitrary sets of partitions. At the rst k partition, we have: (a1  :::ap1 )  (ap1+1  :::ap2 ):::  (ap(k 1)+1  :::an )) = (a1  a2  :::  an ) This holds for another l partition. (a1  :::aq1 )  (aq1+1  :::aq2 ):::  (aq(l 1)+1  :::an )) = (a1  a2  :::  an ) Therefore, two aggregations on any two arbitrary partitions of the same data set generate the identical result. Based on lemma 1 we can prove the following theorem:

For a given pair of MAOs, (Mx ; My ), if My can be derived from through multiple distributive aggregations, then all the aggregated results are x identical. Theorem 1 M

Proof: Consider each path between any pair of derivable MAOs with multiple derivation paths, say Mx ! My . Each resulting entry in My is derived from more detailed entries in Mx , through a combination of partitions, formed by involved dimension hierarchies. Say an entry ty^ in My is derived from a set of entries Tx = ft1 ; t2 ; :::; tl g from Mx , which means the aggregated result ty ^ = f (t1 ; t2 ; :::; tl ), where f is a distributive function using a binary operator . Assume there are two paths for Mx to derive ty^, that is Mx ) Mi ! My ,

A Multidimensional Aggregation Object (MAO) Framework

49

and Mx ) Mj ! My . Thus ty^ can also be derived from a group of entries from Mi or Mj . Consider Tx , which contributes a set of entries Ti = fti1 ; ti2 ; :::tim g for Mi , and another set Tj = ftj1 ; tj2 ; :::; tjn g for Mj . Since direct aggregation is summarized across one level of dimension, Mi and Mj can be considered as intermediate results aggregated by dierent partitions from Mx . Thus Ti = ff (t1 ; :::; tp1 ); f (tp1+1 ; :::; tp2 ); :::; f (tp(m 1)+1 ; :::; tl )g, and similarly Tj = ff (t1 ; :::; tq1 ); f (tq1+1 ; :::; tq2 ); :::; f (tq(n 1)+1 ; :::; tl )g. So when derived from Mi , the corresponding entries are: t ^ = f (f (t1 ; :::; tp1 ); f (tp1+1 ; :::; tp2 ); :::; f (tp(m 1)+1 ; :::; tl )), and y2 t ^ = f (f (t1 ; :::; tq 1 ); f (tq 1+1 ; :::; tp2 ); :::; f (tq (n 1)+1 ; :::; tl )). From lemma 1 we y3 know ty^ = ty^2 = ty^3 . Therefore the theorem is correct under the same operator.

The aggregated results from a distributive aggregation for a MAO that are derived from its multiple lower level MAOs are identical. Corollary 1

Proof: For a given measure, all the distributive MAOs are aggregations based on dierent partitions but on the same raw data set (most detailed data). Suppose the MAO with the most detailed scope is Mx . Because all other MAOs are considered to be derived from Mx directly or indirectly, from theorem 1 we know all distributive MAOs always generate the identical aggregating results. 4

Performance Improvements of Using MAO with Caching

Caching intermediate MAOs can provide signi cant performance improvements by using inputs from lower-level MAOs, which have fewer entries than the detailed fact data. We use the greedy algorithm [Venk 96] that at each step selects the best candidate to cache the MAOs. [Venk 96] proves that the upper bound of the bene t derived from the greedy algorithm from a lattice data structure is at least (e 1)=e ' 63% of that from the optimal solution where e = 2:73. The MAOs in our implementation form a partial order set rather than a lattice. However, the proof is applicable on the partial order set as well. 4.1

Experimental Results

The performance improvements are measured by the number of entries saved by caching. For comparison, scope is used when MAO is not used. Both scope and MAO use the same caching schemes. For example, if a MAO M (or a scope when MAO is not used) can be derived from another cached MAO M (or scope) with N1 entries rather than M with N2 entries (N2 > N1 ), the performance improvement for computing M is N2 N1 . For a given set of MAO SM , The total performance improvement is the sum of improvements for all MAOs in SM from caching. Thus the performance improvement increases as the number of cached MAOs increases.

50

Meng-Feng Tsai and Wesley Chu

We assume that there are four dierent aggregations involved: Sum, Square Sum, Variance, and Average. For the airline enterprise example, the three dimensions are: dates, aircrafts, and ights. The three measures are number of passengers, delay minutes, and fuel burn-o. For the international trading corporation database, the data is classi ed into three dimensions: time, which spans from 12 years on 144 months; product type, which consists of 4 major categories and 18 minor ones; area, which consists of 5 countries and 45 regions. The measures used are: sale price, customer numbers, and cost per item. We assume aggregations are needed by all scopes. All scopes and MAOs are equally likely to be accessed, the cached limit for both scope and MAO are set at 15. The greedy algorithm is used for cache placement. As shown in Figure 3a), for the case without MAO, the performance improvement is insensitive to the number of cached scopes. While for the system using MAO, the performance improvements increase signi cantly as the number of cached MAOs increase. Similar observations can also be seen in the international trading company example in Figure 3b). 9e+06

7e+07 using MAO without MAO(scope) performance improvements

performance improvements

8e+06 7e+06 6e+06 5e+06 4e+06 3e+06 2e+06 1e+06 0

using MAO without MAO(scope)

6e+07 5e+07 4e+07 3e+07 2e+07 1e+07 0

0

2 4 6 8 10 12 14 16 a) number of cached MAOs or scopes

0

2 4 6 8 10 12 14 16 b) number of cached MAOs or scopes

performance comparisons for the airline enterprise example and the international trading company example. Fig. 3.

Because of the problem size, the performance improvements of the international company yield much larger than those of the airline enterprise. Thus the performance improvement is more signi cant when using MAO for larger problem size. 5

Data Synchronization

We shall now discuss the methods for handling intermediate cached data during data updates. There are two approaches based on whether an inverse aggregation function exists.

A Multidimensional Aggregation Object (MAO) Framework 5.1

51

Incremental Compensating Updates

An eÆcient way to compensate updates is to incrementally update the aggregation results according to the changes in input data. This can be accomplished if the corresponding inverse aggregating functions exist. Formally, a function g is called an inverse function of f if for any input set S , g (S )  f (S ) = I , where I is the identity that for any element a, aI = I a = a. When the aggregation is not distributive or an inverse function does not exist, this process may still be done indirectly through transformation to other distributive and inversable MAO. For example, variance is not distributive but can be derived from summation and the summation of squares. Both of the two operations are distributive and have their inverse functions (substraction of sums of data, or sums of squares). Therefore variances can be synchronized without totally recalculating the entire set of updated data if we maintain both sums of data and sums of squares. 5.2

Complete Reaggregation

When impossible or ineÆcient to compensate update, we have to recalculate the entire new data set. To reduce the computing and retrieving costs, we can compute intermediate MAOs rst, then use them to derive higher level MAOs. Therefore the procedure performs in \stages." The cached MAOs that need to be synchronized are the rst stage, then the precedure looks for each of their immediate descendants as candidates for the second stage. By applying the greedy approach again, we search the best candidates one by one until no cost improvements available. Repeating this process, we will nally reach the most detailed fact data where updates were generated. We can derive MAOs at each stage in reverse order until the targets at the rst stage can be derived. Suppose the maximun number of disjointed cached MAOs (can not derive one another) is m and the depth for MAO deriving relationships is limited at d. Then the number of stages is bound by m and d. The greedy process for each stage has an upper bound of m2 (since probing the set in the next stages is less than m). Thus the reaggregation process is bound by (min(m; d)  m2 ). 6

An Execution Plan

The metadata that speci es how to execute the derivation and synchronization consists of three categories: establish derivation relationships among MAOs, estimate costs for cache placement and maintanence, and synchronize the cached intermediate data. An Execution Plan (EP) is proposed to specify this information as shown in Figure 4. Since both computation and retrieval time are linearly related to the size of the processed data, computation time complexity is based on the size of sources and updates.

52

Meng-Feng Tsai and Wesley Chu

<Execution_Plans>-> <Synchronization_scheme> -> <Sources> <Sources> -> mao-list -> mao-list -> | -> current_partial_sum, binary_op, cursor_on_sources -> mapping_function, cursor_on_sources. -> -> -> function1, size_of_sources -> function2, size_of_sources -> | -> accessing_cost, computing_cost -> -> function1, size_updated_sources -> function2, size_updated_sources <Synchronization_scheme> -> -> -> binary_op, insertion_data, inverse_op, deletion_data -> search_for_indirect_sources -> | -> current_partial_sum2, binary_op ,cursor_on_updated_sources -> mapping_function, cursors_on_updated_sources. Fig. 4.

Execution Plan, language and content.

A Multidimensional Aggregation Object (MAO) Framework

53

An example of EP is shown in Figure 5. The system uses this EP to establish and maintain relationships between the source and target MAOs. The derivation relationship re ects that square sums of total passengers by dierent weekdays, aircrafts, and time blocks can be derived from the square sums by dierent dates, aircrafts, and time blocks. Since square sum is a distributive function, this is a distributive relationship and enables the incremental compensation approach for data synchronization. Cost estimations provide information to guide the system for cache placement and synchronization. ((Date,AC,TB),SqrSum, # of passengers) -> ((WD,AC,TB), SqrSum, # of passengers) Derivation_Relationship: Source: ((Date,AC,TB),SqrSum, # of passengers) Target: ((WD,AC,TB), SqrSum, # of passengers) ## W : cursor on target entry; X,Y: cursor on source entry; S: accumulating result. Distributive_Aggregation: if sources are raw data(most detailed fact) => W = X*X + S else W = Y + S Cost_Estimation: For_caching: Caching_computation: f1(size_of_source) Caching_retrieval:f2(size_of_source) For_synchronization: Compensation_cost: computation_cost: f1(size_of_inserted) + f3(size_of_deleted) access_cost: f2(size_of_inserted + size_of_deleted) Recomputation_cost: compute:f1(size_of_updated_source) access:f2(size_of_updated_source) Sunchronization_scheme: direct_compensation: if source are raw data => W = W + I*I - D*D else W = W + I - D indirect_compensation: *pointer for source’s compensation plan. ## recomputation_scheme is the same as in Distributive_Aggregation except setting the source’s cursors on updated source Fig. 5.

7

An example of Execution Plan.

Conclusion

In this paper, we introduced a Multidimensional Aggregation Object (MAO) model which consists of the aggregation function, the aggregation values, and the aggregation scope in a multidimensional environment. MAO represents the aggregated values in a multidimensional structure, and provides information to reuse lower-level and simpler aggregations for compositive aggregations. This information can improve performance and maintain potential data dependancies. The caching placement algorithm is proposed to eÆciently reuse intermediate aggregation results. Because the MAO model provides more information on

54

Meng-Feng Tsai and Wesley Chu

aggregation than presenting data at dierent levels by scope, caching MAOs provides signi cant performance improvements as compared to conventional techniques by caching scopes. To maintain the cached data during updating raw data, two techniques can be used to synchronize the cached data. If an inverse aggregation function exists, then the incremental approach should be used, which uses the inverse function to compensate the cached results. If the inverse aggregation function is not available, then a full reaggregation is needed using the newly updated data. The information for processing MAOs can be speci ed in an Execution Plan (EP). By providing the derivation relationships, cost estimating functions, and synchronization plans in EP, a system can eÆciently reuse and maintain intermediate data. Experiment results show that the application of a caching method using MAO can yield close to an order of a magnitude of improvement in computations as compared with the method that does not use the MAO model. By tracing derivation relationships among the MAOs, the system provides related aggregations at all levels and can therefore be systematicly maintained. Therefore, our proposed methodology provides a more versatile, eÆcient, and coherent environment for complex aggregation tasks. References

[Albr 99] J. Albrecht, H. Gunzel, W. Lehner, \Set-Derivability of Multidimensional Aggregates", DaWak 1999, pp 133-142. [SAgr 96] S. Agrawal, R. Agrawal, P.M. Deshpande, A. Gupta, J.F. Naughton, R. Ramakrishnan, and S. Sarawagi. \On the computation of multidimensional aggregates ", Proc. 1996 Int. Conf. VLDB'96, pp 506-521, Bombay, India, Sept. 1996. [RAgr 97] R. Agrawal, A. Gupta, and S. Sarawagi, \Modeling multidimensional databases" , Proc. 1997 Int. Conf. Data Engineering(ICDE'97), pp 232243, Birmingham, England, Apr. 1997. [Mumi 97] Inderpal Singh Mumick, Dallan Quass, Barinderpal Singh Mumick, \Maintenance of Data Cubes and Summary Tables in a Warehouse ", ACM SIGMOD pp.100{111, '97 AZ, USA. [JimG 96] Jim Gray, Adam Bosworth, Andrew Layman, Hamid Pirahesh, \Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals" , IEEE Data Engineering pp.152-159. [Amit 96] Amit Shukla, Prasad M. Deshpande, Jerey F. Naughton, Karthikeyan Ramasamy, \Storage Estimation for Multidimensional Aggregates in the Presence of Hierarchies", VLDB 96 Mumbai(Bombay), India, 1996. [Venk 96] Venky Harinarayan, Anand Rajaraman, Jerey D. Ullman, \Implementing Data Cubes EÆciently", SIGMOD Conference 1996 pp.205-216. [Jose 97] Joseph M. Hellerstein, Peter J. Haas, Helen J. Wang, \Online Aggregation", SIGMOD Conference 1997 pp.171-182. [Han 01] Jiawei Han, Micheline Kamber, \Data Mining Concepts and Techniques", Morgan Kaufmann Publishers, pp.230-243.

The GMD Data Model for Multidimensional Information: A Brief Introduction Enrico Franconi and Anand Kamble Faculty of Computer Science Free Univ. of Bozen-Bolzano, Italy [email protected] [email protected]

Abstract. In this paper we introduce a novel data model for multidimensional information, GMD, generalising the MD data model first proposed in Cabibbo et al (EDBT-98). The aim of this work is not to propose yet another multidimensional data model, but to find the general precise formalism encompassing all the proposals for a logical data model in the data warehouse field. Our proposal is compatible with all these proposals, making therefore possible a formal comparison of the differences of the models in the literature, and to study formal properties or extensions of such data models. Starting with a logic-based definition of the semantics of the GMD data model and of the basic algebraic operations over it, we show how the most important approaches in DW modelling can be captured by it. The star and the snowflake schemas, Gray’s cube, Agrawal’s and Vassiliadis’ models, MD and other multidimensional conceptual data can be captured uniformly by GMD. In this way it is possible to formally understand the real differences in expressivity of the various models, their limits, and their potentials.

1

Introduction

In this short paper we introduce a novel data model for multidimensional information, GMD, generalising the MD data model first proposed in [2]. The aim of this work is not to propose yet another data model, but to find the most general formalism encompassing all the proposals for a logical data model in the data warehouse field, as for example summarised in [10]. Our proposal is compatible with all these proposals, making therefore possible a formal comparison of the different expressivities of the models in the literature. We believe that the GMD data model is already very useful since it provides a very precise and, we believe, very elegant and uniform way to model multidimensional information. It turns out that most of the proposals in the literature make many hidden assumptions which may harm the understanding of the advantages or disadvantages of the proposal itself. An embedding in our model would make all these assumptions explicit. Y. Kambayashi, M. Mohania, W. W¨ oß (Eds.): DaWaK 2003, LNCS 2737, pp. 55–65, 2003. c Springer-Verlag Berlin Heidelberg 2003


56

Enrico Franconi and Anand Kamble

So far, we have considered, together with the classical basic star and snowflake ER-based models and multidimensional cubes, the logical data models introduced in [2, 5, 1, 6, 9, 11, 3, 7, 8]. A complete account of both the GMD data model (including and extended algebra) and of the various encodings can be found in [4]; in this paper we just give a brief introduction to the basic principles of the data model. GMD is completely defined using a logic-based approach. We start introducing a data warehouse schema, which is nothing else than a set of fact definitions which restricts (i.e., constrains) the set of legal data warehouse states associated to the schema. By systematically defining how the various operators used in a fact definition constrain the legal data warehouse states, we give a formal logic-based account of the GMD data model.

2

The Syntax of the GMD Data Model

We introduce in this Section the notion of data warehouse schema. A data warehouse schema basically introduces the structures of the cubes that will populate the warehouse, together with the types allowed for the components of the structures. The definition of a GMD schema that follows is explained step by step. Definition 1 (GMD schema). Consider the signature < F , D, L, M, V, A >, where F is a finite set of fact names, D is a finite set of dimension names, L is a finite set of level names – each one associated to a finite set of level element names, M is a finite set of measure names, V is a finite set of domain names – each one associated to a finite set of values, A is a finite set of level attributes. ➽ We have just defined the alphabet of a data warehouse: we may have fact names (like SALES, PURCHASES), dimension names (like Date, Product), level name (like year, month, product-brand, product-category) and their level elements (like 2003, 2004, heineken, drink), measure names (like Price, UnitSales), domain names (like integers, strings), and level attributes (like is-leap, country-of-origin). A GMD schema includes: – a finite set of fact definitions of the form . F = E {D1 |L1 , . . . , Dn |Ln } : {M1 |V1 , . . . , Mm |Vm }, where E,F ∈ F, Di ∈ D, Li ∈ L, Mj ∈ M, Vj ∈ V. We call the fact name F a defined fact, and we say that F is based on E. A fact name not appearing at the left hand side of a definition is called an undefined fact. We will generally call fact either a defined fact or an undefined fact. A fact based on an undefined fact is called basic fact. A fact based on a defined fact is called aggregated fact. A fact is dimensionless if n = 0; it is measureless if m = 0. The orderings in a defined fact among dimensions and among measures are irrelevant.

The GMD Data Model for Multidimensional Information

➽ We have here introduced the building block of a GMD schema: the fact definition. A basic fact corresponds to the base data of any data warehouse: it is the cube structure that contains all the data on which any other cube will be built upon. In the following example, BASIC-SALES is a basic fact, including base data about sale transactions, organised by date, product, and store (which are the dimensions of the fact) which are respectively restricted to the levels day, product, and store, and with unit sales and sale price as measures: . BASIC-SALES = SALES {Date|day , Product|product, Store|store } : {UnitSales|int , SalePrice|int} – a partial order (L, ≤) on the levels in L. We call the immediate predecessor relation on L induced by ≤. ➽ The partial order defines the taxonomy of levels. For example, day

month quarter and day week; product type category – a finite set of roll-up partial functions between level elements → Lj ρLi ,Lj : Li  for each Li , Lj such that Li Lj . We call ρ∗Li ,Lj the reflexive transitive closure of the roll-up functions inductively defined as follows: ρ∗Li ,Li = id
ρ∗Li ,Lj = k ρLi ,Lk ◦ ρ∗Lk ,Lj where (ρLp ,Lq ∪ ρLr ,Ls )(x) = y

for each k such that Li Lk   ρLp ,Lq (x) = ρLr ,Ls (x) = y, or ρL ,L (x) = y and ρLr ,Ls (x) = ⊥, or iff  p q ρLp ,Lq (x) = ⊥ and ρLr ,Ls (x) = y

➽ When in a schema various levels are introduced for a dimension, it is also necessary to introduce a roll-up function for them. A rollup function defines how elements of one level map to elements of a superior level. Since we just require for the roll-up function to be a partial order, it is possible to have elements of a level which rollup to an upper level, while other elements may skip that upper level to be mapped to a superior one. For example, ρday,month(1/1/01) = Jan-01, ρday,month (2/1/01) = Jan-01, . . . ρquarter,year(Qtr1-01) = 2001, ρquarter,year(Qtr2-01) = 2001, . . .

57

58

Enrico Franconi and Anand Kamble

– a finite set of level attribute definitions: . L = {A1 |V1 , . . . , An |Vn } where L ∈ L, Ai ∈ A and Vi ∈ V for each i, 1 ≤ i ≤ n. ➽ Level attributes are properties associated to levels. For example, . product = {prodname|string , prodnum|int , prodsize|int , prodweight|int } – a finite set of measure definitions of the form . N = f(M ) where N, M ∈ M, and f is an aggregation function f : B(V)  → W, for some V, W ∈ V. B(V) is the finite set of all bags obtainable from values in V whose cardinality is bound by some finite integer Ω. ➽ Measure definitions are used to compute values of measures in an aggregated fact from values of the fact it is based on. For example: . . Total-UnitSales = sum(UnitSales) and Avg-SalePrice = average(SalePrice) Levels and facts are subject to additional syntactical well-foundedness conditions: – The connected components of (L, ≤) must have a unique least element each, which is called basic level. ➽ The basic level contains the finest grained level elements, on top of which all the facts are identified. For example, store city country; store is a basic level. – For each undefined fact there can be at most one basic fact based on it. ➽ This allows us to disregard undefined facts, which are in one-toone correspondence with basic facts. – Each aggregated fact must be congruent with the defined fact it is based on, i.e., for each aggregated fact G and for the defined fact F it is based on such that . F = E {D1 |L1 , . . . , Dn |Ln } : {M1 |V1 , . . . , Mm |Vm } . G = F {D1 |R1 , . . . , Dp |Rp } : {N1 |W1 , . . . , Nq |Wq } the following must hold (for some reordering on the dimensions): • the dimensions in the aggregated fact G are among the dimensions of the fact F it is based on: p≤n • the level of a dimension in the aggregated fact G is above the level of the corresponding dimension in the fact F it is based on: Li ≤ R i

for each i ≤ p

The GMD Data Model for Multidimensional Information

59

• each measure in the aggregated fact G is computed via an aggregation function from some measure of the defined fact F it is based on: . N1 = f1 (Mj(1) )

...

. Nq = fq (Mj(q) )

Moreover the range and the domain of the aggregation function should be in agreement with the domains specified respectively in the aggregated fact G and in the fact F it is based on. ➽ Here we give a more precise characterisation of an aggregated fact: its dimensions should be among the dimensions of the fact it is based on, its levels should be generalised from the corresponding ones in the fact it is based on, and its measures should be all computed from the fact it is based on. For example, given the basic fact BASIC-SALES: . BASIC-SALES = SALES {Date|day , Product|product, Store|store } : {UnitSales|int , SalePrice|int} the following SALES-BY-MONTH-AND-TYPE is an aggregated fact computed from the BASIC-SALES fact: . SALES-BY-MONTH-AND-TYPE = BASIC-SALES {Date|month , Product|type } : {Total-UnitSales|int, Avg-SalePrice|real} with the following aggregated measures: . Total-UnitSales = sum(UnitSales) . Avg-SalePrice = average(SalePrice) 2.1

Example

The following GMD schema summarises the examples shown in the previous Section: – Signature: • F = {SALES, BASIC-SALES, SALES-BY-MONTH-AND-TYPE, PURCHASES} • M = {UnitSales, Price, Total-UnitSales, Avg-Price} • D = {Date, Product, Store} • L = {day, week, month, quarter, year, product, type, category, brand, store, city, country } day = {1/1/01, 2/1/01, . . . , 1/1/02, 2/1/02, . . . } month = {Jan-01, Feb-01, . . . , Jan-02, Feb-02, . . . } quarter = {Qtr1-01, Qtr2-01, . . . , Qtr1-02, Qtr2-02, . . . } year = {2001, 2002} ···

60

–

–

–

–

–

3

Enrico Franconi and Anand Kamble • V = {int, real, string} • A = {dayname, prodname, prodsize, prodweight, storenumb} Partial order over levels: • day month quarter year, day week; day is a basic level • product type category, product brand; product is a basic level • store city country; store is a basic level Roll-up functions: ρday,month (1/1/01) = Jan-01, ρday,month (2/1/01) = Jan-01, . . . ρmonth,quarter (Jan-01) = Qtr1-01, ρmonth,quarter (Feb-01) = Qtr1-01, . . . ρquarter,year (Qtr1-01) = 2001, ρquarter,year (Qtr2-01) = 2001, . . . ρ∗day,year (1/1/01) = 2001, ρ∗day,year (2/1/01) = 2001, . . . ··· Level Attributes: . day = {dayname|string , daynum|int } . product = {prodname|string , prodnum|int , prodsize|int , prodweight|int } . store = {storename|string , storenum|int , address|string } Facts: . BASIC-SALES = SALES {Date|day , Product|product , Store|store } : {UnitSales|int , SalePrice|int } . SALES-BY-MONTH-AND-TYPE = BASIC-SALES {Date|month , Product|type } : {Total-UnitSales|int , Avg-SalePrice|real } Measures: . Total-UnitSales = sum(UnitSales) . Avg-SalePrice = average(SalePrice)

GMD Semantics

Having just defined the syntax of GMD schemas, we introduce now their semantics through a well founded model theory. We define the notion of a data warehouse state, namely a specific data warehouse, and we formalise when a data warehouse state is actually in agreement with the constraints imposed by a GMD schema. Definition 2 (Data Warehouse State). A data warehouse state over a schema with the signature < F , D, L, M, V, A > is a tuple I = < ∆, Λ, Γ, ·I >, where – ∆ is a non-empty finite set of individual facts (or cells) of cardinality smaller than Ω; ➽ Elements in ∆ are the object identifiers for the cells in a multidimensional cube; we call them individual facts. – Λ is a finite set of level elements; – Γ is a finite set of domain elements;

The GMD Data Model for Multidimensional Information

61

– ·I is a function (the interpretation function) such that FI ⊆ ∆

for each F ∈ F, where FI is disjoint from any other EI such that E ∈ F for each L ∈ L, where LI is disjoint from any other HI LI ⊆ Λ such that H ∈ L for each V ∈ V, where VI is disjoint from any other WI VI ⊆ Γ such that W ∈ V DI = ∆  →Λ for each D ∈ D MI = ∆  →Γ for each M ∈ M L I → Γ for each L ∈ L and AL (Ai ) = L  i ∈ A for some i (Note: in the paper we will omit the ·I interpretation function applied to some symbol whenever this is non ambiguous) ➽ The interpretation functions defines a specific data warehouse state given a GMD signature, regardless from any fact definition. It associates to a fact name a set of cells (individual facts), which are meant to form a cube. To each cell corresponds a level element for some dimension name: the sequence of these level elements is meant to be the “coordinate” of the cell. Moreover, to each cell corresponds a value for some measure name. Since fact definitions in the schema are not considered yet at this stage, the dimensions and the measures associated to cells are still arbitrary. In the following, we will introduce the notion of legal data warehouse state, which is the data warehouse state which conforms to the constraints imposed by the fact definitions. A data warehouse state will be called legal for a given GMD schema if it is a data warehouse state in the signature of the GMD schema and it satisfies the additional conditions found in the GMD schema. A data warehouse state is legal with respect to a GMD schema if: . – for each fact F = E {D1 |L1 , . . . , Dn |Ln } : {M1 |V1 , . . . , Mm |Vm } in the schema: • the function associated to a dimension which does not appear in a fact is undefined for its cells: ∀f. F(f ) → f ∈ dom(D) for each D ∈ D such that D = Di for each i ≤ n ➽ This condition states that the level elements associated to a cell of a fact should correspond only to the dimensions declared in the fact definition of the schema. That is, a cell has only the declared dimensions in any legal data warehouse state. • each cell of a fact has a unique set of dimension values at the appropriate level: ∀f. F(f ) → ∃l1 , . . . , ln . D1 (f ) = l1 ∧ L1 (l1 ) ∧ . . . ∧ Dn (f ) = ln ∧ Ln (ln ) ➽ This condition states that the level elements associated to a cell of a fact are unique for each dimension declared for the fact in the schema. So, a cell has a unique value for each declared dimension in any legal data warehouse state.

62

Enrico Franconi and Anand Kamble

• a set of dimension values identifies a unique cell within a fact: ∀f, f  , l1 , . . . , ln . F(f ) ∧ F(f  ) ∧ D1 (f ) = l1 ∧ D1 (f  ) = l1 ∧ . . . ∧ Dn (f ) = ln ∧ Dn (f  ) = ln → f = f ➽ This condition states that a sequence of level elements associated to a cell of a fact are associated only to that cell. Therefore, the sequence of dimension values can really be seen as an identifying coordinate for the cell. In other words, these conditions enforce the legal data warehouse state to really model a cube according the specification given in the schema. • the function associated to a measure which does not appear in a fact is undefined for its cells: ∀f. F(f ) → f ∈ dom(M) for each M ∈ M such that M = Mi for each i ≤ n ➽ This condition states that the measure values associated to a cell of a fact in a legal data warehouse state should correspond only to the measures explicitly declared in the fact definition of the schema. • each cell of a fact has a unique set of measures: ∀f. F(f ) → ∃m1 , . . . , mm . M1 (f ) = m1 ∧ V1 (m1 ) ∧ . . . ∧ Mm (f ) = mm ∧ Vm (mm ) ➽ This condition states that the measure values associated to a cell of a fact are unique for each measure explicitly declared for the fact in the schema. So, a cell has a unique measure value for each declared measure in any legal data warehouse state. – for each aggregated fact and for the defined fact it is based on in the schema: . F = E {D1 |L1 , . . . , Dn |Ln } : {M1 |V1 , . . . , Mm |Vm } . G = F {D1 |R1 , . . . , Dp |Rp } : {N1 |W1 , . . . , Nq |Wq } . . N1 = f1 (Mj(1) ) . . . Nq = fq (Mj(q) ) each aggregated measure function should actually compute the aggregation of the values in the corresponding measure of the fact the aggregation is based on: ∀g, v. Ni (g) = v ↔ ∃r1 , . . . , rp . G(g) ∧ D1 (g) = r1 ∧ . . . ∧ Dp (g) = rp ∧ v = fi ({|Mj(i) (f ) | ∃l1 , . . . , lp . F(f )∧ D1 (f ) = l1 ∧ . . . ∧ Dp (f ) = lp ∧ ρ∗L1 ,R1 (l1 ) = r1 ∧ . . . ∧ ρ∗Lp ,Rp (lp ) = rp |}) for each i ≤ q, where {| · |} denotes a bag. ➽ This condition guarantees that if a fact is the aggregation of another fact, then in a legal data warehouse state the measures associated to the cells of the aggregated cube should be actually computed by applying the aggregation function to the measures of the corresponding cells in the original cube. The correspondence between a cell in the aggregated cube and a set of cells in the original cube is found by looking how their coordinates – which are level elements – are mapped through the roll-up function dimension by dimension.

The GMD Data Model for Multidimensional Information

63

According to the definition, a legal data warehouse state for a GMD schema is a bunch of multidimensional cubes, whose cells carry measure values. Each cube conforms to the fact definition given in the GMD schema, i.e., the coordinates are in agreement with the dimensions and the levels specified, and the measures are of the correct type. If a cube is the aggregation of another cube, in a legal data warehouse state it is enforced that the measures of the aggregated cubes are correctly computed from the measures of the original cube. 3.1

Example

A possible legal data warehouse state for (part of) the previous example GMD schema is shown in the following. BASIC-SALESI = {s1 , s2 , s3 , s4 , s5 , s6 , s7 } SALES-BY-MONTH-AND-TYPEI = {g1 , g2 , g3 , g4 , g5 , g6 } Date(s1 ) Date(s2 ) Date(s3 ) Date(s4 ) Date(s5 ) Date(s6 ) Date(s7 )

= = = = = = =

1/1/01 7/1/01 7/1/01 10/2/01 28/2/01 2/3/01 12/3/01

UnitSales(s1 ) UnitSales(s2 ) UnitSales(s3 ) UnitSales(s4 ) UnitSales(s5 ) UnitSales(s6 ) UnitSales(s7 ) Date(g1 ) Date(g2 ) Date(g3 ) Date(g4 ) Date(g5 ) Date(g6 )

= = = = = =

= = = = = = =

100 500 230 300 210 150 100

daynum(day) = 1

= = = = = = =

Organic-milk-1l Organic-yogh-125g Organic-milk-1l Organic-milk-1l Organic-beer-6pack Organic-milk-1l Organic-beer-6pack

EuroSalePrice(s1) EuroSalePrice(s2) EuroSalePrice(s3) EuroSalePrice(s4) EuroSalePrice(s5) EuroSalePrice(s6) EuroSalePrice(s7)

Jan-01 Feb-01 Jan-01 Feb-01 Mar-01 Mar-01

Total-UnitSales(g1 ) Total-UnitSales(g2 ) Total-UnitSales(g3 ) Total-UnitSales(g4 ) Total-UnitSales(g5 ) Total-UnitSales(g6 )

4

Product(s1 ) Product(s2 ) Product(s3 ) Product(s4 ) Product(s5 ) Product(s6 ) Product(s7 )

Product(g1 ) Product(g2 ) Product(g3 ) Product(g4 ) Product(g5 ) Product(g6 ) = = = = = =

830 300 0 210 150 100

= = = = = =

= = = = = = =

Store(s1 ) Store(s2 ) Store(s3 ) Store(s4 ) Store(s5 ) Store(s6 ) Store(s7 )

= = = = = = =

Fair-trade-central Fair-trade-central Ali-grocery Barbacan-store Fair-trade-central Fair-trade-central Ali-grocery

71,00 250,00 138,00 210,00 420,00 105,00 200,00

Dairy Dairy Drink Drink Dairy Drink Avg-EuroSalePrice(g1) Avg-EuroSalePrice(g2) Avg-EuroSalePrice(g3) Avg-EuroSalePrice(g4) Avg-EuroSalePrice(g5) Avg-EuroSalePrice(g6)

prodweight(product) = 100gm

= = = = = =

153,00 210,00 0,00 420,00 105,00 200,00 storenum(store) = S101

GMD Extensions

For lack of space, in this brief report it is impossible to introduce the full GMD framework [4], which includes a full algebra in addition to the basic aggregation operation introduced in this paper. We will just mention the main extensions with respect to what has been presented here, and the main results. The full GMD schema language includes also the possibility to define aggregated measures with respect to the application of a function to a set of original

64

Enrico Franconi and Anand Kamble

measures, pretty much like in SQL. For example, it is possible to have an aggregated cube with a measure total-profit being the sum of the differences between the cost and the price in the original cube; the difference is applied cell by cell in the original cube (generating a profit virtual measure), and then the aggregation computes the sum of all the profits. Two selection operators are also in the full GMD language. The slice operation simply selects the cells of a cube corresponding to a specific value for a dimension, resulting in a cube which contains a subset of the cells of the original one and one less dimension. The multislice allows for the selection of ranges of values for a dimension, so that the resulting cube will contain a subset of the cells of the original one but retains the selected dimension. A fact-join operation is defined only between cubes sharing the same dimensions and the same levels. We argue that a more general join operation is meaningless in a cube algebra, since it may leads to cubes whose measures are no more understandable. For similar reasons we do not allow a general union operator (like the one proposed in [6]). As we were mentioning in the introduction, one main result is in the full encoding of many data warehouse logical data models as GMD schemas. We are able in this way to give an homogeneous semantics (in terms of legal data warehouse states) to the logical model and the algebras proposed in all these different approaches, we are able to clarify ambiguous parts, and we argue about the utility of some of the operators presented in the literature. The other main result is in the proposal of a novel conceptual data model for multidimensional information, that extends and clarifies the one presented in [3].

References [1] R. Agrawal, A. Gupta, and S. Sarawagi. Modeling multidimensional databases. In Proc. of ICDE-97, 1997. 56 [2] Luca Cabibbo and Riccardo Torlone. A logical approach to multidimensional databases. In Proc. of EDBT-98, 1998. 55, 56 [3] E. Franconi and U. Sattler. A data warehouse conceptual data model for multidimensional aggregation. In Proc. of the Workshop on Design and Management of Data Warehouses (DMDW-99), 1999. 56, 64 [4] Enrico Franconi and Anand S. Kamble. The GMD data model for multidimensional information. Technical report, Free University of Bozen-Bolzano, Italy, 2003. Forthcoming. 56, 63 [5] M. Golfarelli, D. Maio, and S. Rizzi. The dimensional fact model: a conceptual model for data warehouses. IJCIS, 7(2-3):215–247, 1998. 56 [6] J. Gray, A. Bosworth, A. Layman, and H. Pirahesh. Data cube: a relational aggregation operator generalizing group-by, cross-tabs and subtotals. In Proc. of ICDE-96, 1996. 56, 64 [7] M. Gyssens and L. V. S. Lakshmanan. A foundation for multi-dimensional databases. In Proc. of VLDB-97, pages 106–115, 1997. 56 [8] A. Tsois, N. Karayiannidis, and T. Sellis. MAC: Conceptual data modelling for OLAP. In Proc. of the International Workshop on Design and Management of Warehouses (DMDW-2001), pages 5–1–5–13, 2001. 56

The GMD Data Model for Multidimensional Information

65

[9] P. Vassiliadis. Modeling multidimensional databases, cubes and cube operations. In Proc. of the 10th SSDBM Conference, Capri, Italy, July 1998. 56 [10] P. Vassiliadis and T. Sellis. A survey of logical models for OLAP databases. In SIGMOD Record, volume 28, pages 64–69, December 1999. 55 [11] P. Vassiliadis and S. Skiadopoulos. Modelling and optimisation issues for multidimensional databases. In Proc. of CAiSE-2000, pages 482–497, 2000. 56

An Application of Case-Based Reasoning in Multidimensional Database Architecture* Dragan Simiü1, Vladimir Kurbalija2, Zoran Budimac2 1

2

Novi Sad Fair, Hajduk Veljkova 11, 21000 Novi Sad, Yugoslavia [email protected]

Department of Mathematics and Informatics, Fac. of Science, Univ. of Novi Sad Trg D. Obradoviüa 4, 21000 Novi Sad, Yugoslavia [email protected], [email protected]

ABSTRACT. A concept of decision support system is considered in this paper. It provides data needed for fast, precise and good business decision making to all levels of management. The aim of the project is the development of a new online analytical processing oriented on case-based reasoning (CBR) where a previous experience for every new problem is taken into account. Methodological aspects have been tested in practice as a part of the management information system development project of "Novi Sad Fair". A case study of an application of CBR in prediction of future payments is discussed in the paper.

1

Introduction

In recent years, there has been an explosive growth in the use of database for decision support systems. This phenomenon is a result of the increased availability of new technologies to support efficient storage and retrieval of large volumes of data: data warehouse and online analytical processing (OLAP) products. A data warehouse can be defined as an online repository of historical enterprise data that is used to support decision-making. OLAP refers to technologies that allow users to efficiently retrieve data from the data warehouse. In order to help an analyst focus on important data and make better decisions, case-based reasoning (CBR - an artificial intelligence technology) is introduced for making predictions based on previous cases. CBR will automatically generate an answer to the problem using stored experience, thus freeing the human expert of obligations to analyse numerical or graphical data. The use of CBR in predicting the rhythm of issuing invoices and receiving actual payments based on the experience stored in the data warehouse is presented in this paper. Predictions obtained in this manner are important for future planning of a company such as the ”Novi Sad Fair” because achievement of sales plans, revenue and *

Research was partially supported by the Ministry of Science, Technologies and Development of Republic of Serbia, project no. 1844: ”Development of (intelligent) techniques based on software agents for application in information retrieval and workflow”

Y. Kambayashi, M. Mohania, W. W¨oß (Eds.): DaWaK 2003, LNCS 2737, pp. 66-75, 2003. c Springer-Verlag Berlin Heidelberg 2003

An Application of Case-Based Reasoning in Multidimensional Database Architecture

67

company liquidation are measures of success in business. Performed simulations show that predictions made by CBR differ only for 8% in respect to what actually happened. With inclusion of more historical data in the warehouse, the system gets better in predictions. Furthermore, the system uses not only a data warehouse but also previous cases and previous predictions in future predictions thus learning during the operating process. The combination of CBR and data warehousing, i.e. making an OLAP intelligent by the use of CBR is a rarely used approach, if used at all. The system also uses a novel CBR technique to compare graphical representation of data which greatly simplifies the explanation of the prediction process to the end-user [3]. The rest of the paper is organized as follows. The following section elaborates more on motivations and reasons for inclusion of CBR in decision support system. This section also introduces our case-study on which we shall describe the usage of our system. Section three overviews the case based reasoning technique, while section four describes the original algorithm for searching the previous cases (curves) looking for the most similar one. Fifth section describes the actual application of our technique to the given problem. Section six presents the related work, while the seventh section concludes the paper.

2

User requirements for decision support system

“Novi Sad Fair” represents a complex organization considering the fact that it is engaged in a multitude of activities. The basic Fair activity is organizing fair exhibitions, although it has particular activities throughout the year. Ten times a year, 27 fair exhibitions are organized where nearly 4000 exhibitors take part, both from the country and abroad. Besides designing a ‘classical’ decision support system based on a data warehouse and OLAP, requirements of the company management clearly showed that it will not be enough for good decision making. The decision to include artificial intelligence methods in general and CBR in particular into the whole system was driven by the results of the survey. The survey was made on the sample of 42 individuals (users of the current management information system) divided into three groups: strategictactical management (9 people), operational managers (15 people), and transactional users (18 people). After a statistical evaluation of the survey [5], the following conclusions (among others) were drown: ƒ Development of the decision support systems should be focussed on problems closely related to financial estimates and financial marker trends tracking which span several years. ƒ The key influences on business (management) are political and economic environment of the country and region, which induces the necessity of exact implementation of those influences in the observed model (problem). Also it is necessary to take them into account in future events estimations.

68

Dragan Simi´c et al.

ƒ

The behavior of the observed case does not depend on its pre-history but only on initial state, respectively.

Implementation of this non-exact mathematical model is a very complex problem. As an example, let us take a look into the problem pointed to us by company managers. During any fair exhibition the total of actual income is only 30% to 50% of the total invoice value. Therefore, managers want to know how high the payment of some fair services would be in some future time, with respect to invoicing. If they could predict reliably enough what would happen in the future, they could make important business activities to ensure faster arrival of invoiced payments and plan future activities and exhibitions better. The classical methods can not explain influences on business and management well enough. There are political and economic environments of the country and region that cannot be successfully explained and used with classical methods: war in Iraq, oil deficiency, political assassinations, terrorism, spiral growth in mobile telecommunication industry, general human occupation and motivation. And this is even more true in an enterprise such as Fair whose success depends on many external factors. One possible approach in dealing with external influences is observing the case histories of similar problems (cases) for a longer period of time, and making estimations according to that observation. This approach, generally speaking, represents intelligent search which is applied to solving new problems by adapting solutions that worked for similar problems in the past - case-based reasoning.

3

Case based reasoning

Case-Based Reasoning is a relatively new and promising area of artificial intelligence and it is also considered a problem solving technology (or technique). This technology is used for solving problems in domains where experience plays an important role [2]. Generally speaking, case-based reasoning is applied to solving new problems by adapting solutions that worked for similar problems in the past. The main supposition here is that similar problems have similar solutions. The basic scenario for mainly all CBR applications looks as follows. In order to find a solution of an actual problem, one looks for a similar problem in an experience base, takes the solution from the past and uses it as a starting point to find a solution to the actual problem. In CBR systems experience is stored in a form of cases. The case is a recorded situation where problem was totally or partially solved, and it can be represented as an ordered pair (problem, solution). The whole experience is stored in case base, which is a set of cases and each case represents some previous episode where the problem was successfully solved. The main problem in CBR is to find a good similarity measure – the measure that can tell to what extent the two problems are similar. In the functional way similarity can be defined as a function sim : U u CB o [0 , 1] where U refers to the universe of all objects (from a given domain), while CB refers to the case base (just those objects

An Application of Case-Based Reasoning in Multidimensional Database Architecture

69

which were examined in the past and saved in the case memory). The higher value of the similarity function means that these objects are more similar [1]. The case based reasoning system has not the only goal of providing solutions to problems but also of taking care of other tasks occurring when used in practice. The main phases of the case-based reasoning activities are described in the CBR-cycle (fig. 1) [1].

Fig. 1. The CBR-Cycle after Aamodt and Plaza (1994)

In the retrieve phase the most similar case (or k most similar cases) to the problem case is retrieved from the case memory, while in the reuse phase some modifications to the retrieved case are done in order to provide better solution to the problem (case adaptation). As the case-based reasoning only suggests solutions, there may be a need for a correctness proof or an external validation. That is the task of the phase revise. In the retain phase the knowledge, learned from this problem, is integrated in the system by modifying some knowledge containers. The main advantage of this technology is that it can be applied to almost any domain. CBR system does not try to find rules between parameters of the problem; it just tries to find similar problems (from the past) and to use solutions of the similar problems as a solution of an actual problem. So, this approach is extremely suitable for less examined domains – for domains where rules and connections between parameters are not known. The second very important advantage is that CBR approach to learning and problem solving is very similar to human cognitive processes – people take into account and use past experiences to make future decisions.

70

4

Dragan Simi´c et al.

CBR for predicting curves behaviour

The CBR system for its graphics in presenting both the problem and the cases is used [3]. The reasons are that in many practical domains some decisions depend on behaviour of time diagrams, charts and curves. The system therefore analyses curves, compares them to similar curves from the past and predicts the future behaviour of the current curve on the basis of the most similar curves from the past. The main problem here, as almost in every CBR system, was to create a good similarity measure for curves, i.e. what is the function that can tell to what extent the two curves are similar. In many practical domains data are represented with the set of points, where the point is an ordered pair (x,y). Very often the pairs are (t,v) where t represents time and v represents some value in the time t. When the data is given in this way (as a set of points) then it can be graphically represented. When the points are connected, then they represent some kind of a curve. If the points are connected only with straight lines then it represents the linear interpolation, but if someone wants smoother curves then some other kind of interpolation with polynomials must be used. There was a choice between a classical interpolating polynomial and a cubic spline. The cubic spline was chosen for two main reasons: x x

Power: for the n+1 points classical interpolating polynomial has the power n, while cubic spline always has the power 4. Oscillation: if only one point is moved (which can be the result of bad experiment or measuring) classical interpolating polynomial significantly changes (oscillates), while cubic spline only changes locally (which is more appropriate for real world domains).

Fig. 2. Surface between two curves

When the cubic spline is calculated for curves then one very intuitive and simple similarity (or distance – which is the dual notion for similarity1) measure can be used. 1

When the dictance d i known then the similarity sim can be easily computed using for example function: sim = 1/(1+d)

An Application of Case-Based Reasoning in Multidimensional Database Architecture

71

The distance between two curves can be represented as a surface between these curves as seen on the fig 2. This surface can be easily calculated using the definitive integral. Furthermore, the calculation of the definitive integral for polynomials is a very simple and efficient operation.

5

Application of the system

A data warehouse of “Novi Sad Fair” contains data about payment and invoicing processes from the last 3 years for every exhibition - containing between 25 and 30 exhibitions every year. Processes are presented as sets of points where every point is given with the time of the measuring (day from the beginning of the process) and the value of payment or invoicing on that day. It can be concluded that these processes can be represented as curves. Note that the case-base consists of cases of all exhibitions and that such a case-base is used in solving concrete problems for concrete exhibitions. The reason for this is that environmental and external factors influence business processes of the fair to a high extent. The measurement of the payment and invoicing values was done every 4 days from the beginning of the invoice process in duration of 400 days, therefore every curve consists of approximately 100 points. By analysing these curves, the process of invoicing usually starts several months before the exhibition and that value of invoicing rapidly grows approximately to the time of the beginning of exhibition. After that time the value of invoicing remains approximately the same till the end of the process. That moment, when the value of invoicing reaches some constant value and stays the same to the end, is called the time of saturation for the invoicing process, and the corresponding value – the value of saturation. The process of payment starts several days after the corresponding process of invoicing (process of payment and invoicing for the same exhibition). After that the value of payment grows, but not so rapidly as the value of invoicing. At the moment of exhibition the value of payment is between 30% and 50% of the value of invoicing. After that, the value of payment continues to grow to some moment when it reaches a constant value and stays approximately constant till the end of the process. That moment is called the time of saturation for the payment process, and the corresponding value – the value of saturation. Payment time of saturation is usually couple of months after the invoice time of the saturation, and the payment value of saturation is always less than the invoice value of saturation or equal. The analysis shows that payment value of saturation is between 80% and 100% of the invoice value of saturation. The maximum represents a total of services invoiced and that amount is to be paid. The same stands for the invoicing curve where the maximum amount of payment represents the amount of payment by regular means. The rest will be paid later by court order, other special business agreements or, perhaps, will not be paid at all (debtor bankruptcy).

72

Dragan Simi´c et al.

Fig. 3. The curves from the data mart, as the "Old payment curve" and the "Old invoice curve"

One characteristic invoice and a corresponding payment curve as the "Old payment curve" and "Old invoice curve" from the ”curve base” are shown (fig. 3). The points of saturation (time and value) are represented with the emphasised points on curves. At the beginning system reads the input data from two data marts: one data mart contains the information about all invoice processes for every exhibition in the past 3 years, while the other data mart contains the information about the corresponding payment processes. After that, the system creates splines for every curve (invoice and payment) and internally stores the curves in the list of pairs containing the invoice curve and the corresponding payment curve. In the same way system reads the problem curves from the third data mart. The problem is invoice and a corresponding problem curve at the moment of the exhibition. At that moment, the invoice curve reaches its saturation point, while the payment curve is still far away from its saturation point. These curves are shown as the "Actual payment curve" and the "Actual invoice curve" (fig. 4). The solution of this problem would be the saturation point for the payment curve. This means that system helps experts by suggesting and predicting the level of future payments. At the end of the total invoicing for selected fair exposition, operational exposition manager can get a prediction from CBR system of a) the time period when payment of a debt will be made and b) the amount paid regularly.

An Application of Case-Based Reasoning in Multidimensional Database Architecture

73

Fig. 4. Problem payment and invoice curves, as the "Actual payment curve" and the "Actual invoice curve" and prediction for the future payments Time point and the amount of payment of a debt are marked on the graphic by a big red dot (fig. 4). When used with the subsets of already known values, CBR predicts the results that differed around 10% in time and 2% in value from actually happened. 5.1 Calculation of saturation points and system learning The saturation point for one prediction is calculated by using 10% of the most similar payment curves from the database of previous payment processes. The similarity is calculated by using the previosly described algorithm. Since the values of saturation are different for each exhibition, every curve from the database must be scaled with a particular factor so that the invoice values of saturation of the old curve and actual curve are the same. That factor is easily calculated as: actual _ value _ of _ saturation Factor old _ value _ of _ saturation where the actual value of saturation is in fact the value of the invoice in the time of the exhibition. The final solution is then calculated by using payment saturation points of the 10% most similar payment curves. Saturation points of the similar curves are multiplied with the appropriate type of goodness and then summed. The values of goodness are directly proportional to the similarity between old and actual curves, but the sum of all goodnesses must be 1. Since the system calculates the distance, the similarity is calculated as:

74

Dragan Simi´c et al.

1 1  dist The goodness for every old payment curve is calculated as: sim

goodness i

simi ¦ sim j

all _ j

At the end, the final solution – payment saturation point is calculated as: sat _ point ¦ goodness i ˜ sat _ point i all _ i

The system draws the solution point at the diagram combined with the saturation time and value. The system also supports solution revising and retaining (fig. 1). By memorizing a) the problem, b) suggested solution, c) the number of similar curves used for obtaining the suggestion and d) the real solution (obtained later), the system uses this information in the phase of reusing the solution for future problems. The system will then use not only 10% percent of the most similar curves but will also inspect the previous decisions in order to find ‘better’ number of similar curves that would lead to the better prediction.

6

Related work

The system presented in the paper represents a useful coexistence of a data warehouse and a case based reasoning resulting in a decision support system. The data warehouse (part of the described system) has been in operation in “Novi Sad Fair” since 2001 and is described in more details [5] [6] [7]. The part of the system that uses CBR in comparing curves has been done during the stay of the second author at Humboldt University in Berlin and is described in [3] in more detail. Although CBR is successfully used in many areas (aircraft conflict resolution in air traffic control, optimizing rail transport, subway maintenance, optimal job search, support to help-desks, intelligent search on the internet) [4], it is not very often used in combination with data warehouse and in collaboration with classical OLAP, probably due to novelty of this technique. CBR does not require causal model or deep understanding of a domain and therefore it can be used in domains that are poorly defined, where information is incomplete, contradictory, or where it is difficult to get sufficient domain of knowledge. All this is typical for business processing. Besides CBR, other possibilities are rule base knowledge or knowledge discovery in database where knowledge evaluation is based on rules [1]. The rules are usually generated by combining propositions. As the complexity of the knowledge base increases, maintaining becomes problematical because changing rules often implies a lot of reorganization in a rule base system. On the other side, it is easier to add or delete a case in a CBR system, which finally provides the advantages in terms of learning and explicability. Applying CBR to curves and its usage in decision making is also a novel approach. According to the authors' findings, the usage of CBR, looking for similarities in curves and predicting future trends is by far superior to other currently used techniques.

An Application of Case-Based Reasoning in Multidimensional Database Architecture

7

75

Conclusion

The paper presented the decision support system that uses CBR as an OLAP to the data warehouse. The paper has in greater detail described the CBR part of the system giving a thorough explanation of one case study. There are numerous advantages of this system. For instance, based on CBR predictions, operational managers can make important business activities, so they would: a) make payment delays shorter, b) make the total of payment amount bigger, c) secure payment guarantee on time, d) reduce the risk of payment cancellation and e) inform senior managers on time. By combining graphical representation of predicted values with most similar curves from the past, the system enables better and more focussed understanding of predictions with respect to real data from the past. Senior managers can use these predictions to better plan possible investments and new exhibitions, based on the amount of funds and the time of their availability, as predicted by the CBR system. Presented system is not only limited to this case-study but it can be applied to other business values as well (expenses, investments, profit) and it guarantees the same level of success.

Acknowledgement The CBR system that uses graphical representation of problem and cases [3] was implemented by V. Kurbalija at Humboldt University, Berlin (AI Lab) under the leadership of Hans-Dieter Burkhard and sponsorship of DAAD (German academic exchange service). Authors of this paper are grateful to Prof. Burkhard and his team for their unselfish support without which none of this would be possible.

References 1. Aamodt, A., Plaza, E.: Case-Based Reasoning: Foundational Issues, Methodological Variations and System Approaches, AI Commutations, pp. 39-58. 1994. 2. Zoran Budimac, Vladimir Kurbalija: Case-based Reasoning – A Short Overview, Conference of Informatics and IT, Bitola, 2001. 3. Vladinmir Kurbalija: On Similarity of Curves – project report, Humboldt University, AI Lab, Berlin, 2003. 4. Mario Lenz, Brigitte Bartsh-Sporl, Hans-Dieter Burkhard, Stefan Wess, G. Goos, J. Van Leeuwen, B. Bartsh: Case-Based Reasoning Technology: From Foundations to Aplications, Springer Verlag, October 1998. 5. Dragan Simic: Financial Prediction and Decision Support System Based on Artificial Intelligence Technology, Ph.D. thesis, draft text – manuscript, Novi Sad 2003. 6. Dragan Simic: Reengineering management information systems, contemporary information technologies perspective, Master thesis, Novi Sad 2001. 7. Dragan Simic: Data Warehouse and Strategic Management, Strategic management and decision support systems, Palic, 1999.

MetaCube XTM: A Multidimensional Metadata Approach for Semantic Web Warehousing Systems Thanh Binh Nguyen1, A Min Tjoa1, and Oscar Mangisengi2 1

Institute of Software Technology (E188) Vienna University of Technology, Favoritenstr. 9-11/188, A-1040 Vienna, Austria {binh,tjoa}@ifs.tuwien.ac.at 2 Software Competence Center Hagenberg Hauptstrasse 99, A-4232 Hagenberg, Austria [email protected]

Abstract. Providing access and search among multiple, heterogeneous, distributed and autonomous data warehouses has become one of the main issues in the current research. In this paper, we propose to integrate data warehouse schema information by using metadata represented in XTM (XML Topic Maps) to bridge possible semantic heterogeneity. A detailed description of an architecture that enables the efficient processing of user queries involving data from heterogeneous is presented. As a result, the interoperability is accomplished by a schema integration approach based on XTM. Furthermore, important implementation aspects of the MetaCube-XTM prototype, which makes use of the Meta Data Interchange Specification (MDIS), and the Open Information Model, complete the presentation of our approach.

1

Introduction

The advent of the World Wide Web (WWW) in the mid –1990s has resulted in even greater demand for effectively managing data, information, and knowledge. Web sources consist of very large information resources that are distributed into different location, sites, and systems. According to [15], Web warehousing is a novel and very active research area, which combines two rapidly developing technologies, i.e. data warehousing and Web technology depicted in figure 1. However, the emerging challenge of Web warehousing is how to manage Web OLAP data warehouse sites in a unified way and to provide a unified access among different Web OLAP resources [15]. Therefore, a multidimensional metadata standard or framework is necessary to enable the data warehousing interoperability. As a result, we are addressing the following issues:

Y. Kambayashi, M. Mohania, W. Wßö (Eds.): DaWaK 2003, LNCS 2737, pp. 76-88, 2003.  Springer-Verlag Berlin Heidelberg 2003

MetaCube XTM

77

Data Warehousing contributes: Data management warehousing approach

Web Warehousing

The Web contributes: Web technology text and multimedia managament

Fig. 1. The hybrid of Web warehousing systems

Multidimensional Metadata Standard. In the database community there exists some research efforts for formal multidimensional data models and their corresponding query languages [1,4,6,9, 12,13,19]. However, each approach presents its own view of multidimensional analysis requirements, terminology and formalism. As a result, none of the models is capable of encompassing the others. Data Warehousing Interoperability. The relevance of interoperability for future data warehouse architectures is described in detail in [5]. Interoperability not only has to resolve the differences in data structures; it also has to deal in addition with semantic heterogeneity. In this context, the MetaCube concept is proposed in [19] as a multidimensional metadata framework for cross-domain data warehousing systems. In the further development, the MetaCube concept is extended to MetaCube-X by using XML [20], to support interoperability for web warehousing applications. MetaCube-X is an XML (extensible markup language) instance of MetaCube, and provides a n“ eutral” syntax for interoperability among different web warehousing systems. In the described framework, we define a global MetaCube-X stored in the server site and local MetaCube-X(s), each of which is stored in a local Web warehouse. The emerging issues to be handled in the global MetaCube-X are mainly issues concerning semantic heterogeneities of the local MetaCube-X, while the capability for accessing data at any level of complexity should still be provided by local Web data warehouse. In this paper we extend the concept of MetaCube-X using Topic Maps (TMs) [23] (MetaCube-XTM). Research is showing that topic maps can provide a sound basis for the Semantic Web. In addition, the Topic Maps also builds a bridge between the domains of knowledge representation and information management. The MetaCubeXTM system provides a unified view for users that address the semantic heterogeneities. On the other hand, it also supports data access of any level complexity on the local data warehouses using local MetaCube-XTMs.

78

Thanh Binh Nguyen et al.

Prototyping. Both the MetaCube-XTM concept and the web technologies are now sufficiently mature to move from proof of concept towards a semi-operational prototype – the MetaCube-XTM prototype. The remainder of this paper is organized as follows. Section 2 presents related works. In Section 3 we summarize the concepts of MetaCube [19] and introduce MetaCube-XTM protocol. Hereafter, we show the implementation of MetaCube-XTM prototype. The conclusion and future works appear in Section 5.

2

Related Works

The DARPA Agent Markup Language (DAML) [21] developed by DARPA aims at developing a language and tools to facilitate the concept of the Semantic Web [22]: the idea of having data on the Web defined and linked in a way that it can be used by machines not just for display purposes, but for automation, integration and reuse of data across various applications. The DAML language is being developed as an extension to XML and the Resource Description Framework (RDF). The latest extension step of this language (DAML + OIL - Ontology Inference Layer) provides a rich set of constructs to create ontologies and to markup information so that it is machine readable and understandable. The Ontology Inference Layer (OIL) [7,11] from the On-To-Knowledge Project is a proposal for such a standard way of expressing ontologies based on the use of web standards like XML schema and RDF schemas. OIL is the first ontology representation language that is properly grounded in W3C standard such as RDF/RDF-schema and XML/XML-schema. DAML and OIL are general concepts not specifically related to database or data warehouse interoperability. In the field of federated data warehouses, a variety of approaches to interoperability have been proposed. In [14] the authors describe the usage of XML to enable interoperability of data warehouses by an additional architectural layer used for exchanging schema metadata. Distributed DWH architectures based on CORBA [2], and centralized virtual data warehouses based on CORBA and XML [3] have been proposed recently. All of these approaches propose distributed data warehouse architectures based on a kind of restricted data and metadata interchange format using particular XML terms and RDF extensions, respectively. Basically, they achieve syntactical integration - but these concepts do not address semantic heterogeneity to enable a thorough description of mappings between federated, heterogeneous data warehouse systems. [14] presents distributed and parallel computing issues in data warehousing. [2] also presents the prototypical distributed OLAP system developed in the context of the CUBE-STAR project. In [20], the MetaCube-X is proposed as an XML instance of the MetaCube 's concept [19] for supporting data warehouses in the federated environment. It provides a framework for supporting integration and interoperability of data warehouses. Moreover, this paper, the MetaCube-XTM, a new MetaCube generation, is addressed to the semantic heterogeneity for data warehousing interoperability.

MetaCube XTM

3

79

The Concepts of MetaCube-XTM

In this section MetaCube-XTM is presented as a framework of DWH interoperability. Based on this concept, a protocol is studied and proposed as a generic framework to support data access of any complexity level on local data warehouses. 3.1

MetaCube Conceptual Data Model

In [19], a conceptual multidimensional data model that facilitates a precise rigorous conceptualization for OLAP has been introduced and presented. This approach is built on basic mathematic concepts, i.e. partial order, partially ordered set (poset) [10]. The mathematical foundation provides the basis for handling natural hierarchical relationships among data elements along (OLAP)dimensions with many levels of complexity in their structures. We summarize the MetaCube concepts introduced in [19] as follows: Dimension Concepts In [19] we introduced hierarchical relationships among dimension members by means of one hierarchical domain per dimension. A hierarchical domain is a poset (partially ordered set), denoted by < dom( D ), p D > , of dimension ele-

ments dom(D ) = {dm } ∪ {dm ,L, dm }, organized in hierarchy of levels, corresponding to different levels of granularity. An example of the hierarchy domain of the dimension Time with unbalanced and multiple hierarchical structure is showed in figure 2. Afterwards, it allows us to consider a dimension schema as a poset of levels, denoted by DSchema(D)= Levels(D) , p L . Figure 3 shows examples of dimension all

1

n

schemas of three dimensions Product, Geography and Time. Furthermore, a family of sets {dom(l0 ),.., dom(lh )} is a partition [10]of dom(D) . In this concept, a dimension hierarchy is a path along the dimension schema, beginning at the root level and ending at a leaf level [19]. all

1999

Q1.1999

Jan.1999

Feb.1999

Mar.1999 W1.1999

1.Jan.1999

6.Jan.1999

1.Feb.1999

3.Feb.1999

W5.1999

W9.1999

3.Mar.1999

Fig. 2. An example of the hierarchy domain of the dimension Time with unbalanced and multiple hierarchical structure

80

Thanh Binh Nguyen et al. All

All

All

Category

Country

Year Quarter

Type

State

Month Week

Item

City

Product

Geography

Day Time

Fig. 3. Examples of dimension schemas of three dimensions Product, Geography and Time

The Concept of Measures

[Measure Schema] A schema of a measure M is a tuple MSchema(M ) = Fname, O , where: • •

Fname is a name of a corresponding fact, O ∈ Ω ∪ {NONE, COMPOSITE} is an operation type applied to a specific fact [2]. Furthermore: - Ω={SUM, COUNT, MAX, MIN} is a set of aggregation functions. - COMPOSITE is an operation (e.g. average), where measures cannot be utilized in order to automatically derive higher aggregations. - NONE measures are not aggregated. In this case, the measure is the fact.

[Measure Domain] Let N be a numerical domain where a measure value is defined (e.g. N, Z, R or a union of these domains). The domain of a measure is a subset of N. We denote by dom(M ) ⊂

N

.

The Concept of MetaCube

First, a MetaCube schema is defined by a triple of a MetaCube name, an x tuple of dimension schemas, and a y tuple of measure schemas, denoted by CSchema(C)= Cname, DSchemas, MSchemas . Furthermore, the hierarchy domain of a MetaCube, denoted by dom(C) = Cells(C) , p C is a poset, where each data cell is an intersection among a set of dimension members and measure data values, each of which belongs to one dimension or one measure. Afterwards, data cells of within the MetaCube hierarchy domain are grouped into a set of associated granular groups, each of which expresses a mapping from the domains of x-tuple of dimension levels (independent variables) to y-numerical domains of y-tuple of numeric measures (dependent variables). Hereafter, a MetaCube is constructed based on a set of dimensions, and consists of a MetaCube schema, and is associated with a set of groups.

Product

St or e

MetaCube XTM

81

Mexico USA

Alcoholic

10

Dairy

50

Beverage

20

Baked Food

12

Meat

15

Seafood

10

1 2 3 4 5 6 Time Fig. 4. Sales MetaCube is constructed from three dimensions: Store, Product and Time and one fact: TotalSale

3.2

MetaCube XTM Protocol

The MetaCube-XTM protocol is proposed to handle the design, integration, and maintenance of heterogeneous schemas of the local data warehouses. It describes each local schema including its dimensions, dimension hierarchies, dimension levels, cubes, and measures. With the means of the The MetaCube-XTM protocol it should be possible to describe any schema represented by any multidimensional data models (i.e. star schema, snow-flake model, etc.). Furthermore, it is also aimed to provide abilities for interoperability searching and data integration among web-data warehouses as shown in Figure 5. The architecture of MetaCube-XTM systems consists of clients, server protocol (i.e., the global MetaCube-XTM at an information server, and several distributed local data warehouses and their local MetaCube-XTMs). The functionalities are given as follows: •

•

•

MetaCube–XTM Services. A set of MetaCube–XTM services at the information server is intended to provide searching and navigation abilities for clients and to manage the access to local DWHs from the federated information sever (figure 5). Global MetaCube-XTM. The global MetaCube-XTM is stored at the server, and is intended to provide a multidimensional metadata framework for a class of local data warehouses managed by MetaCube-XTM Services. Thus, it has to solve semantic heterogeneity and support the search facility to the local data warehouse. Local MetaCube-XTM. Each local MetaCube-XTM is used to describe the multidimensional data model for each local data warehouse based on the global MetaCube-XTM. The Local MetaCube-XTM is stored in the local data warehouse.

82

Thanh Binh Nguyen et al.

Client

Web Data Warehouse Queries

MetaCube-X Server MetaCube-XTM Services

XML

Global MetaCubeXTM

locatorDB

XML

XML

XML

Local MetaCubeXTM

Local MetaCubeXTM

Local MetaCubeXTM

Data Warehouse 1

Data Warehouse n

Fig. 5. MetaCube-XTM architecture

4

MetaCube-XTM Prototype

The entire idea behind prototyping is to cut down on the complexity of implementation by eliminating parts of the full system. In this context, the MetaCube-XTM prototype has been implemented. First UML (Unified Modelling Language) is used to model the MetaCube concept. UML modeling provides a framework to implement MetaCube in the XTM (XML Topic Maps). Hereafter, we describe the local MetaCube-XTM as a local presentation of DWH schemas using topic maps (bottomup approach). Then we describe the integration of heterogeneous schemas in subsection 4.2. We are going to use only predefined XTM tags as proposed by the XTM Standard (topicMap, topic, baseName, association, occurrence, topicRef, etc.). Therefore it will be possible to use tools based on the XTM standard to create, generate, and maintain such XTM descriptions easily. In this section we also present the process of MetaCube-XTM prototype. 4.1

Modeling MetaCube-XTM with UML

The common or MetaCube-XTM is a model that is used for expressing all schema objects available in the different local data warehouses. To model the MetaCube-

MetaCube XTM

83

XTM, UML is used to model dimensions, measures and data cubes in context of MetaCube data model (figure 6) [19,20]. The approach is implemented by a mapping into XML schema based on the following standard specifications: Meta Data Interchange Specification (MDIS) [16], and the Open Information Model (OIM) [17,18] of the Meta Data Coalition (MDC). 4.2

Implementation with XML Topic Maps

Topic Maps (TMs) provides a solution for organizing and navigating information resources into a unified view on the Web. In this paper we use XTM is to represent the MetaCube concept, to model data to any dimensional level of complexity, to check data for structural correctness, to define new tags corresponding to a new dimension, and to show hierarchical information corresponding to dimension hierarchies. These functionalities are necessary for data warehouse schema handling and OLAP application.

Has Child

+Ch ild

0..*

NestedElelement

+Fa th er

Des cri ption : String;

0..*

+Fa the r +Chi ld

Has Father

MDElement

belongs to

Cell MeasureValue

Gro upby Gnam e : String;

1 ..*

DimensionElement

Des cription : type;

1..* b elongs to

GSchema IntergerValue Des cription : int;

floatValue

r efers to

Level

Des cription : float;

Lnam e : String;

Gnam e : String;

1..*

1..*

1.. *

refers to

1..*

1 .. *

refers to

MeasureSchema

Hierarchy Hnam e : Stri ng;

belongs to

Fnam e : Str ing; AggFunc ti on : Str ing;

refers to

1..* 1 ..*

refers to

DimensionSchema Dname : String;

belongs to

belo ngs to

refers to

Dimension 1 ..*

Fig. 6. The MetaCube-XTM model with UML

Cube Cnam e : String; Bas icGroupby : Groupby;

84

Thanh Binh Nguyen et al. /* local Topic Maps */ /* define topic Cube */ Meta Cube-1 /* define instance of Cube */ Instance of Cube-1

……….. Meta Dimension Meta Cube-Dimension Meta Dimension-Level

Fig. 7. An example of local MetaCube-XTM

The MetaCube-XTM is an XML Topic Maps (XTM) instance of MetaCube concept for supporting interoperability and integration among data warehouse systems. This metadata provides description of different multidimensional data models. It covers heterogeneity problems, such as syntactical, data model, semantic, schematic, and structural heterogeneities. 4.2.1 Schema Integration

Schema integration is intended to overcome the semantic heterogeneity and to provide a global view for clients. The process of schema integration consists of integration of local MetaCube-XTM(s) into the global MetaCube-XTM, and merging. The following section discusses issues concerning local MetaCube-XTM(s), the global MetaCubeXTM. • Local MetaCube-XTM With reference to the MetaCube-XTM UML modeling given in figure 5, the global MetaCube-XTM is represented in XTM document supporting multidimensional data model, such as cube, dimension, dimension schema, hierarchy, measures for each data warehouse. The local MetaCube-XTM is intended to provide data access at any level of data complexity. Figure 7 shows an example of a local MetaCube-XTM describing a local Web warehouse. • Global MetaCube-XTM The global MetaCube-XTM is aimed to provide a common framework for describing a multidimensional data model for Web warehouses. Therefore, the global MetaCubeXTM is a result of the integration of local MetaCube-XTMs. In the integration process, merging tools resolve heterogeneity problems (i.e., naming conflicts among different local MetaCube-XTMs). The merging process is based on the subject of topics available among local MetaCube-XTMs. The global MetaCube-XTM provides the

MetaCube XTM

85

logic to reconcile differences, and drive Web warehousing systems conforming to a global schema. In addition, the global MetaCube-XTM represents metadata that is used for query processing. If there is a query posted by users, the MetaCube-XTM service receives the query from the user, parses, checks, and compares it with the global MetaCubeXTM, and distributes it to selected local Web warehouses. Thus, in this model the global MetaCube-XTM must be able to represent heterogeneity of dimensions and measures from local Web warehouses in relation to the MetaCube-XTM model. An example of global MetaCube-XTM is given in figure 8. 4.2.2 Prototyping

To demonstrate the capability and efficiency of the proposed concept we use the prototype for the International Union of Forest Research Organizations (IUFRO) data warehouses, which are distributed in different Asian, African and European countries. Because of the genesis of the local (national) data warehouses, they are by nature heterogeneous. Local topic maps are used as an additional layer for the representation of local schema information, whereas in the global layer topic maps acts as mediator to hide conflicts of different local schemas. Currently we have implemented an incremental prototype with full data warehousing funtionality as a proof of concept for the feasibility of our approach of an XTM based data warehouse federation http://iufro.ifs.tuwien.ac.at/metacube2.0/. In detail, the MetaCube-XTM prototype has been implemented to function as follows: Each time a query is posed for the MetaCube-XTM systems, the following 5 steps are required (see figure 9). Steps 1-3 belong to the build-time stage at the global Metacube-XTM server. This stage covers the modeling and the design of the search metadata. In this stage all definitions for the required metadata are completed which are required to search into local DWH. The next step 4 belongs to the run-time stage or searching processes at local DWH systems by means of local MetaCube-XTMs. Steps 5 and 6 are used for displaying retrieved information. All steps are described following in more detail. /* global Topic Maps */ /* define topic Global Cube */ Global Meta Cube

…. Meta Cube-1 …

Fig. 8. An example of the global MetaCube-XTM

86

•

•

Thanh Binh Nguyen et al.

MetaCube-XTM Definitions. Dependent on the characteristics of the required data, different global MetaCube-XTM structures can be defined at the MetaCube-XTM server. In this step, user can select a number of dimensions and measure to define a MetaCube-XTM schema. MetaCube-XTM Browser. Based on the tree representation of selected dimension domains, user can roll up or drill down along each dimension to select elements for searching. These selected dimension elements will be used to for query in local Web warehouses.

Local MetaCube-XTM DWH Selections. This step provides flexibility in support of interoperability searching among multiple heterogeneous DWHs.

Global MetaCube-XTM

MetaCube-XTM Multi-Host Search Process

Searching Preparation 1

MetaCube-XTM Definitions

2

2) Dimension MetaCube-XTM Browser Definitions

3

DWH Selections

Local DWHs 4

Local MetaCube-X TM

Multi-Host Search

Local MetaCube-X

Local MetaCube-X TM

SampleTM Data Definitions

.......... DWH 1

DWH 2

DWH n

Resuts Information Retrievations

5

List of Results

6

Detail Result

Fig. 9. MetaCube-XTM Multi Host Search Processes

MetaCube XTM

5

87

Conclusion and Future Works

In this paper we have presented the concept of MetaCube-XTM, which is an XMLTopic Maps instance of MetaCube concept [19]. The MetaCube-XTM provides a framework to achieve interoperability between heterogeneous schema models, which enable the joint querying of distributed web data warehouse (OLAP) systems. We also describe how to use topic maps to deal with these issues. Local topic maps are used as an additional layer for the representation of local schema information, whereas in the global layer topic maps acts as mediator to hide conflicts of different local schemas. This concept facilitates to achieve semantic integration.

Acknowledgment The authors are very indebted to IUFRO for supporting our approach from the very beginning in the framework of GFIS (Global Forest Information Systems http://www.gfis.net).

References [1] [2] [3] [4] [5]

[6] [7]

Agrawal, R., Gupta, A., Sarawagi, A.: Modeling Multidimensional Databases. IBM Research Report, IBM Almaden Research Center, September 1995. Albrecht. J., Lehner, W.: On-Line Analytical Processing in Distributed Data Warehouses. International Databases Engineering and Applications Symposium (IDEAS), Cardiff, Wales, U.K, July 8-10, 1998. Ammoura A., O. Zaiane, and R. Goebel. Towards a Novel OLAP Interface for Distributed Data Warehouses. Proc. of DAWAK 2001, Springer LNCS 2114, pp. 174-185, Munich, Germany, Sept. 2001. Blaschka M., Sapia C., Höfling G., Dinter B.: Finding your way through multidimensional data models. In Proceeding of the 9th International DEXA Workshop, Vienna, Austria, August 1998. Bruckner R. M, Ling T. W., Mangisengi O., Tjoa A M.. A Framework for a Multidimensional OLAP Model using Topic Maps. In Proceedings of the Second International Conference on WebInformation Systems Engineering (WISE 2001) Conference (Web Semantics Workshop), Vol.2, pp. 109-118, IEEE Computer Society Press.Kyoto, Japan, December 2001. Computing Surveys, Vol. 22, No. 3, September 1990. Chaudhuri, S., Dayal, U.: An Overview of Data Warehousing and OLAP Technology. SIGMOD Record Volume 26, Number 1, September 1997. D. Fensel, I. Horrocks, F. Van Harmelen, S. Decker, M. Erdmann, and M. Klein. OIL in a Nutshell. In: Knowledge Acquisition, Modeling, and Management, Proc. of the 12th European Knowledge Acquisition Conference (EKAW2000), R. Dieng et al. (eds.), Springer-Verlag LNAI 1937, pp. 1-16, Oct. 2000.

88

[8] [9] [10] [11] [12] [13]

[14]

[15] [16] [17] [18] [19]

[20]

[21] [22] [23]

Thanh Binh Nguyen et al.

Garcia-Molina, H., Labio, W., Wiener, J.L., Zhuge, Y.: Distributed and Parallel Computing Issues in Data Warehousing. In Proceedings of ACM Principles of Distributed Computing Conference, 1999. Invited Talk. Gray J., Bosworth A., Layman A., Pirahesh H.: Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tabs, and Sub-Totals. Proceedings of ICDE '96, New Orleans, February 1996. Gross, J., Yellen, J.: Graph Theory and its applications. CRC Press,1999. I. Horrocks, D. Fensel, J. Broekstra, S. Decker, M. Erdmann, C. Goble, F. van Harmelen, M. Klein, S. Staab, R. Studer, and E. Motta. The Ontology Inference Layer OIL. Li, C., Wang, X.S.: A Data Model for Supporting On-Line Analytical Processing. CIKM 1996. Mangisengi, O., Tjoa, A M., Wagner, R.R.: Multidimensional Modelling Approaches for OLAP. Proceedings of the Ninth International Database Conference H “ eterogeneous and Internet Databases 1999, ISBN 962-937-046-8. Ed. J. Fong, Hong Kong, 1999. Mangisengi O, J. Huber, Ch. Hawel, and W. Essmayr. A Framework for Supporting Interoperability of Data Warehouse Islands Using XML. Proc. of DAWAK 2001, Springer LNCS 2114, pp. 328-338, Munich, Germany, Sept. 2001. Mattison R. Web Warehousing and Knowledge Management. McGraw-Hill, 1999. Meta Data Coalition. Metadata Interchange Specification (MDIS) Version 1.1, August 1997. Meta Data Coalition. Open Information Model XML Encoding. Version 1.0, December 1999. http://www.mdcinfo.com/. Meta Data Coalition. Open Information Model. Version 1.1, August 1999. http://www.mdcinfo.com/. Nguyen, T.B., Tjoa, A M., Wagner, R.R.: Conceptual Multidimensional Data Model Based on MetaCube. In Proc. of First Biennial International Conference on Advances in Information Systems (ADVIS'2000), Izmir, TURKEY, October 2000. Lecture Notes in Computer Science (LNCS), Springer, 2000. Nguyen T.B., Tjoa A M., Mangisengi O.. MetaCube-X: An XML Metadata Foundation for Interoperability Search among Web Warehouses. In Proceedings of the 3rd Intl. Workshop DMDW'2001, Interlaken, witzerland, June 4, 2000. The DARPA Agent Markup Language Homepage. http://daml.semanticweb.org/. The Semantic Web Homepage. http://www.semanticweb.org/ XML Topic Maps (XTM) 1.0 Specification. http://www.topicmaps.org/xtm/1.0/.

Designing Web Warehouses from XML Schemas Boris Vrdoljak1, Marko Banek1, and Stefano Rizzi2 1

FER – University of Zagreb Unska 3, HR-10000 Zagreb, Croatia {boris.vrdoljak,marko.banek}@fer.hr 2 DEIS – University of Bologna Viale Risorgimento 2, 40136 Bologna, Italy [email protected]

Abstract. Web warehousing plays a key role in providing the managers with up-to-date and comprehensive information about their business domain. On the other hand, since XML is now a standard de facto for the exchange of semi-structured data, integrating XML data into web warehouses is a hot topic. In this paper we propose a semi-automated methodology for designing web warehouses from XML sources modeled by XML Schemas. In the proposed methodology, design is carried out by first creating a schema graph, then navigating its arcs in order to derive a correct multidimensional representation. Differently from previous approaches in the literature, particular relevance is given to the problem of detecting shared hierarchies and convergence of dependencies, and of modeling many-to-many relationships. The approach is implemented in a prototype that reads an XML Schema and produces in output the logical schema of the warehouse.

1

Introduction

The possibility of integrating data extracted from the web into data warehouses (which in this case will be more properly called web warehouses [1]) is playing a key role in providing the enterprise managers with up-to-date and comprehensive information about their business domain. On the other hand, the Extensible Markup Language (XML) has become a standard for the exchange of semi-structured data, and large volumes of XML data already exist. Therefore, integrating XML data into web warehouses is a hot topic. Designing a data/web warehouse entails transforming the schema that describes the source operational data into a multidimensional schema for modeling the information that will be analyzed and queried by business users. In this paper we propose a semiautomated methodology for designing web warehouses from XML sources modeled by XML Schemas, which offer facilities for describing the structure and constraining the content of XML documents. As HTML documents do not contain semantic description of data, but only the presentation, automating design from HTML sources is unfeasible. XML models semi-structured data, so the main issue arising is that not Y. Kambayashi, M. Mohania, W. Wöß (Eds.): DaWaK 2003, LNCS 2737, pp. 89-98, 2003.  Springer-Verlag Berlin Heidelberg 2003

90

Boris Vrdoljak et al.

all the information needed for design can be safely derived. In the proposed methodology, design is carried out by first creating a schema graph, then navigating its arcs in order to derive a correct multidimensional representation in the form of a dependency graph where arcs represent inter-attribute relationships. The problem of correctly inferring the needed information is solved by querying the source XML documents and, if necessary, by asking the designer's help. Some approaches concerning related issues have been proposed in the literature. In [4] a technique for conceptual design starting from DTDs [12] is outlined. That approach is now partially outdated due to the increasing popularity of XML Schema; besides, some complex modeling situations were not specifically addressed in the paper. In [5] and [6] DTDs are used as a source for designing multidimensional schemas (modeled in UML). Though that approach bears some resemblance to ours, the unknown cardinalities of relationships are not verified against actual XML data, but they are always arbitrarily assumed to be to-one. Besides, the id/idref mechanism used in DTDs is less expressive than key/keyref in XML Schema. The approach described in [8] is focused on populating multidimensional cubes by collecting XML data, but assumes that the multidimensional schema is known in advance (i.e., that conceptual design has been already carried out). In [9], the author shows how to use XML to directly model multidimensional data, without addressing the problem of how to derive the multidimensional schema. Differently from previous approaches in the literature, in our paper particular relevance is given to the problem of detecting shared hierarchies and convergence of dependencies, and of modeling many-to-many relationships within hierarchies. The approach is implemented in a prototype that reads an XML Schema and produces in output the star schema for the web warehouse.

2

Relationships in XML Schema

The structure of XML data can be visualized by using a schema graph (SG) derived from the Schema describing the data. The method is adopted from [10], where simpler, but less efficient DTD is still used as a grammar. The SG for the XML Schema describing a purchase order, taken from the W3C's document [14] and slightly extended, is shown in Fig. 1. In addition to the SG vertices that correspond to elements and attributes in the XML Schema, the operators inherited from the DTD element type declarations are also used because of their simplicity. They determine whether the sub-element or attribute may appear one or more (“+”), zero or more (“*”), or zero or one times (“?”). The default cardinality is exactly one and in that case no operator is shown. Attributes and sub-elements are not distinguished in the graph. Since our design methodology is primarily based on detecting many-to-one relationships, in the following we will focus on the way those relationships can be expressed. There are two different ways of specifying relationships in XML Schemas. •

First, relationships can be specified by sub-elements with different cardinalities. However, given an XML Schema, we can express only the cardinality of the relationship from an element to its sub-elements and attributes. The cardinality

Designing Web Warehouses from XML Schemas

91

in the opposite direction cannot be discovered by exploring the Schema; only by exploring the data that conforms to the Schema or by having some knowledge about the domain described, it can be concluded about the cardinality in the direction from a child element to its parent. Second, the key and keyref elements can be used for defining keys and their references. The key element indicates that every attribute or element value must be unique within a certain scope and not null. If the key is an element, it should be of a simple type. By using keyref elements, keys can be referenced. Not just attribute values, but also element content and their combinations can be declared to be keys, provided that the order and type of those elements and attributes is the same in both the key and keyref definitions. In contrast to id/idref mechanism in DTDs, key and keyref elements are specified to hold within the scope of particular elements.

•

3

From XML Schema to Multidimensional Schema

In this section we propose a semi-automatic approach for designing a web warehouse starting from an XML Schema. The methodology consists of the following steps: 1. 2. 3. 4.

Preprocessing the XML Schema. Creating and transforming the SG. Choosing facts. For each fact: 4.1 Building the dependency graph from the SG. 4.2 Rearranging the dependency graph. 4.3 Defining dimensions and measures. 4.4 Creating the logical schema.

Given a fact, the dependency graph (DG) is an intermediate structure used to provide a multidimensional representation of the data describing the fact. In particular, it is a directed rooted graph whose vertices are a subset of the element and attribute vertices of the SG, and whose arcs represent associations between vertices. The root of the DG corresponds to the fact. purchaseOrder

shipTo

orderDate

billTo ?

name street city

state zip

+ ? items comment *

? country

?

item

product brand productCode weight key size

partNum ? keyref productName quantity shipDate USPrice

Fig. 1. The Schema Graph

92

Boris Vrdoljak et al.

While in most cases the hierarchies included in the multidimensional schema represent only to-one associations (sometimes called roll-up relationships since they support the roll-up OLAP operator), in some applications it is important to model also many-to-many associations. For instance, suppose the fact to be modeled is the sales of books, so book is one of the dimensions. Although books that have many authors certainly exist, it would be interesting to aggregate the sales by author. It is remarkable that summarizability is maintained through many-to-many associations, if a normalized weight is introduced [7]. Besides, some specific solutions for logical design in presence of many-to-many associations were devised [11]. However, since modeling many-to-many associations in a warehouse should be considered an exception, their inclusion in the DG is subject to the judgment of the designer, who is supposed to be an expert of the business domain being modeled. After the DG has been derived from the SG, it may be rearranged (typically, by dropping some uninteresting attributes). This phase of design necessarily depends on the user requirements and cannot be carried out automatically; since it has already been investigated (for instance in [2]), it is considered to be outside the scope of this paper. Finally, after the designer has selected dimensions and measures among the vertices of the DG, a logical schema can be immediately derived from it. 3.1

Choosing Facts and Building Dependency Graphs

The relationships in the Schema can be specified in a complex and redundant way. Therefore, we transform some structures to simplify the Schema, similarly as DTD was simplified in [10] and [6]. A common example of Schema simplification concerns the choice element, which denotes that exactly one of the sub-elements must appear in a document conforming to that Schema. The choice element is removed from the schema and a minOccurs attribute with value 0 is added to each of its subelements. The resulting simplified structure, although not being equivalent to the choice expression, preserves all the needed information about the cardinalities of relationships. After the initial SG has been created [10], it must undergo two transformations. First, all the key attributes or elements are located and swapped with their parent vertex in order to explicitly express the functional dependency relating the key with the other attributes and elements. Second, some vertices that do not store any value are eliminated. A typical case is an element that has only one sub-element of complex type and no attributes, and the relationship with its sub-element is to-many. We name such an element a container. Note that, when a vertex v is deleted, the parent of v inherits all the children of v and their cardinalities. The next step is choosing the fact. The designer chooses the fact among all the vertices and arcs of the SG. An arc can be chosen as a fact if it represents a many-tomany relationship. For the purchase order SG presented in Fig. 1, after the items element has been eliminated as a container, the relationship between purchaseOrder and item is chosen as a fact, as in Fig. 2. For each fact f, the corresponding DG must be built by including a subset of the vertices of the SG. The DG is initialized with the root f, to be enlarged by recursively navigating the relationships between vertices in the SG. After a vertex v of the SG is inserted in the DG, navigation takes place in two steps:

Designing Web Warehouses from XML Schemas

93

purchaseOrder ...

? ... orderDate

* fact

item

Fig. 2. Choosing a fact

1.

2.

For each vertex w that is a child of v in the SG: When examining relationships in the direction expressed by arcs of the SG, the cardinality information is expressed either explicitly by “?”, “*” and “+” vertices, or implicitly by their absence. If w corresponds to an element or attribute in the Schema, it is added to the DG as a child of v; if w is a “?” operator, its child is added to the DG as a child of v. If w is a “*” or “+” operator, the cardinality of the relationship from u, child of w, to v is checked by querying the XML documents (see Section 4.5): if it is to-many, the designer decides whether the many-to-many relationship between v and u is interesting enough to be inserted into the DG or not. For each vertex z that is a parent of v in the SG: When examining relationships in this direction, vertices corresponding to “?”, “*” and “+” operators are skipped since they only express the cardinality in the opposite direction. Since the Schema yields no further information about the relationship cardinality, it is necessary to examine the actual data by querying the XML documents conforming to the Schema (see Section 4.5). If a to-one relationship is detected, z is included in the DG.

Whenever a vertex corresponding to a keyref element is reached, the navigation algorithm “jumps” to its associated key vertex, so that descendants of the key become descendants of the keyref element. A similar approach is used in [3], where the operational sources are represented by a relational schema, when a foreign key is met during navigation of relations. See for instance Fig. 3, showing the resulting DG for the purchase order example. From the fact, following to-one relationship, the item vertex is added to the DG. Vertex productCode is defined to be a key (Fig.1). It is swapped with product, which then is dropped since it carries no value. The partNum vertex is a child of item and is defined as a key reference to the productCode attribute. size, weight and brand, the children of productCode, become descendants of the partNum attribute in the DG. 3.2

Querying XML Documents

In our approach, XQuery [15] is used to query the XML documents in three different situations: 1. 2. 3.

examination of convergence and shared hierarchies searching for many-to-many relationships between the descendants of the fact in SG searching for to-many relationships towards the ancestors of the fact in the SG

94

Boris Vrdoljak et al.

orderDate

name

productName

street

quantity USPrice

purchaseOrder-item zip

purchaseOrder USAddress

city

comment state

item

FACT

comment partNum

shipDate

country size

weight

brand

Fig. 3. The DG for the purchase order example

Note that, since in all three cases querying the documents is aimed at counting how many distinct values of an attribute v are associated to a single value of an attribute w, it is always preliminarily necessary to determine a valid identifier for both v and w. To this end, if no key is specified for an attribute, the designer is asked to define an identifier by selecting a subset of its non-optional sub-elements. Convergence and Shared Hierarchies. Whenever a complex type has more than one instance in the SG, and all of the instances have a common ancestor vertex, either a convergence or a shared hierarchy may be implied in the DG. A convergence holds if an attribute is functionally determined by another attribute along two or more distinct paths of to-one associations. On the other hand, it often happens that whole parts of hierarchies are replicated two or more times. In this case we talk of a shared hierarchy, to emphasize that there is no convergence. In our approach, the examination is made by querying the available XML documents conforming to the given Schema. In the purchase order example, following a to-one relationship from the fact, the purchaseOrder vertex is added to the DG. It has two children, shipTo and billTo (Fig. 1), that have the same complex type USAddress. The purchaseOrder element is the closest common ancestor of shipTo and billTo, thus all the instances of the purchaseOrder element have to be retrieved. For each purchaseOrder instance, the content of the first child, shipTo, is compared to the content of the second one, billTo, using the deep-equal XQuery operator as shown in Fig. 4.

let $x:= for $c in $retValue where not(deep-equal($c/first/content, $c/second/content)) return $c return count($x) Fig. 4. A part of the XQuery query for distinguishing convergence from shared hierarchy

Designing Web Warehouses from XML Schemas

95

By using the COUNT function, the query returns the number of couples with different contents. If at least one couple with different contents is counted, a shared hierarchy is introduced. Otherwise, since in principle there still is a possibility that documents in which the contents of the complex type instances are not equal will exist, the designer has to decide about the existence of convergence by leaning on her knowledge of the application domain. In our example, supposing it is found that shipTo and billTo have different values in some cases, a shared hierarchy is introduced. Many-to-Many Relationships between the Descendants of the Fact. While in most cases only to-one associations are included into the DG, there are situations in which it is useful to model many-to-many associations. Consider the SG in Fig. 5, modeling the sales of the books, where the bookSale vertex is chosen as the fact. After the book vertex is included into the DG, a to-many relationship between book and author is detected. Since including a one-to-many association would be useless for aggregation, the available XML documents conforming to the bookSale Schema are examined by using XQuery to find out whether the same author can write multiple books. A part of the query is presented in Fig. 6: it counts the number of distinct books (i.e. different parent elements) for each author (child) and returns the maximum number. If the returned number is greater than one, the relationship is many-to-many, and the designer may choose whether it should be included in the DG or not. If the examination of the available XML documents has not proved that the relationship is many-to-many, the designer can still, leaning on his or her knowledge, state the relationship as many-to-many and decide if it is interesting for aggregation. bookSale

book ?

+ title

quantity

date

store

year publisher

author

price

city storeNo

nameLast nameFirst

address

Fig. 5. The book sale example

max( ... for $c in distinct-values($retValue/child) let $p:=for $exp in $retValue where deep-equal($exp/child,$c) return $exp/parent return count(distinct-values($p)) ) Fig. 6. A part of a query for examining many-to-many relationships

96

Boris Vrdoljak et al. TIME timeKey orderDate dayOfWeek holiday month PRODUCT productKey partNum productName size weight brand

PURCHASE_ORDER shipToCustomerKey billToCustomerKey orderDateKey productKey USPrice quantity income

CUSTOMER customerKey customer name street zip city state country

Fig. 7. The star schema for the purchase order example

To-Many Relationships towards the Ancestors of the Fact. This type of search should be done because the ancestors of the fact element in the SG will not always form a hierarchically organized dimension in spite of the nesting structures in XML. When navigating the SG upwards from the fact, the relationships must be examined by XQuery since we have no information about the relationship cardinality, which is not necessarily to-one. The query is similar to the one for examining many-to-many relationships, and counts the number of distinct values of the parent element corresponding to each value of the child element. 3.3

Creating the Logical Scheme

Once the DG has been created, it may be rearranged as discussed in [3]. Considering for instance the DG in Fig. 3, we observe that there is no need for the existence of both purchaseOrder and purchaseOrder-item, so only the former is left. Considering item and partNum, only the latter is left. The comment and shipDate attributes are dropped to eliminate unnecessary details. Finally, attribute USAddress is renamed into customer in order to clarify its role. The final steps of building a multidimensional schema include the choice of dimensions and measures as described in [2]. In the purchase order example, USPrice and quantity are chosen as measures, while orderDate, partNum, shipToCustomer, and billToCustomer are the dimensions. Finally, the logical schema is easily obtained by including measures in the fact table and creating a dimension table for each hierarchy in the DG. Fig. 7 shows the resulting star schema corresponding to the DG in Fig. 3; note how the shared hierarchy on customer is represented in the logical model by only one dimension table named CUSTOMER, and how a derived measure, income, has been defined by combining quantity and USPrice. In the presence of many-to-many relationships one of the logical design solution proposed in [11] is to be adopted.

4

Conclusion

In this paper we described an approach to design a web warehouse starting from the XML Schema describing the operational source. As compared to previous approaches

Designing Web Warehouses from XML Schemas

97

based on DTDs, the higher expressiveness of XML Schema allows more effective modeling. Particular relevance is given to the problem of detecting shared hierarchies and convergences; besides, many-to-many relationships within hierarchies can be modeled. The approach is implemented in a Java-based prototype that reads an XML Schema and produces in output the star schema for the web warehouse. Since all the needed information cannot be inferred from XML Schema, in some cases the source XML documents are queried using XQuery language, and if necessary, the designer is asked for help. The prototype automates several parts of the design process: preprocessing the XML Schema, creating and transforming the schema graph, building the dependency graph, querying XML documents. All phases are controlled and monitored by the designer through a graphical interface that also allows some restructuring interventions on the dependency graph.

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

S. S. Bhowmick, S. K. Madria, W.-K. Ng, and E. P. Lim, “Web Warehousing: Design and Issues”, Proc. DWDM'98, Singapore, 1998. M. Golfarelli, D. Maio, and S. Rizzi, “Conceptual design of data warehouses from E/R schemes”, Proc. HICSS-31, vol. VII, Kona, Hawaii, pp. 334-343, 1998. M. Golfarelli, D. Maio, S. Rizzi, “The Dimensional Fact Model: a Conceptual Model for Data Warehouses”, International Journal of Cooperative Information Systems, vol. 7, n. 2&3, pp. 215-247, 1998. M. Golfarelli, S. Rizzi, and B. Vrdoljak, “Data warehouse design from XML sources”, Proc. DOLAP'01, Atlanta, pp. 40-47, 2001. M. Jensen, T. Møller, and T.B. Pedersen, “Specifying OLAP Cubes On XML Data”, Journal of Intelligent Information Systems, 2001. M. Jensen, T. Møller, and T.B. Pedersen, “Converting XML Data To UML Diagrams For Conceptual Data Integration”, Proc. DIWeb'01, Interlaken, 2001. R. Kimball. “The data warehouse toolkit”. John Wiley & Sons, 1996. T. Niemi, M. Niinimäki, J. Nummenmaa, and P. Thanisch, “Constructing an OLAP cube from distributed XML data”, Proc. DOLAP'02, McLean, 2002. J. Pokorny, “Modeling stars using XML”, Proc. DOLAP'01, 2001. J. Shanmugasundaram et al., “Relational Databases for Querying XML Documents: Limitations and Opportunities”, Proc. 25th VLDB, Edinburgh, 1999. I.Y. Song, W. Rowen, C. Medsker, and E. Ewen, “An analysis of many-tomany relationships between fact and dimension tables in dimensional modeling”, Proc. DMDW, Interlaken, Switzerland, pp. 6.1-6.13, 2001. World Wide Web Consortium (W3C), “XML 1.0 Specification”, http://www.w3.org/TR /2000/REC-xml-20001006. World Wide Web Consortium (W3C), “XML Schema”, http://www.w3.org/XML/Schema.

98

[14] [15]

Boris Vrdoljak et al.

World Wide Web Consortium (W3C), “XML Schema Part 0: Primer”, http://www.w3.org /TR/xmlschema-0/. World Wide Web Consortium (W3C), “XQuery 1.0: An XML Query Language (Working Draft)”, http://www.w3.org/TR/xquery/.

Building XML Data Warehouse Based on Frequent Patterns in User Queries Ji Zhang1, Tok Wang Ling1, Robert M. Bruckner2, A Min Tjoa2 1

2

Department of Computer Science National University of Singapore Singapore 117543

Institute of Software Technology Vienna University of Technology Favoritenstr. 9/188, A-1040 Vienna, Austria

{zhangji, lingtw}@comp.nus.edu.sg

{bruckner, tjoa}@ifs.tuwien.ac.at

Abstract. With the proliferation of XML-based data sources available across the Internet, it is increasingly important to provide users with a data warehouse of XML data sources to facilitate decision-making processes. Due to the extremely large amount of XML data available on web, unguided warehousing of XML data turns out to be highly costly and usually cannot well accommodate the users’ needs in XML data acquirement. In this paper, we propose an approach to materialize XML data warehouses based on frequent query patterns discovered from historical queries issued by users. The schemas of integrated XML documents in the warehouse are built using these frequent query patterns represented as Frequent Query Pattern Trees (FreqQPTs). Using hierarchical clustering technique, the integration approach in the data warehouse is flexible with respect to obtaining and maintaining XML documents. Experiments show that the overall processing of the same queries issued against the global schema become much efficient by using the XML data warehouse built than by directly searching the multiple data sources.

1. Introduction A data warehouse (DWH) is a repository of data that has been extracted, transformed, and integrated from multiple and independent data source like operational databases and external systems [1]. A data warehouse system, together with its associated technologies and tools, enables knowledge workers to acquire, integrate, and analyze information from different data sources. Recently, XML has rapidly emerged as a standardized data format to represent and exchange data on the web. The traditional DWH has gradually given way to the XML-based DWH, which becomes the mainstream framework. Building a XML data warehouse is appealing since it provides users with a collection of semantically consistent, clean, and concrete XML-based data that are suitable for efficient query and analysis purposes. However, the major drawback of building an enterprise wide XML data warehouse system is that it is usually extremely time and cost consuming that is unlikely to be successful [10]. Furthermore, without proper guidance on which information is to be stored, the resulting data warehouse cannot really well accommodate the users’ needs in XML data acquirement. In order to overcome this problem, we propose a novel XML data warehouse approach by taking advantage of the underlying frequent patterns existing in the query Y. Kambayashi, M. Mohania, W. W¨oß (Eds.): DaWaK 2003, LNCS 2737, pp. 99-108, 2003. c Springer-Verlag Berlin Heidelberg 2003

100

Ji Zhang et al.

history of users. The historical user queries can ideally provide us with guidance regarding which XML data sources are more frequently accessed by users, compared to others. The general idea of our approach is: Given multiple distributed XML data sources and their globally integrated schema represented as a DTD (data type definition) tree, we will build a XML data warehouse based on the method of revealing frequent query patterns. In doing so, the frequent query patterns, each represented as a Frequent Query Pattern Tree (FreqQPT), are discovered by applying a rule-mining algorithm. Then, FreqQPTs are clustered and merged to generate a specified number of integrated XML documents. Apparently, the schema of integrated XML documents in the warehouse is only a subset of the global schema and the size of this warehouse is usually much smaller than the total size of all distributed data sources. A smaller sized data warehouse can not only save storage space but also enable query processing to be performed more efficiently. Furthermore, this approach is more user-oriented and is better tailored to the user’s needs and interests. There has been some research in the field of building and managing XML data warehouse. The authors of [2] present a semi-automated approach to building a conceptual schema for a data mart starting from XML sources. The work in [3] uses XML to establish an Internet-based data warehouse system to solve the defects of client/server data warehouse systems. [4] presents a framework for supporting interoperability among data warehouse islands for federated environments based on XML. A change-centric method to manage versions in a web warehouse of XML data is published in [5]. Integration strategies and their application to XML Schema integration has been discussed in [6]. The author of [8] introduces a dynamic warehouse, which supports evaluation, change control and data integration of XML data. The remainder of this paper is organized as follows. Section 2 discusses the generation of XML data warehouses based on frequent query patterns of users’ queries. In Sections 3, query processing using the data warehouse is discussed. Experimental results are repoeted in Section 4. The final section conclude this paper.

2. Building a XML DWH Based on Frequent Query Patterns 2.1.

Transforming Users’ Queries into Query Path Transactions

XQuery is a flexible language commonly used to query a broad spectrum of XML information sources, including both databases and documents [7]. The following XQuery-formatted query aims to extract the ISBN, Title, Author and Price of books with a price over 20 dollars from a set of XML documents about book-related information. The global DTD tree is shown in Figure 1. FOR $a IN DOCUMENT (book XML documents)/book SATIFIES $a/Price/data()>20 RETURN {$a/ISBN, $a/Title, $a/Author, $a/Price}

Building XML Data Warehouse Based on Frequent Patterns in User Queries

101

Book

ISBN

Title

Author+

Name

Affiliation

Section+

Title

Publisher

Para*

Price

Year

Figure*

Title

Image

QP1: Book/ISBN QP2: Book/Title QP3: Book/Author/Name QP4:Book/Author/Affiliation QP5: Book/Price

Fig. 1. Global DTD Tree of multiple XML documents. Fig. 2. QPs of the XQuery sample.

A Query Path is a path expression of a DTD tree that starts at the root of tree. QPs can be obtained from the query script expressed using XQuery Statements. The sample query above can be decomposed into five QPs, as shown in Figure 2. The root of a QP is denoted as Root(QP) and all QPs in a query have the same root. Please note that two QPs with different roots are regarded as different QPs, although these two paths may have some common nodes. This is because different roots of paths often indicate dissimilar contexts of the queries. For example, two queries Author/Name and Book/Author/Name are different because Root(Author/Name)=Author Root(Book/Author/Name)=Book. A query can be expressed using a set of QPs which includes all the QPs that this query consists. For example, the above sample query, denoted as Q, can be expressed using a QP set such as Q={QP1, QP2, QP3, QP4, QP5}. By transforming all the queries into QP sets, we now obtain a database containing all these QP sets of queries, denoted as DQPS. We will then apply a rule-mining techniques to discover significant rules among the users’ query patterns. 2.2.

Discovering Frequent Query Path Sets in DQPS

The aim of applying a rule mining technique in DQPS is to discover Frequent Query Path Sets (FreqQPSs) in DQPS. A FreqQPS contains frequent QPs that jointly occur in DQPS. Frequent Query Pattern Trees (FreqQPTs) are built from these FreqQPSs and serve as building blocks of schemas of the integrated XML documents in the data warehouse. Formal definition of FreqQPTs is given as follows. Definition 1. Frequent Query Path Set (FreqQPS): From all the occurring QPs in DQPS transformed from user’s queries, a Frequent Query Path Set (FreqQPS) is a set of QPs: {QP1, QP2,…,QPn} that satisfies the following two requirements: (1) Support requirement: Support ({QP1, QP2,…,QPn}) • minsup; (2) Confidence requirement: For each QPi, Freq({QP1, QP2,…,QPn}) / Freq(QPi) • minconf. where Freq(s) counts the occurrence of set s in DQPS. In (1), Support({QP1, QP2,…,QPn}) = freq({QP1, QP2,…,QPn}) / N(DQPS), where N(DQPS) is the total number of QPs in DQPS. The constants minsup and minconf are the minimum support and confidence thresholds, specified by the user. A FreqQPS that consists of n QPs is termed as an n-itemset FreqQPS. The definition of a FreqQPS is similar to that of association rules. The support requirement is identical to the traditional definition of large association rules. The confidence requirement is, however, more rigid than the traditional definition. Setting a more rigid confidence requirement is to ensure the joint occurrence of QPs in a FreqQPS should be significant enough with respect to an individual occurrence of any

102

Ji Zhang et al.

QP. Since the number of QPs in the FreqQPS is unknown in advance, we will mine all FreqQPSs containing various numbers of itemsets. The FreqQPS mining algorithm is presented in Figure 3. The n-itemset QPS candidates are generated by joining (n-1)-itemset FreqQPSs. A pruning mechanism is devised to delete those candidates of the n-itemset QPSs that do not have n (n-1)-itemset subsets in the (n-1)-itemset FreqQPS list. The reason is that if one or more (n-1)-subsets of a n-itemset QPS candidate are missing in the (n-1)-itemset FreqQPS list, this n-itemset QPS cannot become a FreqQPS. This is obviously more rigid than pruning mechanism used in conventional association rule mining. For example, if one or more of the 2-itemset QPSs {QP1, QP2}, {QP1, QP3} and {QP2, QP3} are not frequent, then the 3-itemset QPS {QP1, QP2, QP3} cannot become a frequent QPS. The proof of this pruning mechanism is given below. The pruning the nitemset QPS candidates are evaluated in terms of the support and confidence requirements to decide whether or not they are a FreqQPS. The (n-1)-itemset FreqQPSs are finally deleted if they are subsets of some n–itemset FreqQPSs. For example, the 2itemset FreqQPT {QP1, QP2} will be deleted from 2-itemset FreqQPT list if the 3itemset {QP1, QP2, QP3} exists in the 3-itemset FreqQPT list. Algorithm MineFreqQPS Input: DQPS, minsup, minconf. Output: FreqQPS of varied number of itemsets. FreqQPS1={QP in DQPS| SatisfySup(QP)=true}; i=2; WHILE (CanFreqQPSi-1 is not empty) { CanQPSi=CanQPSGen(FreqQPSi-1); CanQPSi= CanQPSiʊ{QPSi| NoSubSet(QPSi, FreqQPSi-1)
Proof: Suppose a n-itemset QPS has only p (n-1)-itemset subsets QPSn-1i|1”i”p, meaning that there are (n-p) subsets of QPSn are missing in the (n-1)-itemset QPS list. These missing (n-p) subsets of QPSn, denoted as QPSn-1i| (p+1)”i”n, are definitely not FreqQPSs and they fail to satisfy the support or the confidence requirement or both. Specifically, (1) If QPSn-1i|(p+1)”i”n does not satisfy support requirement, then Support(QP1, QP2,…,QPn-1) < minsup. Because Support(QP1, QP2,…,QPn) ” Support(QP1, QP2 ,…,QPn-1), so Support(QP1, QP2,…,QPn) < minsup, meaning QPSn cannot become a n-itemset FreqQPS; (2) If QPSn-1i|(p+1)”i”n does not satisfy confidence requirement, then for a certain QPi, Freq(QP1, QP2,…,QPn-1) / Freq(QPi) < minconf. Because Support(QP1, QP2, …,QPn) ” Support(QP1, QP2,…,QPn-1), so for QPi, Freq(QP1, QP2,…,QPn) / Freq(QPi) < minconf, meaning that QPSn cannot become a n-itemset FreqQPS.

Building XML Data Warehouse Based on Frequent Patterns in User Queries

103

After we have obtained a number of FreqQPSs, their corresponding Frequent Query Pattern Trees (FreqQPTs) will be built. Definition 2. Frequent Query Pattern Tree (FreqQPT): Given a FreqQPS, its corresponding Frequent Query Pattern Tree (FreqQPT) is a rooted tree FreqQPT=, where V and E denote its vertex and edge sets, which are the union of the vertices and edges of QPs in this FreqQPS, respectively. The root of a FreqQPT, denoted as Root(FreqQPT), is the root of its constituting QPs. For example, suppose a FreqQPS has two QPs: Book/Title and Book/Author/Name. The resulting FreqQPT is shown in the Figure 4. Book

{Book/ Title, Book/ Author/ Name}

Title

Author Name

Fig. 4. Build a FreqQPT for a FreqQPS.

2.3.

Generating Schemas of Integrated XML Documents

When all FreqQPTs have been mined, the schema of the integrated XML document will be built. We have noticed that a larger integrated XML document usually requires larger space when it is loaded into main memory. In order to solve this problem, we alternatively choose to build a few, rather than only one, integrated XML documents from the FreqQPTs mined, making the integration more flexible. The exact number of integrated XML documents to be obtained is user- specified. The basic idea is to use a clustering technique to find a pre-specified number of clusters of FreqQPTs. The integration of the FreqQPTs is performed within each of the clusters. Similarity measurement of FreqQPTs We need to measure the similarity between two FreqQPTs in order to find the closest pair in each step of the clustering process. It is noticed that the complexity of merging two FreqQPTs is dependant on the distance of the roots of the FreqQPTs involved, rather than on the other nodes in the FreqQPTs. Intuitively, the closer the two roots are to each other, the easier the merging can be done and vice versa. To measure the similarity between the roots of two FreqQPTs, we have to first discuss the similarity between two nodes in the hierarchy of a global schema. In our work, the similarity computation between two nodes in the hierarchy is based on the edge counting method. We measure the similarity of nodes by first computing the distance between two nodes, since the distance can be easily obtained by edge counting. Naturally, the larger the number of edges between two nodes, the further apart the two nodes are. The distance between two nodes n1 and n2, denoted as NodeDist(n1, n2), is computed as NodeDist(n1, n2)= Nedge(n1, n2), where Nedge() returns the number of edges between n1 and n2. This distance can be normalized by dividing the maximum possible distance between two nodes in the hierarchy, denoted by LongestDist. The normalized distance between n1 and n2, denoted as NodeDistN(n1, n2), is computed as follows: NodeDistN(n1, n2)= Nedge(n1, n2)/LongestDist Thus the similarity between n1 and n2 is computed as: NodeSimN(n1, n2)=1- NodeDistN(n1, n2) We now give an example to show how the similarity between two roots of FreqQPTs is computed. Suppose there are two QPs, QP1: Book/ Price and QP2:

104

Ji Zhang et al.

Section/ Figure/ Image as shown in Figure 5. What we should do is to compute the similarity between the roots of these two QPs, namely Book and Section. The maximum length between two nodes in the hierarchy as shown in Figure 1 is 5 (from Name or Affiliation to Title or Image). Thus NodeSimN(Book, Section) = 1 – NodeDistN(Book, Section) = 1–1/5 = 4/5 = 0.8. Book Section Figure

Price

Image

Fig. 5. Similarity between two QPs.

Merging of FreqQPTs When a nearest pair of FreqQPTs is found in each step of the clustering, merging of these two FreqQPTs is performed. Let FreqQPT1=, FreqQPT2=, Root(FreqQPT1)=root1, Root(FreqQPT2)=root2, and FreQPTM be the new FreqQPT merged from FreqQPT1 and FreqQPT2. We will now present the definition of Nearest Common Ancestor Node (NCAN) of two nodes in the DTD tree before we give details of FreqQPT merging. Definition 3. Nearest Common Ancestor Node (NCAN): The NCAN of root nodes of two FreqQPTs root1 and root2 in the hierarchical structure of a global DTD tree H, denoted as NCANH(root1, root2), is the common ancestor node in H that are closest to both root1 and root2. To merge two closest FreqQPTs, the Nearest Common Ancestor Node (NCAN) of root1 and root2 has to be found, thereby these two FreqQPTs can be connected. We denote the vertex and edge set of the paths between NCANH(root1, root2) and root1 as VNCANĺroot1 and ENCANĺroot1, and those between NCANH(root1, root2) and root2 as VNCANĺroot2 and ENCANĺroot2. The FreQPTM in this case can be expressed as FreQPTM={Union(V1, V2, VNCANĺroot1,VNCANĺroot2), Union (E1, E2, ENCANĺroot1, ENCANĺroot2)} and Root(FreQPTM)= NCANH(root1, root2). Specifically, there are three scenarios in merging two FreqQPTs, namely, (1) the two FreqQPTs have the same root; (2) The root of one FreqQPT is an ancestor node of another FreqQPT’s root; (3) case other than (a) and (b). Figure 6 (a)-(c) give examples for each of the cases of FreqQPT merging discussed above. The dot-lined edges in the integrated schema, if any, are the extra edges that have to be included into the integrated schema in merging the two separate FreqQPTs.

Title

Book

Book

Book Price

Author Nam e

Title

Author Name

Price

(a) Example for Case 1. Book Title Title

Section Figure

Book Section Figure

Title

Image Title

Image

Building XML Data Warehouse Based on Frequent Patterns in User Queries

105

(b) Example for Case 2. Author

Book

Figure Author

Name

Affiliation

Title

Image

Name

Affiliation

Section Figure Title

Image

(c) Example for Case 3. Fig. 6 (a) – (c). Examples of FreqQPT merging.

Clustering of FreqQPTs The aim of clustering FreqQPTs is to group similar FreqQPTs together for further integration. Merging two closer FreqQPTs is cheaper and requires less re-structuring operations compared to merging two FreqQPTs far apart from each other. In our work, we utilize the agglomerative hierarchical clustering paradigm. The basic idea of agglomerative hierarchical clustering is to begin with each FreqQPT as a distinct cluster and merge the two closest clusters in each subsequent step until a stopping criterion is met. The stop criterion of the clustering is typically either the similarity threshold or the number of clusters to be obtained. We choose to specify the number of clusters since it is more intuitive and easy to specify, compared to the similarity threshold that is typically not known before the clustering process. Please note that k , the specified number of clusters to be obtained, should not be larger than the number of FreqQPTs, otherwise the error message will be returned. This is because the QPs in the same FreqQPTs are not allowed to be further split. In each step, the two closest FreqQPT pair will be found and merged into one FreqQPT and the number of current clusters will be decreased by 1 accordingly. This clustering process is terminated when k clusters are obtained. 2.4.

Acquire Data to Feed the Warehouse

The last step of building the XML data warehouse is to read data from XML data sources when the schemas of the integrated XML documents are ready. Coming from different data source across the Internet, these data may be incomplete, noisy, inconsistent, and duplicate. Processing efforts such as standardization, data cleaning and conflict solving need to be performed to make the data in the warehouse more consistent, clean, and concrete.

3 Processing of Queries Using the Date Warehouse One of the main purposes of building data warehouse is to facilitate the query processing. When there is no data warehouse, processing of queries use the single mediator architecture (shown in Figure 7), in which all the queries will be processed in this mediator and directed to the multiple XML data sources. When the data warehouse has been built, a dual-mediator architecture is adopted (shown in Figure 8). Mediator 1 processes all the incoming queries from users, and each query will be directed to either the data warehouse or mediator 2 which is responsible for further directing the queries to the XML data sources or both.

106

Ji Zhang et al.

Specifically, let QPSdwh be QP set of the integrated XML documents in the data warehouse. QPTra(q) be the QP transaction of the query q. (i) If QPTra(q)  QPSdwh, meaning that all the QPs of q can be found in the schemas of integrated XML documents in the data warehouse, and this query can be answered by using the data warehouse alone, then q will only be directed by mediator 1 to the XML data warehouse; (ii) if QPTra(q) Œ QPSdwh and QPTra(q)ŀQPSdwh is not empty, meaning that not all QPs of q can be found in the schemas of integrated XML documents in the data warehouse, and the data warehouse does not contain enough information to answer q, then q will be directed by mediator 1 to both the data warehouse and mediator 2; (iii) if QPTra(q)ŀQPSdwh is empty, indicating that the information needed to answer q is not contained in the warehouse, thus q will only directed by mediator 1 to mediator 2. X M L d ata so urces

… ... XML data sources …

.. . Mediator

… .. Users . Fig. 7. Query processing without data warehouse

X M L da ta w areh o use

M ed iato r 2

M ed ia to r 1

… .. U sers .

Fig. 8. Query processing with data warehouse

4. Experimental Results In this section, we will conduct experiments to evaluate the efficiency of the constructing schema of XML data integration and the speedup of query processing by means of the data warehouse we have built. We use a set of 50 XML documents about book information and generate their global DTD tree. Zipfian distribution is employed to produce transaction file of queries, because web queries and surfing patterns typically conform to the Zipf’s law [9]. In our work, the query transaction file contains 500 such synthetic queries based on which the data warehouse is built. All these experiments are carried out on the PC of 900 MHz PC with 256 megabytes of main memory running on Windows 2000. 4.1 Construction of the Data Warehouse Schema under Varying Number of Queries

Building XML Data Warehouse Based on Frequent Patterns in User Queries

107

Fig. 9. Efficiency of constructing the data warehouse schema under varying number of queries

Fig. 10. Comparative study on query answering time

First, we will evaluate the time spent in constructing the schema of XML data integration of the data warehouse under varying number of queries from which frequent query patterns are extracted. The number of the queries used is varied from 100 to 1,000. As shown in Figure 9, the time increases approximately in an exponential rate since the number of candidates of FreqQPSs generated increases exponentially as the number of queries goes up. 4.2 Speedup of Query Processing Using Data Warehouse The major benefits of building data warehouse system based on frequent query patterns are to not only obtain a smaller but more concrete and clean subset of original XML data sources but also helps speedup the query processing. In this experiment, we measure the response time for answering queries with and without the aid of the data warehouse, respectively. The number of queries to be answered ranges from 100 to 1,000. The results shown in Figure 10 justifies that, by using data warehouse we have built, the query answering is faster than that the case when there is no such a data warehouse. This is because that the potion of information contained in the data

108

Ji Zhang et al.

warehouse is smaller in size than that stored in the original data sources, reducing the volume of data needed to scanned in the query answering. In addition, the data has been undergone the processing such as standardization, data cleaning and conflict solving, thus the duplication of data is lower. The smaller size and lower duplication of the data in the warehouse contribute to the higher efficiency in query answering.

5

Conclusions

In this paper, we propose a novel approach to perform XML data warehousing based on the frequent query patterns discovered from historical user’s queries. A specific rule mining technique is employed to discover these frequent query patterns, based on which the schemas of integrated XML documents are built. Frequent query patterns are represented using Frequent Pattern Trees (FreqQPTs) that are clustered using a hierarchical clustering technique according to the integration specification to build the schemas of integrated XML documents. Experimental results show that query answering time is reduced when compared to the case when there is no such a data warehouse.

References [1] H. Garcia-Molina, W. Labio, J. L.Wiener, and Y. Zhuge: Distributed and Parallel Computing Issues in Data Warehousing. In Proc. of ACM Principles of Distributed Computing Conference (PODS), Puerto Vallarta, Mexico 1998. [2] M. Golfarelli, S. Rizzi, and B. Vrdoljak: Data Warehouse Design from XML Sources. In Proc. of ACM DOLAP’01, Atlanta, Georgia, USA, Nov. 2001. [3] S. M. Huang and C.H. Su: The Development of an XML-based Data Warehouse System. In Proc. of 3rd Intl. Conf. of Intelligent Data Engineering and Automated Learning (IDEAL’02), Springer LNCS 2412, pp. 206-212, Manchester, UK, Aug. 2002. [4] O. Mangisengi, J. Huber, C. Hawel and W. Essmayr: A Framework for Supporting Interoperability of Data Warehouse Islands using XML. In Proc. of 3rd Intl. Conf. DaWaK’01, Springer LNCS 2114, pp. 328-338, Munich, Germany, Sept. 2001. [5] A. Marian, S. Abiteboul, G. Cobena, and L. Mignet: Change-centric Management of Versions in an XML Warehouse. In Proc. of Intl. Conf. on Very Large Data Bases (VLDB’01), pp. 581-590, Roma, Italy, Sept. 2001. [6] K. Passi, L. Lane, S. Madria, B.C. Sakamuri, M. Mohania and S. Bhowmick: A Model for XML Schema Integration. In Proc. of 3rd Intl. Conf. EC-Web, Springer LNCS 2455, pp. 193-202, Aix-en-Provence, France, Sept. 2002. [7] XQuery Language 1.0. http://www.w3.org/TR/xquery/. [8] L. Xyleme. A Dynamic Warehouse for XML Data of the Web. IEEE Data Engineering Bulletin, Vol. 24(2), pp. 40-47, 2001. [9] L. H. Yang, M. L. Lee, W. Hsu, S. Acharya. Mining Frequent Query Patterns from XML Queries. In Proc. of 8th Intl. Symp. on Database Systems for Advanced Applications (DASFAA’03), Kyoto, Japan, March 2003. [10] L.Garber. Michael StoneBraker on the Importance of Data Integration. IT Professional, Vol. 1, No.3, pp 80, 77-79, 1999.

A Temporal Study of Data Sources to Load a Corporate Data Warehouse Carme Martín and Alberto Abelló Departament de Llenguatges i Sistemes Informàtics Universitat Politècnica de Catalunya Jordi Girona Salgado 1-3, E-08034 Barcelona, Catalunya {martin,aabello}@lsi.upc.es

Abstract. The input data of the corporate data warehouse is provided by the data sources, that are integrated. In the temporal database research area, a bitemporal database is a database supporting valid time and transaction time. Valid time is the time when the fact is true in the modeled reality, while transaction time is the time when the fact is stored in the database. Defining a data warehouse as a bitemporal database containing integrated and subject-oriented data in support of the decision making process, transaction time in the data warehouse can always be obtained, because it is internal to a given storage system. When an event is loaded into the data warehouse, its valid time is transformed into a bitemporal element, adding transaction time, generated by the database management system of the data warehouse. However, depending on whether the data sources manage transaction time and valid time or not, we could obtain the valid time for the data warehouse or not. The aim of this paper is to present a temporal study of the different kinds of data sources to load a corporate data warehouse, using a bitemporal storage structure.

1

Introduction

As defined in [4], a data warehouse (DW) is an architectural structure that supports the management of "Subject-oriented", "Integrated", "Time-variant", and "Nonvolatile" data. A temporal database (TDB) is introduced in [9] as a database that supports "Valid time" (VT) (i.e. the time when the fact becomes effective in reality), or "Transaction time" (TT) (i.e. the time when the fact is stored in the database), or both times. Note that this definition excludes "User-defined time", which is an uninterpreted attribute domain of time directly managed by the user and not by the database system. A "bitemporal database" is a database that supports VT and TT. The affinity between both concepts (i.e. DW and TDB) may not be obvious. However, time references are essential in business decisions, and the dissection of both definitions shows their closeness.

Y. Kambayashi, M. Mohania, W. Wöß (Eds.): DaWaK 2003, LNCS 2737, pp. 109-118, 2003.  Springer-Verlag Berlin Heidelberg 2003

110

Carme Martín and Alberto Abelló

Fig. 1. A Data Warehouse as a Bitemporal Database

We consider the accepted definition of DW in [4] could be rewritten in terms of TDB concepts. Firstly, "Time-variance" simply specifies that every record in the DW is accurate relative to some moment in time. On the other hand, the definition of VT in [6] states that it is the time when the fact is true in the modeled reality. Therefore, both outline the importance of showing when data is correct and exactly corresponds to reality. Moreover, "Non-volatility" refers to the fact that changes in the DW are captured in the form of a "time-variant snapshot". Instead of true updates, a new "snapshot" is added to the DW in order to reflect changes. This concept can be clearly identified with that of TT, defined in [6] as the time when the fact is current in the database. In [1], we define a DW as a bitemporal database containing integrated and subject-oriented data in support of the decision making process, as it is sketched in figure 1. The first implication of this definition is that TT is entirely maintained by the system, and no user is allowed to change it. Moreover, the system should also provide specific management mechanisms for VT. The importance of this temporal conception is also outlined in [8], which asks DW systems for support of advanced temporal concepts. Data in the DW comes from independent heterogeneous sources. Therefore, TT and, specially, VT, introduced in the DW depend on which times are provided by the data sources. In [1], we propose a storage structure to implement a bitemporal DW. We concentrate our previous work is the most common case of data sources, i.e. specific and logged sources. In this paper, we are going to explain the behaviour of our bitemporal storage structure for all the different kinds of data sources to show that it can also be used with all possible data sources. The paper is organized as follows. Next section describes the bitemporal storage structure used throughout the paper. Section 3 explains the temporal study of the

A Temporal Study of Data Sources to Load a Corporate Data Warehouse

111

different kinds of data sources and the implementation of these in the bitemporal storage structure. Finally, section 4 provides some conclusions.

2

A Bitemporal Storage Structure

In [1], we present a bitemporal storage structure, named Current/Historical, consisting of a Current table that reflects current data and a set of Historical tables that show the historical evolution of the data, as depicted in figure 2. The set of Current tables in the DW give rise to the Operational Data Store, defined in [4]. A bitemporal event occurs at a starting VT and is true until an ending VT. However, if the ending VT is not known at this moment, since data is currently valid in the sources, we use the special VT value "Now", whose semantics are explained in [3]. For example, if we hire an employee, the starting VT will be her/his starting hiring data, and the ending VT will be the value "Now", until s/he is fired. DW insertions initialize the starting TT to the current TT and the ending TT to the value "Until_changed" (UC). As the current time inexorably advances, the value of UC always reflects the current time. Moreover, the current TT is denoted by “Current”. The Current table only contains two different temporal values for each attribute (Ai), i.e. the starting VT (Vis) and the starting TT (Tis). The other time values for current data always contain “Now” for ending VT and “UC” for ending TT, so they do not need to be stored. Moreover, we have one Historical table for each set of attributes with the same temporal behaviour. In this way, each Historical table contains only one semantic concept, and all attributes change at a time. For example, if we store employee information (only one subject), we should use a different structure for home address and telephone, and another one for work address and telephone (two different temporal behaviours). If the company reallocates somebody in a new room, only her/his job data will change. If s/he moves to a bigger house, only home data will change. Even though all data regards the same subject, it shows a different time behaviour. Without loss of generality, in this paper, we will assume that every table contains at most one attribute. Therefore, Historical tables contain the history for each attribute, with the starting VT (Vs), the ending VT (Ve), the TT when the insert was processed (Ti) and the TT when the delete was processed (Td). From these four temporal values, we can easily reconstruct the whole history of the value.

Fig. 2. Current and Historical tables in the bitemporal DW

112

Carme Martín and Alberto Abelló

In TDB research area, insertion, deletion, and modification operations, are defined in [7]. In addition, in [10], temporal insertion, deletion, and modification operations are explained for relational databases with time support. However, in our bitemporal storage structure, we require a redefinition of TDB insertion, deletion and modification operations. On loading the DW, all we need to do to process an insert is add a new record to the Current table. The processing of a delete is not much more difficult: the corresponding record is removed from the Current table, and a new one is added to each one of the Historical tables. The modification changes the old values of the Current table with the new values and adds a new record in the corresponding Historical table of the modified attribute. This behaviour can be easily inferred from deletion and insertion behaviours.

3

Temporal Study of Data Sources

The input data of the DW is provided by the data sources, that are integrated. Depending on whether the data sources manage TT and VT or not, we could obtain the VT for the DW or not. TT in the DW can always be obtained, because it is internal to a given storage system. When an event is loaded into the DW, its VT, supplied by the "Extraction, Transformation and Load" (ETL) module, is transformed into a bitemporal element, adding TT, generated by the DW DBMS. In the next sections, let us study the different kinds of data sources proposed in [5] to be implemented in our bitemporal storage structure. We consider throughout the paper, as example, a bitemporal DW with a courses relation with an OID, a Name and a Cost attributes. 3.1

Snapshot and Queryable Sources

Snapshot sources are sources that the only way to access to the data source content is through a dump of its data. A queryable source is a data source that offers a query interface. From snapshot and queryable sources that do not keep any kind of time, we can only store the TT in the DW. From snapshot and queryable sources, if they do not have any temporal information, we can only consider timestamping the data while we extract them. In the absence of true VT, all we can do is approximate it by the DW TT. In figure 3, we show the insertion of a databases course with a 100 euros cost, for the course relation. Inserting this course into the bitemporal DW, the starting TT is recorded. Given the bitemporal nature of the DW, all we can do in this case is approximate the starting VT by means of TT information. The best approximation we could obtain for the starting VT is the starting point of the "update window", because all we know is that the event occurred before the beginning of the DW load. Moreover, since during the "update window" queries are not allowed, we can consider that the load is atomic, in the sense that there is no temporal order among the operations. Therefore, the TT can also be fixed at the beginning of the "update window". Thus, TT and VT will have the same value.

A Temporal Study of Data Sources to Load a Corporate Data Warehouse

113

Fig. 3. The databases course insertion in snapshot and queryable sources

Fig. 4. The databases course deletion in snapshot and queryable sources

In figure 4, the deletion process of the databases course is described. The deletion operation eliminates the bitemporal element of the Current table and adds a new one in each Historical table. Similar to the insertion operation, we approximate now the ending VT by means of TT. 3.2

Specific and Logged Sources

Specific sources are able to write "delta files" that describe the system actions. Logged sources have a "log file" where all their actions are registered. From specific and logged (those able to keep track of the performed operations) sources, if they timestamp the entries with the source TT, we can approximate the VT by means of it. If no other information exists, the data can be considered valid while it is current in the operational database. This is the most usual environment for a DW.

114

Carme Martín and Alberto Abelló

Fig. 5. The databases course insertion in specific and logged sources

Fig. 6. The databases course deletion in specific and logged sources

In figure 5, we show the insertion of a databases course with a 100 euros cost and with a source TT value of 1. We can see that source TT is converted into VT, and we also have the TT of the DW. In figure 6, the deletion process of the databases course is described. When a deletion operation comes from the data source, it has an ending TT value to be converted in an ending VT value in the DW. As it is shown, a logical deletion generates the physical removal of the existing bitemporal element in the Current table. The ending TT of the Historical table is the TT of the DW. However, this is not enough and a new bitemporal element is added to the Historical tables, which expresses that from now on we know the ending VT value, i.e. the timestamp of the data source. 3.3

Callback and Internal Action Sources

Callback sources are sources that provide triggers or other active capabilities. Internal action sources are similar to callback sources, except that to define triggers in the data source requires to create auxiliary relations called "delta tables". In these

A Temporal Study of Data Sources to Load a Corporate Data Warehouse

115

"cooperative" sources, the TT in the sources needs to be used again to approximate the VT. However, this kind of data sources offer two different kinds of DW load: Deferred Load. The most common possibility is to use the triggers to generate "delta files", and later on use these files to load the DW. Thus, the temporal information in the "delta files" will be the source TT. We have already described this case in section 3.2. Real-Time Load. Another possibility would be that both repositories (i.e. the data source and the DW) are updated at the same time. Then, the TT of the DW corresponds to source TT. Notice that, it is the same assumption that we have considered for snapshot and queryable sources. Therefore, this case has the same temporal behaviour than snapshot and queryable sources, explained in section 3.1. Nevertheless, they do not have the same temporal knowledge. In callback and internal action sources we really have a source TT to approximate the VT, while in snapshot and queryable sources we have no temporal information in the source, so this is a better approximation. 3.4

Bitemporal Data Sources

Bitemporal data sources are sources whose data are stored in a bitemporal database. From bitemporal data sources (not considered in [5]), we could obtain true VT besides TT. Moreover, we also know the TT of the DW. Therefore, we should choice one temporal attribute out of those two of the sources to be used as VT in the DW (the other one will be managed as a user-defined time attribute): Source TT Used as VT. If the VT of the DW is obtained from the source TT, the source VT could be an additional user-defined time attribute to be considered. In this case, bitemporal data sources will have the same temporal behaviour than specific and logged sources (explained in section 3.2) with an additional user-defined time attribute to express true valid time information. The temporal information provided to the ETL process will be: source TT (that will give rise to the VT of the DW), and source VT (that will be treated as a user-defined time attribute). Considering temporal attributes in this way allows to keep using effectively the Current/Historical storage structure. Source VT Used as VT. If the VT of the DW is obtained from the source VT, the source TT could be an additional user-defined time attribute to be considered. These bitemporal sources provide the following temporal information: source VT (that will give rise to the VT of the DW), and source TT (that will be treated as a user-defined time attribute). Since VT in the data source is an interval, we need to add another temporal attribute to the Current table to record the ending VT. Figure 7 presents this new possibility having bitemporal sources. Two new attributes need to be added to the Current table for every attribute: One for ending VT and another one for the user-defined time representing source TT. Notice that, the Current table represents current data in the source. Therefore, the ending source TT

116

Carme Martín and Alberto Abelló

will always be UC so that it is not necessary to record it. In this example, starting and ending VT are later than the load of the DW. Figure 8 shows the deletion of the databases course inserted in figure 7. Regarding the Historical tables, also two new user-defined time attributes need to be added to each table: one for starting source TT and another one for ending TT. We already had two attributes for starting and ending VT.

Fig. 7. The databases course insertion in bitemporal sources

Fig. 8. The databases course deletion in bitemporal sources

A Temporal Study of Data Sources to Load a Corporate Data Warehouse

117

Fig. 9. A coalescing example

The main problem in this case (not taking into account the increase in the number of attributes of the different tables) is that the OID is no more an identifier by itself for the Current table. Even more, we cannot guarantee that there are the same number of current values for each attribute, so that we should divide the Current table into independent tables for every attribute. Thus, we would have the same number of Current and Historical tables, which would worsen the performance of this storage technique. In this kind of sources, in order to reduce storage space, when two time intervals overlap or are adjacent, the coalescing operation of TDB [2] could be applied either to Current or Historical tables. When either VT or TT intervals have identical nontemporal attribute values in two different tuples, then the coalescing operation can obtain only one tuple from both if the other time interval that do not coincide overlap or the ending point of one of them is the starting point of the other. Even if the bitemporal source uses the coalescing operation, we can still find in the DW tuples to be coalesced, if they come from different sources. As example, in figure 9, we see that the databases course information could be coalesced in only one tuple.

4

Conclusions

In this paper, we have presented a temporal study of the different kinds of data sources to load a DW. The correspondences between temporal attributes in the data sources and those in the DW have been analyzed. In the corporate DW, we have identified the two existing orthogonal temporal dimensions: valid time dimension and transaction time dimension. In this bitemporal DW environment, we have used our bitemporal storage structure [1] to represent all the temporal data source knowledge obtained by the different kinds of data sources. Analyzing these different data sources, presented in [5] with an additional type, i.e. bitemporal source, we found that in some cases snapshot and queryable sources could have the same temporal behaviour than callback and internal action sources (just the latter being more

118

Carme Martín and Alberto Abelló

precise). In general, these "cooperative" sources would behave like specific and logged sources. Regarding bitemporal data sources, they are more difficult to manage and, in general, would need an ad hoc storage structure. However, using an appropriate interpretation, they can also be treated as specific and logged sources. Thus, we can use the Current/Historical bitemporal storage structure to warehouse any kind of data sources.

Acknowledgments The authors would like to thank Núria Castell for the support she has given to this work. This work has been partially supported by the Spanish Research Program PRONTIC under project TIC2000-1723-C02-01.

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]

Abelló, A.; Martín, C. "A Bitemporal Storage Structure for a Corporate Data Warehouse". Proc. of the 5th. Int. Conf. on Enterprise Information Systems (ICEIS). pages 177-183, 2003. Böhlen, M.; Snodgrass, R.T.; Soo, M.D. "Coalescing in Temporal Databases". In Proc. of the 22nd. Int. Conf. on Very Large Data Bases (VLDB). pages 180191, 1996. Clifford, J.; Dyreson, C.; Isakowitz, T.; Jensen, C.S., Snodgrass, R.T. "On the Semantics of 'Now' in Databases". ACM Transactions on Database Systems. 22(2):171-214, 1997. Inmon, W. H., Imhoff, C.; Sousa, R. "Corporate Information Factory". John Wiley & Sons, second edition, 1998. Jarke, M.; Lenzerini, M.; Vassilious, Y.; Vassiliadis, P., editors, "Fundamentals of Data Warehousing". Springer-Verlag, 2000. Jensen, C.S.; Clifford, J.; Gadia, S.K.; Segev, A.; Snodgrass, R.T. "A Glossary of Temporal Database Concepts". ACM SIGMOD Record. 21(3):35-43, 1992. Jensen, C.S.; Soo, M.D.; Snodgrass R.T. “Unifying Temporal Data Models via a Conceptual Model". Information Systems. 19(7):513-547, 1994. Pedersen, T.B.; Jensen, C.S. "Research Issues in Clinical Data Warehousing". In Proc. of the 10th. Int. Conf. on Statistical and Scientific Database Management (SSDBM). pages 43-52, 1998. Snodgrass, R.T.; Ahn, I. "Temporal Databases". IEEE Computer. 19(9):35-42, 1986. Snodgrass, R.T. "Developing Time-Oriented Database Applications in SQL". Morgan Kaufmann Publishers, 2000.

Automatic Detection of Structural Changes in Data Warehouses Johann Eder
, Christian Koncilia
, and Dieter Mitsche§




University of Klagenfurt Dep. of Informatics-Systems {eder,koncilia}@isys.uni-klu.ac.at § [email protected]

Abstract. Data Warehouses provide sophisticated tools for analyzing complex data online, in particular by aggregating data along dimensions spanned by master data. Changes to these master data is a frequent threat to the correctness of OLAP results, in particular for multi- period data analysis, trend calculations, etc. As dimension data might change in underlying data sources without notifying the data warehouse, we are exploring the application of data mining techniques for detecting such changes and contribute to avoiding incorrect results of OLAP queries.

1

Introduction and Motivation

A data warehouse is a collection of data stemming from different frequently heterogeneous data sources and is optimized for complex data analysis operation rather than for transaction processing. The most popular architecture are multidimensional data warehouses (data cubes) where facts (transaction data) are “indexed” by several orthogonal dimensions representing a hierarchical organization of master data. OLAP (on-line analytical processing) tools allow the analysis of this data, in particular by aggregating data along the dimensions with different consolidation functions. Although data warehouses are typically deployed to analyse data from a longer time period than transactional databases, they are not well prepared for changes in the structure of the dimension data. This surprising observation originates in the (implicit) assumption that the dimensions of data warehouses ought to be orthogonal, which, in the case of the dimension time means that all other dimensions ought to be time-invariant. In this paper we address another important issue: how can such structural changes be recognized, even if the sources do not notify the data warehouse about the changes. Such “hidden” changes are a problem, because (usually) such changes are not modifications of the schema of the data source. E.g., inserting the data of a new product or a new employee is a modification on the instance level. However, in the data warehouse such changes result in a modification of its structure. Of course, this defensive strategy of recognizing structural changes can only be an aid to avoid some problems, it is not a replacement for adequate means for Y. Kambayashi, M. Mohania, W. W¨oß (Eds.): DaWaK 2003, LNCS 2737, pp. 119-128, 2003. c Springer-Verlag Berlin Heidelberg 2003

120

Johann Eder et al.

managing knowledge about changes. Nevertheless, in several practical situation we able trace erroneous results of OLAP queries back to structural changes not known by the analysts and the data warehouse operators, erroneous in the sense that the the resulting data din not correctly represent the state of affairs in the real world. As means for detecting such changes we propose the use of data mining technique. In a nutshell, the problem can be described as a multidimensional outlier detection problem. In this paper we report on some experiments we were conducting to analyse – which data mining techniques might be applied for the detection of structural changes – how these techniques are best applied – whether these techniques effectively detect structural changes – whether these techniques scale up to large data warehouses. To the best of our knowledge it is the first time that the problem of changes in dimensions of data warehouses is addressed with data mining techniques. The problems related to the effects of structural changes in data warehouses and approaches to overcome the problems they cause were subject of several projects [Yan01, BSH99, Vai01, CS99] including our own efforts [EK01, EKM02] to build a temporal data warehouse structure with means to transform data between structural versions such that OLAP tools work on data cleaned of the effects of structural changes. The remainder of this paper is organized as follows: in section 2 we bring basic definitions and discuss the notion of structural changes in data warehouses. In section 3 we briefly introduce the data mining techniques we analyzed for the detection of structural changes and we introduce a procedure for applying these techniques. In section 4 we briefly discuss the experiments we conducted as proof of concept. Finally in section 5 we draw some conclusions.

2

Structural Changes

We will now briefly discuss different types of structural changes. Furthermore, we will argue why some of these structural changes do not need to be detected automatically. In [EK01] we showed how the basic operations INSERT, UPDATE and DELETE have to be adopted for a temporal data warehouse. With respect to dimension members, i.e., the instances of dimensions, these basic operations may be combined to represent the following complex operations (where Q is the chronon, i.e. it is a “a non-decomposable time interval of some fixed, minimal duration” [JD98]): i.) SPLIT: One dimension member M splits into n dimension members, M1 , ... ,Mn . This operation translates into a DELETE(M , Ts −Q) and a set of insert operations INSERT(Mi , Ts ).

Automatic Detection of Structural Changes in Data Warehouses Initial Outline

Insert new Div. A and Change Name of SubDiv. C

Split Div. A into A1 and A2

Merge Div. A1 and A2 Delete SubDiv. X

SV1

SV2

SV3

SV4

Div. E

Div. A

Div. A1

Div. E

Div. A2

Subdiv. C

Subdiv. X

Subdiv. D

Subdiv. D

121

Div. F

Div. E

Div. E

Subdiv. X

Subdiv. D

Subdiv. D Legend mapping function for Facts F1, ..., Fn

Fig. 1. An example of structural changes

ii.)

iii.)

iv.)

v.)

For instance,