Data Quality (The Kluwer International Series on Advances in Database Systems Volume 23) (Advances in Database Systems)

DATA QUALITY

The Kluwer International Series on ADVANCES IN DATABASE SYSTEMS Series Editor

Ahmed K. Elmagarmid Purdue University West Lafayette, IN 47907

Other books in the Series: THE FRACTAL STRUCTURE OF DATA REFERENCE: Applications to the Memory Hierarchy, Bruce McNutt; ISBN: 0-7923-7945-4 SEMANTIC MODELS FOR MULTIMEDIA DATABASE SEARCHING AND BROWSING, Shu-Ching Chen, R.L. Kashyap, and Arif Ghafoor; ISBN: 0-79237888-1 INFORMATION BROKERING ACROSS HETEROGENEOUS DIGITAL DATA: A Metadata-based Approach, Vipul Kashyap, Amit Sheth; ISBN: 0-7923-7883-0 DATA DISSEMINATION IN WIRELESS COMPUTING ENVIRONMENTS, KianLee Tan and Beng Chin Ooi; ISBN: 0-7923-7866-0 MIDDLEWARE NETWORKS: Concept, Design and Deployment of Internet Infrastructure, Michah Lerner, George Vanecek, Nino Vidovic, Dad Vrsalovic; ISBN: 0-7923-7840-7 ADVANCED DATABASE INDEXING, Yannis Manolopoulos, Yannis Theodoridis, Vassilis J. Tsotras; ISBN: 0-7923-7716-8 MULTILEVEL SECURE TRANSACTION PROCESSING, Vijay Atluri, Sushil Jajodia, Binto George ISBN: 0-7923-7702-8 FUZZY LOGIC IN DATA MODELING, Guoqing Chen ISBN: 0-7923-8253-6 INTERCONNECTING HETEROGENEOUS INFORMATION SYSTEMS, Athman Bouguettaya, Boualem Benatallah, Ahmed Elmagarmid ISBN: 0-7923-8216-1 FOUNDATIONS OF KNOWLEDGE SYSTEMS: With Applications to Databases and Agents, Gerd Wagner ISBN: 0-7923-8212-9 DATABASE RECOVERY, Vijay Kumar, Sang H. Son ISBN: 0-7923-8192-0 PARALLEL, OBJECT-ORIENTED, AND ACTIVE KNOWLEDGE BASE SYSTEMS, Ioannis Vlahavas, Nick Bassiliades ISBN: 0-7923-8117-3 DATA MANAGEMENT FOR MOBILE COMPUTING, Evaggelia Pitoura, George Samaras ISBN: 0-7923-8053-3 MINING VERY LARGE DATABASES WITH PARALLEL PROCESSING, Alex A. Freitas, Simon H. Lavington ISBN: 0-7923-8048-7 INDEXING TECHNIQUES FOR ADVANCED DATABASE SYSTEMS, Elisa Bertino, Beng Chin Ooi, Ron Sacks-Davis, Kian-Lee Tan, Justin Zobel, Boris Shidlovsky, Barbara Catania ISBN: 0-7923-9985-4 INDEX DATA STRUCTURES IN OBJECT-ORIENTED DATABASES, Thomas A. Mueck, Martin L. Polaschek ISBN: 0-7923-9971-4

DATA QUALITY

by

Richard Y. Wang Mostapha Ziad Yang W. Lee

KLUWER ACADEMIC PUBLISHERS New York / Boston / Dordrecht / London / Moscow

eBook ISBN: Print ISBN:

0-306-46987-1 0-792-37215-8

2002 Kluwer Academic Publishers New York, Boston, Dordrecht, London, Moscow All rights reserved No part of this eBook may be reproduced or transmitted in any form or by any means, electronic, mechanical, recording, or otherwise, without written consent from the Publisher

Created in the United States of America

Visit Kluwer Online at: and Kluwer's eBookstore at:

http: www.kluweronline.com http: www.ebooks.kluweronline.com

To our families ...

This page intentionally left blank.

Table of Contents

Preface

xiii

Chapter 1

1

Introduction

1

Fundamental Concepts Data vs. Information Product vs. Infomation manufacturing The Information Manufacturing System Information Quality is a Multi-Dimensional Concept The TDQM Cycle A Frameworkfor TDQM Define IP Measure IP Analyze IP Improve IP Summary Book Organization

2 2 3 4 4 5 6 7 12 13 14 14 14

Chapter 2

19

Extending the Relational Model to Capture Data Quality Attributes

19

The Polygen Model Architecture Polygen Data Structure Polygen Algebra The Attribute-based Model

20 21 22 23 25

viii Data structure Data manipulation QI-Compatibility and QIV-Equal Quality Indicator Algebra Data Integrity Conclusion

26 27 28 29 31 32

Chapter 3

37

Extending the ER Model to Represent Data Quality Requirements

37

Motivating Example Quality Requirements Identification Requirements Modeling Modeling Data Quality Requirements Data Quality Dimension Entity Data Quality Measure Entity Attribute Gerund Representation Conceptual Design Example Concluding Remarks

38 39 40 41 42 42 43 44 47

Chapter 4

49

Automating Data Quality Judgment

49

Introduction Quality Indicators and Quality Parameters Overview Related Work Data Quality Reasoner Representation of Local Dominance Relationships Reasoning Component of DQR First-Order Data Quality Reasoner The Q-Reduction Algorithm Q-Merge Algorithm Conclusion

50 51 51 52 52 53 54 57 58 60 61

ix

Chapter 5

63

Developing a Data Quality Algebra

63

Assumptions Notation Definitions A Data Quality Algebra Accuracy Estimation of Data Derived by Selection Worst case when error distribution is non-uniform Best case when error distribution is non-uniform Accuracy Estimation of Data derived by Projection Worst case when error distribution is non-uniform Best case when error distribution is non-uniform Illustrative Example Conclusion

64 65 65 69 69 70 71 71 72 73 73 76

Chapter 6

79

The MIT Context Interchange Project

79

Integrating Heterogeneous Sources and Uses The Information Integration Challenge Information Extraction and Dissemination Challenges Information Interpretation Challenges Overview of The Context Interchange Approach Context Interchange Architecture Wrapping Mediation Conclusion

80 81 82 82 84 84 85 86 91

Chapter 7

93

The European Union Data Warehouse Quality Project

93

Introduction The D WQ Project DWQ Project Structure

93 95 95

x DWQ Objectives Data Warehouse Components The Linkage to Data Quality D WQ Architectural Framework The Conceptual Perspective The Logical Perspective The Physical Perspective Relationships between the Perspectives Research Issues Rich Data Warehouse Architecture Modeling Languages Data Extraction and Reconciliation Data Aggregation and Customization Query Optimization Update Propagation Schema and Instance Evolution Quantitative Design Optimization Overview of the DWQ demonstration Source Integration (Steps 1,2) Aggregation and OLAP Query Generation (Steps 3, 4) Design Optimization and Data Reconciliation (Steps 5,6) Summary and Conclusions

96 97 98 100 101 101 102 103 103 104 105 105 106 107 108 108 108 109 113 113 114

Chapter 8

119

The Purdue University Data Quality Project

119

Introduction Background Related Work Searching Process Matching Process Methodology Data Pre-processing Sorting, Sampling & the Sorted Neighborhood Approach Clustering and Classification Decision Tree Transformation and Simplification Experiments and Observations Conclusions

120 121 122 122 123 124 125 126 127 128 130 135

xi

Chapter 9

139

Conclusion

139

Follow-up Research Other Research Future Directions

140 143 143

Bibliography

149

Index

163

This page intentionally left blank.

P R E F A C E

If you would not be forgotten, As soon as you are dead and rotten, Either write things worth reading, Or do things worth the writing. Benjamin Franklin

This book provides an exposé of research and practice in data quality for technically oriented readers. It is based on the research conducted at the MIT Total Data Quality Management (TDQM) program and the work of other leading research institutions. It is intended for researchers, practitioners, educators and graduate students in the fields of Computer Science, Information Technology, and other inter-disciplinary fields. This book describes some of the pioneering research results that form a theoretical foundation for dealing with advanced issues related to data quality. In writing this book, our goal was to provide an overview of the cumulated research results from the MIT TDQM research perspective as it relates to database research. As such, it can be used by Ph.D. candidates who wish to further pursue their research in the data quality area and by IT professionals who wish to gain an insight into theoretical results and apply them in practice. This book also complements the authors’ other well-received book Quality Information and Knowledge that deals with more managerially- and practice-oriented topics (Prentice Hall, 1999) and the book Journey to Data Quality: A Roadmapfor Higher Productivity (in preparation). The book is organized as follows. We first introduce fundamental concepts and a framework for total data quality management (TDQM) that encompasses data quality definition, measurement, analysis and improvement. We then present the database-oriented work from the TDQM program, work that is focused on the data quality area in the database field. Next, we profile research projects from leading research institutions. These projects broaden the reader’s perspective and offer the reader a snapshot of some of the cutting-edge research projects. Finally, concluding remarks and future directions are presented. We thank Cambridge Research Group for sponsoring this project and providing a productive work environment. We are indebted to Professor Stuart Madnick at MIT for his continuous, unreserved support and encouragement. The support ofDean Louis Lataif, Senior Associate Dean Michael Lawson and Professor John Henderson at Boston University’s School of Management is appreciated. Also appreciated is the help and support from Dean John Brennan, Professors Warren

xiv

Preface

Briggs, Jonathan Frank, and Mohamed Zatet at Suffolk University. In addition, we appreciate the support from Dean Ira Weiss and Senior Associate Dean Jim Molloy at Northeastern University’s College of Business Administration. We thank John F. Maglio and Tom J. Maglio for their assistance in fulfilling many of book requirements such as the index and format. Thanks are also due to our colleagues and researchers whose work is included in this book, including Don Ballou, Ahmed Elmagarmid, Chris Firth, Jesse Jacobson, Yeona Jang, Henry Kon, Matthias Jarke, Harry Pazer, Stuart Madnick, Leo Pipino, M. P. Reddy, Giri Kumar Tayi, Diane Strong, Veda Storey, Yannis Vassiliou, Vassilios Verykios, and Yair Wand. We must also thank the many practitioners who have made significant contributions to our intellectual life. Although it is not possible to list all of them, we would like to express our gratitude to Jim Funk at S.C. Johnson, Kuan-Tsae Huang at TASKCo.com, Ward Page at DARPA’s Command Post of the Future program, Sang Park at Hyundai Electronics, Brent Jansen at IBM, Jim McGill at Morgan Stanley Dean Witter, Mel Bergstein, Alan Matsumura, and Jim McGee at Diamond Technology Partners, Jim Reardon at the Under-Secretary’s Office of the Defense Department for Human Affairs, Gary Ropp and David Dickason at U.S. Navy Personnel Research, Studies, and Technology, Terry Lofton and Frank Millard at Georgia State’s Public Health Division, Bruce Davidson and Alein Chun at Cedars Sinai Medical Center, Frank Davis and Joe Zurawski at Firstlogic, Madhavan Nayar at Unitech Systems Inc., and Les Sowa and Beam Suffolk at Dow Chemical. In addition, we thank D.J. Chang at Dong-Nam Corporation, Kyung-Sup Lee at Korean Airlines, Dwight Scott and Karen O’Lessker at Blue Cross Blue Shield, Kary Burin at Hewlett Packard, Don Hinman and John Talburt at Axiom Corporation, Peter Kaomea at Citicorp, Lu-Yue Huang at Edgewave, Col. Lane Ongstad of the U.S. Air Force Surgeon General’s Office, Lt. Col. Dave Corey of the U.S. Army Medical Command, Steve Hemsley, Dane Iverson and Raissa Katz-Haas at United Health Group, Sue Bies at the First Tennessee National Corporation, and Rita Kovac at Concept Shopping. We are grateful for the contributing authors and publishers for granting us permission to reprint their materials in this book. Specifically, Chapter 1 is based, in part, on the paper, A Product Perspective on Total Data Quality Management, authored by Richard Wang and published the Communications of ACM in 1998. Chapter 2 is based, in part, on the paper, A Polygen Model for Heterogeneous Database Systems: The Source Tagging Perspective, co-authored by Richard Wang and Stuart Madnick and published in the Proceedings of the International Conference on Very Large Database Conference in 1990. It is also based on the paper, reprinted from Decision Support Systems, 13, Richard Wang, M. P. Reddy, and Henry Kon, Toward quality data: An attribute-based approach, 349-372, 1995, with permission from Elsevier Science. Chapter 3 has been contributed by Veda Storey and Richard Wang based on the paper, An Analysis of Quality Requirements in Database Design, that they coauthored and published in the Proceedings of the 1998 Conference on Information

xv

Quality. Chapter 4 is based, in part, on the M.I.T. CISL working paper, A Knowledge-Based Approach to Assisting In Data Quality Judgment, co-authored by Yeona Jang, Henry Kon, and Richard Wang. Chapter 5 is based, in part, on the M.I.T. TDQM working paper, A Data Quality Algebra for Estimating Query Result Quality. The MIT Context Interchange Project in Chapter 6 has been contributed by Stuart Madnick based on the paper, Metadata Jones and the Tower of Babel: The Challenge of Large-Scale Semantic Heterogeneity, published in the Proceedings of 1999 IEEE Meta-Data Conference. The European Union Data Warehouse Quality Project in Chapter 7 has been contributed by Matthias Jarke and Yannis Vassiliou based on their paper “Data Warehouse Quality: A Review of the DWQ Project”, published in the Proceedings of the 1997 Conference on Information Quality. The Purdue University Data Quality Project in Chapter 8 has been written by Vassilios S. Verykios under the direction of Ahmed Elmagarmid. It profiles one aspect ofthe ongoing Purdue Univesity’s data quality initiative. Originally, additional chapters based on the work conducted at Telcordia (formerly Bellcore), Georgia Mason University, The University of Queensland, Australia, the University of St. Gallen in Switzerland, Universidad de Buenos Aires in Argentina, and other institutions were planned. Unfortunately, we were unable to incorporate them into the current edition, but we certainly hope to do so in a future publication. We appreciate Elizabeth Ziad for her love, support, and understanding, our children, Fori Wang, and Leyla, Ahmed, and Nasim Ziad who bring so much joy and happiness to our lives, as well as Abdallah, Abdelkrim, Ali and the Ziad family. In addition, we thank A. Russell and Arlene Lucid, Dris Djermoun, Kamel YoucefToumi, Hassan Raffa, Said Naili, Youcef Bennour, Boualem Kezim, Mohamed Gouali, Ahmed Sidi Yekhlef, YoucefBoudeffa, and Nacim Zeghlache for their help, support, and generosity. We also thank Jack and Evelyn Putman, and John, Mary Lou, Patrick, and Helen McCarthy for their love and support. Last, but not least, we would like to thank our parents who instilled in us the love of learning. Richard Y. Wang Boston University, Boston, Massachusetts & MIT TDQM program (http://web.mit.edu/tdqm/) [email protected] Mostapha Ziad Suffolk University, Boston, Massachusetts [email protected] Yang W. Lee Northeastern University, Boston, Massachusetts [email protected]

This page intentionally left blank.

C H A P T E R

1

Introduction

Many important corporate initiatives, such as Business-to-Business commerce, integrated Supply Chain Management, and Enterprise Resources Planning are at a risk of failure unless data quality is seriously considered and improved. In one study of a major manufacturer, it was found that 70% of all orders had errors. Aspects of this problem have even gained the attention of the Wall Street Journal, which reported that “A major financial institution was embarrassed because of a wrong data entry of an execution order of $500 million...” “Some Northeast states are filing multi-million dollar suits against TRW for bad credit data...” “Mortgage companies miscalculated home owners’ adjustable rate monthly mortgage payments, totaling multi-billion dollars...” The major news networks have also reported that many patients died or became seriously ill because of prescription data errors, prompting the Clinton Administration to issue an executive order to address the problem. Anecdotes of high-stake data quality problems abound in the academic literature and news articles.

2

Introduction

Chapter 1

The field of data quality has witnessed significant advances over the last decade. Today, researchers and practitioners have moved beyond establishing data quality as a field to resolving data quality problems, which range from data quality definition, measurement, analysis, and improvement to tools, methods, and processes. As a result, numerous data quality resources are now available for the reader to utilize [ 10, 29]. Many professional books [11, 12, 19, 24, 26], journal articles, and conference proceedings have also been produced. Journal articles have been published in fields ranging from Accounting [6, 17, 21, 36] to Management Science [1, 5] to Management Information Systems [2, 25, 35] to Computer Science [3, 4, 13, 16, 18, 20, 23, 27, 31, 34], and Statistics [7, 14, 22]. For example, Communications of the ACM, a premier publication of the Association of Computing Machinery, featured a special section entitled “Examining Data Quality” [28]. Not-for-profit academic conferences and commercial conferences on data quality are part of today’s data quality landscape. Back in 1996, the Total Data Quality Management (TDQM) program at the Massachusetts Institute of Technology pioneered the first academic conference for exchanging research ideas and results between researchers and practitioners. The overwhelmingly positive feedback from the conference participants led the organizers to hold the conference every year since. Today, it is a well-established tradition to hold the MIT Conference on Information Quality annually. Additionally, many information systems and computer science conferences now feature tracks or tutorials on data quality. Commercial conferences include those organized by Technology Transfer Institute [30] and others. These commercial conferences typically feature invited speakers as well as vendor and consultant workshops. In short, the field of data quality has evolved to such a degree that an exposé of research and practice of the field is both timely and valuable.

FUNDAMENTAL CONCEPTS Before we present the theme of this book, we first introduce some fundamental concepts that we have found to be useful to researchers and practitioners alike.

Data vs. Information Data and information are often used synonymously in the literature. In practice, managers intuitively differentiate between information and data, and describe information as data that have been processed. Unless specified otherwise, this book will use the term data interchangeably with the term information.

Fundamental Concepts

3

Product vs. Information manufacturing An analogy exists between quality issues in product manufacturing and those in information manufacturing, as shown in Table 1.1. Product manufacturing can be viewed as a processing system that acts on raw materials to produce physical products. Analogously, information manufacturing can be viewed as a system acting on raw data to produce information products. The field ofproduct manufacturing has an extensive body of literature on Total Quality Management (TQM) with principles, guidelines, and techniques for product quality. Based on TQM, knowledge has been created for data quality practice. Table 1.1 : Product vs. Information manufacturing

(Source: IEEE Transactions on Knowledge and Data Engineering [34])

An organization would follow certain guidelines to scope a data quality project, identify critical issues, and develop procedures and metrics for continuous analysis and improvement. Although pragmatic, these approaches have limitations that arise from the nature of raw materials, namely data, used in information manufacturing. Whereas data can be utilized by multiple consumers and not depleted, a physical raw material can only be used for a single physical product. Another dissimilarity arises with regard to the intrinsic property of timeliness of data. One could say that a raw material (e.g., copper) arrived just in time (in a timely fashion); however, one would not ascribe an intrinsic property of timeliness to the raw material. Other dimensions such as the believability of data simply do not have a counterpart in product manufacturing [5].

4

Introduction

Chapter 1

The Information Manufacturing System We refer to an information manufacturing system as a system that produces information products. The concept of an information product emphasizes the fact that the information output from an information manufacturing system has value transferable to the consumer, either internal or external. We define three roles (called the three C’s) in an information manufacturing system: 1. Information collectors are those who create, collect, or supply data for the information product. 2. Information custodians are those who design, develop, or maintain the data and systems infrastructure for the information product. 3. Information consumers are those who use the information product in their work. In addition, we define information product managers as those who are responsible for managing the entire information product production process and the information product life cycle. Each of the three C’s is associated with a process or task • Collectors are associated with data-production processes • Custodians with data storage, maintenance, and security • Consumers with data-utilization processes, which may involve additional data aggregation and integration. We illustrate these roles with a financial company’s client account database. A broker who creates accounts and executes transactions has to collect, from his clients, the information necessary for opening accounts and executing these transactions, and thus is a collector. An information systems professional who designs, develops, produces, or maintains the system is a custodian. A financial controller or a client representative who uses the information system is a consumer. Finally, a manager who is responsible for the collection, manufacturing, and delivery of customer account data is an information product manager.

Information Quality is a Multi-Dimensional Concept Just as a physical product (e.g., a car) has quality dimensions associated with it, an information product also has information quality dimensions. Information quality has been viewed as fitness for use by information consumers, with four information quality categories and fifteen dimensions identified [35]. As shown in Table 1.3, intrinsic information quality captures the fact that information has quality in its own right. Accuracy is merely one of the four dimensions underlying this category. Contextual information quality highlights the requirement that information quality must be con-

Fundamental Concepts

5

sidered within the context of the task at hand. Representational and accessibility information quality emphasize the importance ofthe role ofinformation systems. Table 1.3: Information Quality Categories and Dimensions

Information Quality Category

Information Quality Dimensions

Intrinsic Information Quality

Accuracy, Objectivity, Believability, Reputation Accessibility Information Qulity Accessibility, Access eecurity Relevancy, Value-Added, Timeliness, Contextual Information Quality Completeness, Amount of information Representational Information Quality Interpretability, Ease of understanding, Concise representation, Consistent representation, Ease of manipulation (Source: Journal of Management Information Systems [35])

The TDQM Cycle Defining, measuring, analyzing, and continuously improving information quality is essential to ensuring high-quality information products. In the TQM literature, the widely practiced Deming cycle for quality enhancement consists of: Plan, Do, Check, and Act. By adapting the Deming cycle to information manufacturing, we have developed the TDQM cycle [32], which is illustrated in Figure 1.1. The definition component of the TDQM cycle identifies important data quality dimensions [35] and the corresponding data quality requirements. The measurement component produces data quality metrics. The analysis component identifies root causes for data quality problems and calculates the impacts of poor quality information. Finally, the improvement component provides techniques for improving data quality. These components are applied along data quality dimensions according to requirements specified bythe consumer.

Figure 1.1 : The TDQM Cycle (Source: Communications of the ACM [32])

6

Introduction Chapter 1

A FRAMEWORK FOR TDQM In applying the TDQM framework, an organization must: (1) clearly articulate the information product (IP) in business terms; (2) establish an IP team consisting of a senior manager as a champion, an IP engineer who is familiar with the TDQM framework, and members from information collectors, custodians, consumers, and IP managers; (3) teach data quality assessment and data quality management skills to all the IP constituents; and (4) institutionalize continuous IP improvement. A schematic of the TDQM framework is shown in Figure 1.2. The tasks embedded in this framework are performed in an iterative manner. For example, an IP developed in the past may not fit today’s business needs for private client services. This should have been identified in the continuous TDQM cycle. If not, it would be the IP team’s responsibility to ensure that this need is met at a later phase; otherwise this IP will not be fit for use by the private client representatives,

Figure 1.2: A Schematic of the TDQM Framework

(Source: Communications of the ACM [32])

In applying the TDQM framework, one must first define the characteristics of the IP, assess the IP’s data quality requirements, and identify the information manufacturing system for the IP [5]. This can be challenging for organizations that are not familiar with the TDQM framework. Our experience has shown, however, that after the previously mentioned tasks have been performed a number of times and the un-

A Framework for TDQM

7

derlying concepts and mechanisms have been understood; it becomes relatively easy to repeat the work. Once these tasks are accomplished, those for measurement, analysis, and improvement can ensue.

Define IP The characteristics of an IP are defined at two levels. At a higher level, the IP is conceptualized in terms of its functionalities for information consumers just like when we define what constitutes a car, it is useful to first focus on the basic functionalities and leave out advanced capabilities (e.g., optional features for a car such as A/C, stereo, or cruise control). Continuing with the client account database example, the functionalities are the customer information needed by information consumers to perform the tasks at hand. The characteristics for the client account database include items such as account number and stock transactions. In an iterative manner, the functionalities and consumers of the system are identified. The consumers include brokers, client representatives, financial controllers, accountants, and corporate lawyers (for regulatory compliance). Their perceptions ofwhat constitutes important data quality dimensions need to be captured and reconciled. At a lower level, one would identify the IP’s basic units and components, along with their relationships. Defining what constitutes a basic unit for an IP is critical as it dictates the way the IP is produced, utilized, and managed. In the client account database, a basic unit would be a un-grouped client account. In practice, it is often necessary to group basic units together (e.g., eggs are packaged and sold by the dozen). A mutual-fund manager would trade stocks on behalf ofmany clients, necessitating group accounts; top management would want to know how much business the firm has with a client that has subsidiaries in Europe, the Far East, and Australia. Thus, a careful management of the relationship between basic accounts and aggregated accounts, and the corresponding processes that perform the mappings are critical because of their business impacts. Components of the database and their relationships can be represented as an entity-relationship (ER) model. In the client account database, a client is identified by an account number. Company-stock is identified by the company’s ticker symbol. When a client makes a trade (buy/sell), the date, quantity of shares and trade price are stored as a record of the transaction. An ER diagram of the client account database is shown in Figure l.3.

8

Introduction Chapter 1

Figure 1.3: A Client Account Schema

With the characteristics of the IP specified, the next step is to identify data quality requirements from the perspectives of IP suppliers, manufacturers, consumers, and managers. We have developed an instrument for data quality assessment and corresponding software to facilitate the data quality assessment task. After data are collected from information collectors, custodians, consumers, and IP managers and entered into the survey database, the data quality assessment software tool performs the query necessary for mapping the item values in the surveys to the underlying data quality dimensions [9]. Figure 1.4 illustrates the capabilities ofthe software tool.

Figure 1.4: Dimensional Assessment of Data Quality Importance Across Roles

A Framework for TDQM

9

As can be seen from Figure 1.4, the result from the first dimension indicates that the custodian believes the IP to be largely free of error (score 7 on a scale from 0 to 10 with 10 being completely free of error), whereas the consumer does not think so (with a score of 4). Both the custodian and the consumer indicate that the IP contains data that are objective and relatively important (score 7). The biggest contrast shows up for Dimension 8, completeness. Although the custodian assesses the IP to have complete data (score 7.6), the consumer thinks otherwise (score 1)! From the IP characteristics and the data quality assessment results, the corresponding logical and physical design of the IP can be developed with the necessary quality attributes incorporated. Timeliness and credibility are two important data quality dimensions for an IP supporting trading operations. In Figure 1.5, timeliness on share price indicates that the trader is concerned with how old the data is. A special symbol, “√ inspection” is used to signify inspection requirements such as data verification.

Figure 1.5: Information Quality added to the ER Diagram

(Source: Proceedings - 9th International Conference on Data Engineering [33])

The data quality requirements are further refined into more objective and measurable characteristics. As shown in Figure 1.6, these characteristics are depicted as a dotted-rectangle. For example, timeliness is redefined by age (of the data), and credibility of the research report is redefined by analyst name. The quality indicator collection method, associated with the telephone attribute, is included to illustrate that multiple data collection mechanisms can be used for a given type of data. Values for the collection method may include “over the phone” or “from an existing account”.

10

Introduction Chapter 1

The quality indicator media for research report is used to indicate the multiple formats of database-stored documents such as bit-mapped, ASCII or Postscript. The quality indicators derived from “√ inspection” indicate the inspection mechanism desired to maintain data reliability. The specific inspection or control procedures may be identified as part of the application documentation. These procedures might include independent, double entry of important data, front-end rules to enforce domain or update constraints, or manual processes for performing certification of the data.

Figure 1.6: A Quality Entity-Relationship Diagram

(Source: Proceedings - 9th International Conference on Data Engineering [33])

Equally important to the task of identifying data quality dimensions is the identification of the information manufacturing system for the IP. Figure 1.7 illustrates an information manufacturing system which has five data units (DU1 - DU5) supplied by three vendors (VB1 - VB3). Three data units (DU6, DU8, DU10) are formed by having passed through one of the three data quality blocks (QB1 - QB3). For example, DU6 represents the impact of QB1 on DU2 [5].

A Framework for TDQM

11

Figure 1.7: An Illustrative Information Manufacturing System

(Source: Management Science [5])

There are six processing blocks (PB1 - PB6) and, accordingly, six data units (DU7, DU9, DU11, DU12, DU13, DU14) that are the output of these processing blocks. One storage block (SB1) is used both as a pass through block (DU6 enters SB1 and is passed on to PB3) and as the source for data base processing (DU1 and DU8 are jointly processed by PB4). The system has three consumers (CB1 - CB3), each receives some subset of the IP. The placement of a quality block following a vendor block (similar to acceptance sampling) indicates that the data supplied by vendors is deficient with regards to data quality. In the client account database example, identifying such an information manufacturing system would provide the IP team with the basis for assessing the values of data quality dimensions for the IP through the Information Manufacturing Analysis Matrix and for studying options in analyzing and improving the information manufacturing system [5]. The IP definition phase produces two key results: (1) a quality entityrelationship model that defines the IP and its data quality requirements, and (2) an information manufacturing system that describes how the IP is produced, along with the interactions among information suppliers (vendors), manufacturers, consumers, and IP managers. With these results from the IP definition phase, an organization has two alternatives. First, the organization can develop a new information manufacturing system

12

Introduction Chapter 1

for the IP based on these results. The advantage of this approach is that many data quality requirements can be designed into the new information manufacturing system, resulting in quality-information-by-design analogous to that of quality-bydesign in product manufacturing. Many of the data quality problems associated with a legacy system can also be corrected with the new system. The disadvantage is that it would require more initial investment and significant organizational change. Second, the organization can use these results as guidelines for developing mechanisms to remedy the deficiencies of the existing system. However, as the business environment changes, and with it the information needs of the consumers, a new information manufacturing system will ultimately need to be developed.

Measure IP The key to measurement resides in the development of data quality metics. These data quality metrics can be the basic data quality measures such as data accuracy, timeliness, completeness, and consistency [1]. In our client account database example, data quality metrics may be designed to track, for instance: • The percentage of incorrect client address zip code found in a randomly selected client account (inaccuracy) • An indicator of when client account data were last updated (timeliness or currency for database marketing and regulatory purposes) • The percentage of non-existent accounts or the number of accounts with missing value in the industry-code field (incompleteness) • The number of records that violate referential integrity (consistency) At a more complex level, there are business rules that need to be observed. For example, the total risk exposure of a client should not exceed a certain limit. This exposure needs to be monitored for clients who have many accounts. Conversely, a client who has a very conservative position in one account should be allowed to execute riskier transactions in another account. For these business rules to work, however, the IP team needs to develop a proper account linking method and the associated ontology to make the linkage, There are also information-manufacturing-oriented data quality metrics. For example, the IP team may want to track • Which department made most of the updates in the system last week • How many un-authorized accesses have been attempted (security) • Who collected the raw data for a client account (credibility)

A Framework for TDQM

13

Other data quality metrics may measure the distribution of the data qualityrelated collective knowledge across IP roles. Whatever the nature of the data quality metrics, they are implemented as part of a new information manufacturing system or as add-on utility routines in an existing system. With the data quality metrics, data quality measures can be obtained along various data quality dimensions for analysis.

Analyze IP From the measurement results, the IP team investigates the root cause for potential data quality problems. The methods and tools for performing this task can be simple or complex. In the client account database example, one can introduce dummy accounts into the information manufacturing system to identify sources that cause poor data quality. Other methods include statistical process control (SPC), pattern recognition, and Pareto chart analysis for poor data quality dimensions over time. We illustrate other types of analysis through the case of the Medical Command of the Department of Defense, which developed data quality metrics for the information in their Military Treatment Facilities (MTF). In this particular case [8], an analysis of the assumptions and rationale underlying the data quality metrics was conducted. Some ofthe issues raised were: • What are the targeted payoffs? • How do the data quality metrics link to the factors that are critical to the target payoffs • How representative or comprehensive these data quality metrics are • Whether these data quality metrics are the right set of metrics The target payoffs could be two-fold: (1) the delivery of ever-improving value to patients and other stakeholders, contributing to improved health care quality; and (2) improvement of overall organizational effectiveness, use of resources, and capabilities. It would be important to explicitly articulate the scope of these metrics in terms of the categories of payoffs and their linkages to the critical factors. To provide the best health care for the lowest cost, different types of data are needed, MTF commanders need cost and performance data, Managed Care Support contractors need to measure the quality and cost of their services, and patients need data they can use to know what kind of services they get from different health plans. The types of data needed can fall into several categories: patient, provider, type of care, use rate, outcome, and financial. Based on the targeted payoffs, the critical factors, and the corresponding types of data needed, one can evaluate how representative or comprehensive these data quality metrics are and whether these metrics are the right set of metrics.

14

Introduction

Chapter 1

Improve IP Once the analysis phase is complete, the IP improvement phase can start. The IP team needs to identify key areas for improvement such as: (1) aligning information flow and workflow with infrastructure, and (2) realigning the key characteristics of the IP with business needs. As mentioned earlier, the Information Manufacturing Analysis Matrix [5] is designed for the above purposes. Also, Ballou and Tayi [3] have developed a framework for allocating resources for data quality improvement. Specifically, an integer programming model is used to determine which databases should be chosen to maximize data quality improvement given a constrained budget.

SUMMARY In this chapter, we presented concepts and principles for defining, measuring, analyzing, and improving information products (IP), and briefly described a data quality survey software instrument for data quality assessment. We also introduced the Total Data Quality Management (TDQM) framework and illustrated how it can be applied in practice. The power of the TDQM framework stems from a cumulative multidisciplinary research effort and practice in a wide range of organizations. Fundamental to this framework is the premise that organizations must treat information as a product that moves through an information manufacturing system, much like a physical product moves through a manufacturing system, yet realize the distinctive nature of information products. The TDQM framework has been shown to be particularly effective in improving IP in organizations where top management has a strong commitment to a data quality policy. Organizations of the 21st century must harness the full potential of their data in order to gain a competitive advantage and attain their strategic goals. The TDQM framework has been developed as a step towards meeting this challenge.

BOOK ORGANIZATION The remainder of the book is organized as follows. Chapter 2 presents two seminal research efforts, the Polygen Model and the Attribute-based Model, which capture data quality characteristics by extending the Relational Model. Chapter 3 exhibits a research effort that models data quality requirements by extending the Entity Relationship Model. Chapter 4 introduces a knowledge-based model that provides an overall measure that can be useful in automating the judgment of the quality of data in a complex system that uses data from various sources with varying degrees of quality. Chapter 5 describes an attempt to develop a data quality algebra that can be

Book Organization

15

embedded in an extended Relational Model. These chapters summarize our technically oriented work in the database field within the context of data quality. Other leading research institutions have also been working on issues related to data quality within the framework of database systems. Chapter 6 profiles the MIT Context Interchange project. The European Union’s project on data warehouse quality is presented in Chapter 7. In Chapter 8, we profile the data quality project at Purdue University’s database group. Finally, concluding remarks and future directions are presented in Chapter 9.

References [1] Ballou, D.P. and H.L. Pazer, Modeling Data and Process Quality in Multi-input, Multioutput Information Systems. Management Science, 31(2), 1985, 150-162. [2] Ballou, D.P. and H.L. Pazer, Designing Information Systems to Optimize the AccuracyTimeliness Tradeoff. Information Systems Research, 6( 1), 1995, 51-72. [3] Ballou, D.P. and G.K. Tayi, Methodology for Allocating Resources for Data Quality Enhancement. Communications of the ACM, 32(3), 1989, 320-329. [4] Ballou, D.P. and G.K. Tayi, Enhancing Data Quality in Data Warehouse Environment. Communications of the ACM, 42( 1), 1999, 73-78. [5] Ballou, D.P., R.Y. Wang, H. Pazer and G.K. Tayi, Modeling Information Manufacturing Systems to Determine Information Product Quality. Management Science, 44(4), 1998, 462-484. [6] Bowen, P. (1993). Managing Data Quality in Accounting Information Systems: A Stochastic Clearing System Approach. Unpublished Ph.D. dissertation, University of Tennessee. [7] CEIS. Commander‘s Data Quality Assessment Guide for CEIS Beta Test - Region 3.Proceedings of Corporate Executive Information System. 1216 Stanley Road, Fort Sam Houston, Texas 78234-6127, 1996. [8] Corey, D., L. Cobler, K. Haynes and R. Walker. Data Quality Assurance Activities in the Military Health Services System. Proceedings of the 1996 Conference on Information Quality. Cambridge, MA, pp. 127-153, 1996. [9] CRG, Information Quality Assessment (IQA) Software Tool. Cambridge Research Group, Cambridge, MA, 1997. [10] DQJ (1999). Data Quality Journal. [11] English, L.P., Improving Data Warehouse and Business Information Quality. John Wiley & Sons, New York, NY, 1999.

16

Introduction Chapter 1

[12] Huang, K., Y. Lee and R. Wang, Quality Information and Knowledge. Prentice Hall, Upper Saddle River: N.J., 1999. [13] Huh, Y.U., F.R. Keller, T.C. Redman and A.R. Watkins, Data Quality. Information and Software Technology, 32(8), 1990, 559-565. [14] Huh, Y.U., R.W. Pautke and T.C. Redman. Data Quality Control. Proceedings of ISQE, Juran Institute 7A1-27, 1992. [15] ICM (1999). ICM Group, Inc. [16] Kahn, B.K., D.M. Strong and R.Y. Wang (1999). Information Quality Benchmarks: Product and Service Performance. Communications of the ACM, Forthcoming. [17] Khang, S. W. (1990). An Accounting Model-Based Approach Toward Integration of Multiple Financial Databases (No.WP # CIS-90-15). Sloan School of Management, MIT. [18] Laudon, K.C., Data Quality and Due Process in Large Interorganizational Record Systems. Communications of the ACM, 29(1), 1986, 4-11. [19] Liepins, G.E. and V.R.R. Uppuluri, ed. Data Quality Control: Theory and Pragmatics. D. B. Owen. Vol. 112. 1990, Marcel Dekker, Inc., New York, 360 pages, [20] Morey, R.C., Estimating and Improving the Quality of Information in the MIS. Communications of the ACM, 25(5), 1982, 337-342. [21] Nichols, D.R., A Model of auditor's preliminary evaluations of internal control from audit data. 62(1), 1987, 183-190. [22] Pautke, R.W. and T.C. Redman. Techniques to control and improve quality of data in large databases. Proceedings of Statistics Canada Symposium 90, Canada, pp. 319-333, 1990. [23] Reddy, M.P. and R.Y. Wang. Estimating Data Accuracy in a Federated Database Environment. Proceedings of 6th International Conference, CISMOD (Also in Lecture Notes in Computer Science). Bombay, India, pp. 115-134, 1995, [24] Redman, T.C., ed. Data Quality: Management and Technology.1992, Bantam Books: New York, NY. 308 pages. [25] Redman, T.C., Improve Data Quality for Competitive Advantage. Sloan Management Review, 36(2), 1995, 99-109. [26] Redman, T.C., ed. Data Quality for the Information Age. 1996, Artech House: Boston, MA. 303 pages. [27] Strong, D.M., Y.W. Lee and R.Y. Wang, Data Quality in Context. Communications of the ACM, 40(5), 1997, 103-110. [28] Tayi, G.K.and D. Ballou, Examining Data Quality. Communications of the ACM, 41(2), 1999, 54-57.

Book Organization

17

[29] TDQM (1999). Total Data Quality Management Research Program. [30] TTI (1999). Technology Transfer Institute, Inc. Website. [31] Wand, Y. and R.Y. Wang, Anchoring Data Quality Dimensions in Ontological Foundations. Communications of the ACM, 39(11), 1996, 86-95. [32] Wang, R.Y., A Product Perspective on Total Data Quality Management. Communications of the ACM, 41(2), 1998, 58-65. [33] Wang, R.Y., H.B. Kon and S.E. Madnick. Data Quality Requirements Analysis and Modeling. Proceedings of the 9th International Conference on Data Engineering. Vienna: pp. 670-677, 1993. [34] Wang, R.Y., V.C. Storey and C.P. Firth, A Framework for Analysis of Data Quality Research. IEEE Transactions on Knowledge and Data Engineering, 7(4), 1995, 623-640. [35] Wang, R.Y. and D.M. Strong, Beyond Accuracy: What Data Quality Means to Data Consumers. Journal of Management Information Systems, 12(4), 1996, 5-34. [36] Yu, S. and J. Neter, A Stochastic Model of the Internal Control System. Journal of Accounting Research, 1(3), 1973, 273-295.

This page intentionally left blank.

C H A P T E R

2

Extending the Relational Model to Capture Data Quality Attributes

Current commercial relational database management systems (e.g., IBM’s DB2, Oracle’s Oracle DBMS, and Microsoft’s SQL Server) and their underlying relational model are based on the assumption that data stored in the databases are correct. This assumption, however, has some nontrivial ramifications. Consider a join operation in a SQL query. Suppose that the data used in a join operation are incorrect, it would follow that the query results would most likely be incorrect. To what extent the query results would be incorrect and what their impact would be remains an open question, although researchers have started to investigate such issues [1, 22]. The fundamental assumption of the relational model, that “data stored in the underlying databases are correct,” is not without merit. To ensure data integrity, the relational model has facilities such as data dictionaries, integrity rules, and edits checks. In practice, however, dirty data pervade databases for various reasons [14,

20

Extending the Relational Model to Capture Data Quality Attributes

Chapter 2

26]. Furthermore, the scope of data quality goes beyond accuracy and integrity as conceived by many in the database community. It is well established that other aspects of data quality such as believability and timeliness are equally, if not more, important from the end-user’s perspective [2, 3, 15, 24, 27]. In this chapter, we present two early attempts at extending the relational model to capture data quality attributes: The Polygen Model [31]1 and the Attribute-based Model [25]2. We analyze these models from three aspects following the relational model [8, 11, p. 181]: (1) data structure (object types), (2) data integrity (integrity rules), and (3) data manipulation (operators). The polygen model is a seminal research that precipitated the downstream work at the MIT TDQM program and Context Interchange research, such as [18-21]. The idea came from the realization that end-users needed to know where the data came from (i.e., data sources) in order to assess the quality of the data (credibility of the data source, for example) for the tasks at hand. The attribute-based model is a follow-up research that defines a formal data structure and the corresponding algebra and integrity rules to include the quality aspects of data within the framework of the relational model.

THE POLYGEN MODEL Heterogeneous (distributed) database systems strive to encapsulate the heterogeneity of the underlying databases, for example, producing an illusion that all data resides in a single location, which is referred to as location transparency. Indeed, a set of transparency rules for distributed database systems has been proposed [23]. In contrast, the polygen model studies heterogeneous database systems from the multiple (poly) source (gen) perspective. Most end-users wish to know the source of their data (e.g., “Source: Reuters, Thursday, August 17, 2000”). This source knowledge may be important to them for many reasons. For example, it enables them to apply their own judgment to the credibility of the data. We call this the Data Source Tagging problem. A decision maker may need to know not only the sources of data but also the intermediate sources that helped in composing the data. We call this the Intermediate Source Tagging problem. By resolving the Data Source Tagging and Intermediate Source Tagging problems, the polygen model addresses critical issues such as "where did the data come from" and "which intermediate data sources were used to derive the data." In the context of heterogeneous database systems, a data source means the underlying database in which the data 1 Adapted from the Proceedings of the 16th International Conference on Very Large Data Bases, Richard Wang and Stuart Madnick, A Polygen Model for Heterogeneous Database Systems: The Source Tagging Perspective, pp.5 19-538, 1990, Brisbane, Australia. 2 Reprinted from Decision Support Systems, 13, Richard Wang, M. P. Reddy, and Henry Kon, Toward quality data: An attribute-based approach, 349-372, 1995, with permission from Elsevier Science.

The Polygen Model

21

reside. By the same token, an intermediate data source means an intermediate database that contributes to a final query result.

Architecture A query processor architecture for the polygen model is depicted in Figure 2.1. Briefly, the Application Query Processor translates an end-user query into a polygen query for the Polygen Query Processor (PQP) based on the user's application schema. The PQP in turn translates the polygen query into a set of local queries based on the corresponding polygen schema, and routes them to the Local Query Processors (LQP). The details of the mapping and communication mechanisms between an LQP and its local databases are encapsulated in the LQP. To the PQP, each LQP behaves as a local relational system. Upon return from the LQPs, the retrieved data are further processed by the PQP in order to produce the desired composite data.

Figure 2.1: The Query Processor Architecture

22

Extending the Relational Model to Capture Data Quality Attributes

Chapter 2

Many critical problems need to be resolved in order to provide a seamless solution to the end-user. These problems include source tagging, query translation, schema integration [4, 13], inter-database instance matching [30], domain mapping [12, 29], and semantic reconciliation [28]. We focus on the first problem - source tagging - and make the following assumptions: • The local schemata and the polygen schema are all based on the relational model. • Sources are tagged after data has been retrieved from each database. • Schema integration has been performed, and the attribute mapping information is stored in the polygen schema. • The inter-database instance identifier mismatching problem (e.g., IBM vs. 1.B.M or social security identification number vs. employee identification number) has been resolved and the data is available for the PQP to use. • The domain mismatch problem such as unit ($ vs. ¥), scale (in billions vs. in millions), and description interpretation (“expensive” vs. “$$$”,“Chinese Cuisine” vs. “Hunan or Cantonese”) has been resolved in the schema integration phase and the domain mapping information is also available to the PQP.

Polygen Data Structure Let PA be a polygen attribute in a polygen scheme P, LS a local scheme in a local database LD, and LA a local attribute in LS. Let MA be the set of local attributes corresponding to a PA, i.e., MA = {(LD, LS,LA)|(LD,LS, LA) denotes a local attribute of the corresponding PA}. A polygen scheme P is defined as P = ((PA1, MA1), , , . , (PAn, Man)> where n is the number ofattributes in P. A polygen schema is defined as a set {PI, . . . , PN) of N polygen schemes. A polygen domain is defined as a set of ordered triplets. Each triplet consists of three elements: the first is a datum drawn from a simple domain in an LQP. The second is a set of LDs denoting the local databases from which the datum originates. The third is a set of LDs denoting the intermediate local databases whose data led to the selection of the datum. A polygen relation p of degree n is a finite set of time-varying n-tuples, each n-tuple having the same set of attributes drawing values from the corresponding polygen domains. A cell in a polygen relation is an ordered triplet c=(c〈d〉, c〈o〉, c〈i〉) where c〈d〉 denotes the datum portion, c〈o〉 the originating portion, and c〈i〉 the intermediate source portion. Two polygen relations are union-compatible if their corresponding attributes are defined on the same polygen domain.

The Polygen Model

23

Note that P contains the mapping information between a polygen scheme and the corresponding local relational schemes. In contrast, p contains the actual timevarying data and their originating sources. Occasionally, a polygen scheme and a polygen relation may be used synonymously without confusion. The data and intermediate source tags for p are updated along the way as polygen algebraic operations are performed.

Polygen Algebra Let attrs(p) denote the set of attributes of p. For each tuple t in a polygen relation p, let t〈d〉 denote the data portion, t〈o〉 the originating source portion, and t〈i〉 the intermediate source portion. If x E attrs(p) and X = {x 1..., xj, ...,xJ} is a sublist of attrs(p), then let p[x] be the column in p corresponding to attribute x, let p[x] be the columns in p corresponding to the sublist of attributes X, let t[x] be the cell in t corresponding to attribute x, and let t[X] be the cells in t corresponding to the sublist of attributes X. As such, p[x] 〈o〉 denotes the originating source portion of the column corresponding to attribute x in polygen relation p while t[X] 〈i〉 denotes the intermediate source portion of the cells corresponding to the sublist of attributes X in tuple t. On the other hand, p[x] denotes the column corresponding to attribute x in polygen relation p inclusive of the data, originating source, and intermediate source portions; while t[X] denotes the cells corresponding to the sublist of attributes X in tuple t inclusive ofthe data, originating source, and intermediate source portions. The five orthogonal algebraic primitive operators [5-9, 17] in the polygen model are defined as follows:

Projection If p is a polygen relation, and X = {x1. . .,xj,. . .,xJ} is a sublist ofattrs(p), then p[X] = {t'|t' = t[X] if t ∈p t[X]〈d〉 is unique; t'〈d〉 = ti [X]〈d〉, t'[xj]〈o〉= ti[xj]〈o〉 ∪. . . ∪ tk[xj]〈o〉 xj ∈ X, t'[xj]〈i〉= ti[xj]〈i〉 . . . ∪ tk[xj]〈i〉 ∀ xj ∈ X, IF ti , ... , tk ∈ p ti[X]〈d〉=. . .= tk[X]〈d〉}.

Cartesian product If p1 and p2 are two polygen relations, then (p1 x p2 ) = {t1 o t2 | t1 ∈ p1 and t2 ∈ p2, where o denotes concatenation}.

Restriction If p is a polygen relation, x ∈ attrs(p), y ∈ attrs(p), and θ is a binary relation, then p[x θ y]={t'|t'〈d〉=t〈d〉, t'〈o〉=t〈o〉, t'[w] 〈i〉 ∪ t[x] 〈o〉 ∪ t[y] 〈o〉 ∀ w ∈ attrs(p),

24

Extending the Relational Model to Capture Data Quality Attributes

IF t ∈ p

Chapter 2

t[x] 〈d〉 θ t[y]〈d〉}.

Union If p1 and p2 are two polygen relations of degree n, t1 ∈p1, and t2 ∈p2, then (p1 ∪ p2) = {t'|t'=t1 IF t1〈d〉 ∈ p1 t1〈d〉 ∉ p2; t'=t2 IF t2〈d〉 ∉ p1 t2〈d〉 ∈ p2; t'〈d〉 = t1〈d〉, t'〈o〉 = t1〈o〉 t2〈o〉, t'〈i〉 ∪ t2〈i〉 if t1〈d〉 = t2〈d〉}

Difference Let p〈o〉 denote the union of all the t〈o〉 sets in p, and p〈i〉 denote the union of all the t〈i〉 sets in p. If p1 and p2 are two polygen relations of degree n, then (p1-p2)={t'| t'〈d〉=t〈d〉, t'〈o〉=t〈o〉, t'[w] 〈i〉 ∪ p2〈o〉 ∪ p2〈i〉 ∀ w ∈ attrs(p), IF t ∈p1 t〈d〉∉p2}. The intermediate source portion t〈i〉 is updated by the Restriction and Difference operations. The Restriction operation selects the tuples in a polygen relation which satisfies the [x θ y] condition. As such, the originating local databases of the x and y attribute values are added to the t〈i〉 set in order to signify their mediating role. Since Select and Join are defined through Restriction, they also update t〈i〉. The Difference operation selects a tuple in p1 to be a tuple in (p1 – p2) if the data portion of the tuple in p1 is not identical to those of the tuples in p2. Since each tuple in p1 needs to be compared with all the tuples in p2, it follows that all the originating sources of the data in p2 should be included in the intermediate source set of (p1–p2) as is denoted by t'〈i〉 = t〈i〉 ∪ p2〈o〉 ∪ p2〈i〉. In contrast, the Projection, Cartesian Product, and Union operations do not involve intermediate local databases as the mediating sources. Other traditional operators can be defined in terms of the above five operators. The most common are Join, Select, and Intersection. Join and Select are defined as the restriction of a Cartesian product. Intersection is defined as the Projection of a join over all the attributes in each of the relations involved in the Intersection. In order to process a polygen query, we also need to introduce the following new operators to the polygen model: Retrieve, Coalesce, Outer Natural Primary Join, Outer Natural Total Join, andMerge. A local database relation needs to be retrieved from a local database to the PQP first before it is considered as a PQP base relation. This is required in the polygen model because a polygen operation may require data from multiple local databases. Although a PQP base relation can be materialized dynamically like a view in the conventional database system, for conceptual purposes, we define it to reside physically in the PQP. The Retrieve operation can be defined as an LQP Restriction operation without any restriction condition. The Coalesce and Outer Natural Join operations have been informally introduced by Date to handle a surprising number of practical applications. Coalesce

The Attribute-based Model

25

takes two columns as input, and coalesces them into one column. An Outer Natural Join is an outer join with the join attributes coalesced [ 10]. An Outer Natural Primary Join is defined as an Outer Natural Join on the primary key of a polygen relation. An Outer Natural Total Join is an Outer Natural Primary Join with all the other polygen attributes in the polygen relation coalesced as well, Merge extends Outer Natural Total Join to include more than two polygen relations. It can be shown that the order in which Outer Natural Total Join operations are performed over a set of polygen relations in a Merge is immaterial. Since Coalesce can be used in conjunction with the other polygen algebraic operators to define the Outer Natural Primary Join, Outer Natural Total Join, and Merge operations, it is defined as the sixth orthogonal primitive of the polygen model.

Coalesce Let © denote the coalesce operator. If p is a polygen relation, x ∈ attrs(p), y ∈ attrs(p), z = attrs(p) - {x,y}, and w is the coalesced attribute of x and y, then p[x  y:w] = {t'| t'[z]=t[z], t'[w] 〈d〉, t'[w] 〈o〉=t[x] 〈o〉 ∪ t[y] 〈o〉, t'[w] 〈i〉 = t[x] 〈i〉 ∪ t[y] 〈i〉, IF t[x] 〈d〉=t[y] 〈d〉; t'[z]=t[z], t'[w] 〈d〉=t[x] 〈d〉, t'[w] 〈o〉=t[x] 〈o〉, t'[w] 〈i〉=t[x] 〈i〉, IF t[y] 〈d〉=nil; t'[zl=t[z], t'[w] 〈d〉=t[y]〈d〉,t’[w]〈o〉=t[y]〈o〉, t'[w] 〈i〉=t[y]〈i〉, IF t[x] 〈d〉=nil}. Note that in a heterogeneous distributed environment, the values to be coalesced may be inconsistent. It is assumed that inter-database instance mismatching problems will be resolved before the Coalesce operation is performed [30, 33]. We have presented the polygen model and the polygen algebra. The reader is referred to Wang and Madnick [31, 32] for a scenario that exemplifies the polygen model, a detailed presentation of a data-driven query translation mechanism for dynamically mapping a polygen query into a set of local queries, and an example that illustrates polygen query processing.

THE ATTRIBUTE-BASED MODEL The attribute-based data model facilitates cell-level tagging of data. It includes a definition that extends the relational model, a set of quality integrity rules, a quality indicator algebra that can be used to process SQL queries that are augmented with quality indicator requirements. From these quality indicators, the user can make a better interpretation of the data and determine the believability of the data. Given that zero defect data may not be necessary or attainable, it would be useful to be able to judge the quality of data. This suggests that we tag data with quality

26

Extending the Relational Model to Capture Data Quality Attributes

Chapter 2

indicators, which are characteristics of the data. From these quality indicators, the user can make a judgment of the quality of the data for the specific application at hand. For example, in making a financial decision to purchase stocks, it would be useful to know the quality of data through quality indicators such as who originated the data, when and how the data was collected.

Data structure An attribute may have an arbitrary number ofunderlying levels of quality indicators. In order to associate an attribute with its immediate quality indicators, a mechanism must be developed to facilitate the linkage between the two, as well as between a quality indicator and the set of quality indicators associated with it. This mechanism is developed through the quality key concept. In extending the relational model, Codd made clear the need for uniquely identifying tuples through a system-wide unique identifier, called the tuple ID [7, 16]. This concept is applied in the attributebased model to enable such linkage. Specifically, an attribute in a relation scheme is expanded into an ordered pair, called a quality attribute, consisting of the attribute and a quality key. This expanded scheme is referred to as a quality scheme. Correspondingly, each cell in a relational tuple is expanded into an ordered pair, called a quality cell, consisting of an attribute value and a quality key value. This expanded tuple is referred to as a quality tuple and the resulting relation is referred to as a quality relation. Each quality key value in a quality cell refers to the set of quality indicator values immediately associated with the attribute value. The quality indicator values are grouped together to form a kind of quality tuple called a quality indicator tuple. A quality relation composed of a set of these time-varying quality indicator tuples is called a quality indicator relation. The quality scheme that defines the quality indicator relation is referred to as the quality indicator scheme. The quality key thus serves as a foreign key that relates an attribute (or quality indicator) value to its associated quality indicator tuple. A quality scheme set is defined as the collection of a quality scheme and all the quality indicator schemes that are associated with it. We define a quality database as a database that stores not only data but also quality indicators. A quality schema is defined as a set of quality scheme sets that describes the structure of a quality database. Figure 2.2 illustrates the relationship among quality schemes, quality indicator schemes, quality scheme sets, and the quality schema.

The Attribute-based Model

Figure 2.2:

27

Quality schemes, quality indicator schemes, quality scheme sets, and the quality schema

Ifqr1 is a quality relation and a an attribute in qr1 that has quality indicators associated with it, then the quality key for a must be non-null. Ifqr2 is a quality indicator relation containing a quality indicator tuple for a, then all the attributes ofqr2 are called level 1 quality indicators for a. Each attribute in qr2 can have a quality indicator relation associated with it. An attribute can have n levels of quality indicator relations associated with it (n ≥ 0). Following the constructs developed in the relational model, a domain is defined as a set of values of similar type. Let ID be the domain for a system-wide unique identifier. Let D be a domain for an attribute. Let DID be defined on the Cartesian product D X ID. Let id be a quality key value associated with an attribute value d where d ∈D and id ∈ID. A quality relation (qr) of degree m is defined on the m+1 domains if it is a subset ofthe Cartesian product: ID X DID, X DID2 X ... X DIDm. Let qt be a quality tuple, which is an element in a quality relation. Then a quality relation qr is designated as: qr = {qt| qt = where id ∈ID, didj ∈DIDj, j = 1, ... ,m}

Data manipulation In order to present the attribute-based algebra formally, we first define two key concepts that are fundamental to the quality indicator algebra: QI-compatibility and QIVEqual.

28

Extending the Relational Model to Capture Data Quality Attributes

Chapter 2

QI-Compatibility and QIV-Equal Let a1 and a2 be two application attributes. Let QI(ai) denote the set ofquality indicators associated with ai. Let S be a set of quality indicators. If S C QI(a1) and S C QI(a2), then a1 and a2 are defined to be QI-Compatible with respect to S.3 For example, if S = {qi1, qi2, qi21}, then the attributes a1 and a2 shown in Figure 2.3 are QICompatible with respect to S. Whereas if S = {qi1, qi22}, then the attributes a1 and a2 shown in Figure 2.3 are not QI-Compatible with respect to S.

Figure 2.3: QI-Compatibility Example

Let a1 and a2 be QI-Compatible with respect to S. Let w1 and w2 be values of a1 and a2 respectively. Let qi(w1) be the value of quality indicator qi for the attribute value w1 where qi e S (qi2(w1) = v2 in Figure 2.4). Define w1 and w2 to be QIVEqual with respect to S provided that qi(w1) = qi(w2) ∀ qi ∈ S, denoted as w1 =sw2. In Figure 2.4, for example, w1 and w2 are QIV-Equal with respect to S = (qi1, qi21}, but not QIV-Equal with respect to S = {qi1, qi31} because qi 31(w1) = v31 whereas qi31(w2) = x31.

Figure 2.4: QIV-Equal Example

3It is assumed that the numeric subscripts (e.g., qi ) map the quality indicators to unique positions in the 11 quality indicator tree.

The Attribute-based Model

29

In practice, it is tedious to explicitly state all the quality indicators to be compared (i.e., to specify all the elements of s). To alleviate this situation, we introduce i-level QI-compatibility (i-level QIV-Equal) as a special case for QI-compatibility (QIV-equal) in which all the quality indicators up to a certain level of depth in a quality indicator tree are considered. Let a1 and a2 be two application attributes. Let a1 and a2 be QI-Compatible with respect to S. Let w1 and w2 be values of a1 and a2 respectively, then w1 and w2 are defined to be i-level QI-Compatible if the following two conditions are satisfied: (1) a1 and a 2 are QI-Compatible with respect to S, and (2) S consists of all quality indicators present within i levels ofthe quality indicator tree of a1 (thus ofa2). By the same token, i-level QIV-Equal between w1 and w2, denoted by w1 =i w2, can be defined. If ‘i’ is the maximum level of depth in the quality indicator tree, then a1 and a2 are defined to be maximum-level QI-Compatible. Similarly, maximumlevel QIV-Equal between w1 and w2, denoted by w1 = mw2, can also be defined. An example using the algebraic operations in the quality indicator algebra is given in Wang, Reddy, and Kon [25]. In order to illustrate the relationship between the quality indicator algebraic operations and the high-level user query, the SELECT, FROM, WHERE structure of SQL is extended with an extra clause ‘’with QUALITY.” This extra clause enables a user to specify the quality requirements regarding an attribute referred to in a query. If the clause “with QUALITY” is absent in a user query, then it means that the user has no explicit constraints on the quality of data that is being retrieved. In that case, quality indicator values would not be compared in the retrieval process; however, the quality indicator values associated with the applications data would be retrieved.

Quality Indicator Algebra Following the relational algebra, we define the five orthogonal quality relational algebraic operations, namely Selection, Projection, Union, Difference, and Cartesian product [17]. In the following operations, let QR and QS be two quality schemes and qr and qs be two quality relations associated with QR and QS respectively. Let a and b be two attributes in both QR and QS. Let t1 and t 2 be two quality tuples and Sa be a set of quality indicators specified by the user for the attribute a; in other words, Sa is constructed from the specifications given by the user in the “with QUALITY” clause. Let the term t1.a = t2.a denote that the values ofthe attribute a in the tuples t1 S and t2 are identical. Let t1.a = a t2.a denote that the values ofattribute a in the tuples t1 and t2 are QIV-equal with respect to Sa. In a similar fashion, let t1.a =it2.a and m t1.a = t2.a denote i-level QIV-equal and maximum-level QIV-equal between the values oft1.a and t2.a respectively.

30

Extending the Relational Model to Capture Data Quality Attributes

Chapter 2

Selection Selection is a unary operation that selects only a horizontal subset of a quality relation (and its corresponding quality indicator relations) based on the conditions specified in the Selection operation. There are two types of conditions in the Selection operation: regular conditions for an application attribute and quality conditions for the quality indicator relations corresponding to the application attribute. The selection, σqC is defined as follows:

σqC (qr) = {t| ∀ t1 ∈qr, ∀ a ∈QR, ((t.a = t1.a) (t.a = m t1.a)) C(t1)} where C(t1) = e1 Φ e2 Φ ... Φ en Φ e1q Φ ... Φ epq; ei is in one of the forms: (t1.a θ constant) or (t1.a θ t1.b); eiq is in one of the forms (qik = constant) or (t1.a =Sa,b t1.b) or (t1.a =i t1.b) or (t1.a =m t1.b); qik ∈ QI(a); Φ∈{ ,∨, }; θ = {= ≤, ≥, ≤, <, >,=}; and Sa,b is the set of quality indicators used duing the compari son of t1.a and t1.b.

Projection Projection is a unary operation that selects a vertical subset of a quality relation based on the set of attributes specified in the Projection operation. The result includes the projected quality relation and the corresponding quality indicator relations that are associated with the set of attributes specified in the Projection operation. Let PJ be the attribute set specified, then the Projection ΠqPJ (qr) is defined as follows:

ΠqPJ (qr) = {t | ∀ t1 ∈ qr, ∀a PJ, ((t.a = t1•a) (t.a =m t1•a))}

Union In this operation, the two operand quality relations must be QI-Compatible. The result includes (1) tuples from both qr and qs after elimination of duplicates, and (2) the corresponding quality indicator relations that are associated with the resulting tuples. qr ∪q qs = qr { t | ∀ t2 ∈ qs, ∃t1 ∈ qr, ∀ a ∈ QR, ((t.a = t2.a) (t.a =m t2.a) ((t1.a=t2.a) (t1.a =Sa t2.a)))} In the above expression, " ((t1.a = t2.a) (t1.a=Sat2.a))" is meant to eliminate duplicates. Tuples t1 and t 2 are considered duplicates provided that (1) there is a match between their corresponding attribute values (i.e., t1.a = t 2.a ) and (2) these values are QIV-equal with respect to the set of quality indicators (Sa) specified by the user(i.e., t1.a =Sat2.a).

The Attribute-based Model

31

Difference In this operation, the two operand quality relations must be QI-Compatible. The result of the Difference operation consists of all tuples from qr which are not equal to tuples in qs. During this equality test, the quality of attributes specified by the user for each attribute value in the tuples t1 and t2 will also be taken into consideration. qr–q qs = { t |∀ t1 ∈qr, ∃t2 ∈ qs, ((t1.a = t2.a) (t1.a =Sa t2.a)) )} ∀ a ∈ QR, ((t.a = t1.a) (t.a =m t1.a) Just as in relational algebra, other algebraic operators such as Intersection and Join can be derived from these five orthogonal operators. Below is a list of the many different ways the attribute-based model can be applied: • The ability of the model to support quality indicators at multiple levels makes it possible to retain the origin and intermediate data sources. • A user can filter the data retrieved from a database according to quality requirements. • Data authenticity and believability can be improved by data inspection and certification. A quality indicator value could indicate who inspected or certified the data and when it was inspected. The reputation of the inspector will enhance the believability of the data. • The quality indicators associated with data can help clarify data semantics, which can be used to resolve semantic incompatibility among data items received from different sources. This capability is very useful in an interoperable environment where data in different databases have differentsemantics. • Quality indicators associated with an attribute may facilitate a better interpretation ofnull values. • In a data quality control process, when errors are detected, the data administrator can identify the source of error by examining quality indicators such as data source or collection method.

Data Integrity A fundamental property of the attribute-based model is that an attribute value and its corresponding quality indicator values (including all descendants) are treated as an atomic unit. By atomic unit we mean that whenever an attribute value is created, deleted, retrieved, or modified, its corresponding quality indicators also need to be created, deleted, retrieved, or modified respectively. In other words, an attribute value and its corresponding quality indicator values behave atomically. We refer to

32

Extending the Relational Model to Capture Data Quality Attributes

Chapter 2

this property as the atomicity property hereafter. This property is enforced by a set of quality referential integrity rules as defined below. • Insertion: Insertion of a tuple in a quality relation must ensure that for each non-null quality key present in the tuple (as specified in the quality schema definition), the corresponding quality indicator tuple must be inserted into the child quality indicator relation. For each non-null quality key in the inserted quality indicator tuple, a corresponding quality indicator tuple must be inserted at the next level. This process must be continued recursively until no more insertions are required. • Deletion: Deletion of a tuple in a quality relation must ensure that for each non-null quality key present in the tuple, corresponding quality data must be deleted from the table corresponding to the quality key. This process must be continued recursively until a tuple is encountered with all null quality keys. • Modification: If an attribute value is modified in a quality relation, then the descendant quality indicator values of that attribute must be modified.

CONCLUSION Both of the models presented in this chapter define data quality by extending the relational model. The Polygen Model resolves the data source tagging and intermediate source tagging problems. It addresses issues in heterogeneous distributed database systems from the “where” perspective and thus enables us to interpret data from different sources more accurately. Furthermore, it follows the relational model by specifying the data structure and data manipulation components of the data model. However, it does not include the data integrity component. The Attribute-based Model, on the other hand, allows for the structure, storage, and processing of quality relations and quality indicator relations through a quality indicator algebra. In addition, it includes a description of its data structure, a set of data integrity constraints for the model, and a quality indicator algebra.

Conclusion

33

References [1] Ballou, D., I. Chengalur-Smith and R. Y. Wang , A Sampling Procedure for Data Quality Auditing in the Relational Environment (No. TDQM-00-02). Massachusetts Institute of Technology, 2000. [2] Ballou, D. P. and H. L. Pazer, "Designing Information Systems to Optimize the Accuracy-Timeliness Tradeoff," Information Systems Research, 6(1), 1995, pp. 51-72. [3] Ballou, D. P., R. Y. Wang, H. Pazer and G. K. Tayi, "Modeling Information Manufacturing Systems to Determine Information Product Quality," Management Science, 44(4), 1998, pp. 462-484. [4] Batini, C., M. Lenzirini and S. Navathe, "A comparative analysis of methodologies for database schema integration,'' ACM Computing Survey, 18(4), 1986, pp. 323-364. [5] Codd, E. F., "A Relational Model of Data for Large Shared Data Banks," Communications of the ACM, 13(6), 1970, pp. 377-387. [6] Codd, E. F., "Relational completeness of data base sublanguages," in Data Base Systems, R. Rustin, Editor 1972, Prentice Hall, 1972. [7] Codd, E. F., "Extending the relational database model to capture more meaning," ACM Transactions on Database Systems, 4(4), 1979, pp. 397-434. [8] Codd, E. F., "Relational database: A Practical Foundation for Productivity, the 1981 ACM Turing Award Lecture," Communications of the ACM, 25(2), 1982, pp. 109-117. [9] Codd, E. F. "An evaluation scheme for database management systems that are claimed to be relational," in Proceedings of the Second International Conference on Data Engineering. Los Angeles, CA: pp. 720-729, 1986. [10] Date, C. J. "The outer join," in Proceedings of The 2cd International Conference on Databases. Cambridge, England: pp. 76-106, 1983. [11] Date, C. J., An Introduction to Database Systems. 5th ed. Addison-Wesley Systems Programming Series, Addison-Wesley, Reading, 1990. [12] DeMichiel, L. G. Performing operations over mismatched domains. In pp. 36-45. Los Angeles, CA: 1989. [13] Elmasri, R., J. Larson and S. Navathe, Schema integration algorithms for federated databases and logical database design (No. 1987 [14] Huang, K., Y. Lee and R. Wang, Quality Information and Knowledge. Prentice Hall, Upper Saddle River: N.J., 1999. [15] Kahn, B. K., D. M. Strong and R. Y. Wang. "Information Quality Benchmarks: Product and Service Performance," Communications of the ACM, 1999.

34

Extending the Relational Model to Capture Data Quality Attributes

Chapter 2

[16] Khoshafian, S. N. and G. P. Copeland, "Object Identity," in The Morgan Kaufmann Series in Data Management Systems, S. B. Zdonik and D. Maier, Ed. 1990, Morgan Kaufmann, San Mateo, CA, 1990. [17] Klug, A., "Equivalence of relational algebra and relational calculus query languages having aggregate functions," The Journal of ACM, 29, 1982, pp. 699-717. [18] Lee, T. and S. Bressen. "Multimodal Integration of Disparate Information Sources with Attribution," in Proceedings of Entity Relationship Workshop on Information Retrieval and Conceptual Modeling, 1997. [19] Lee, T., S. Bressen and S. Madnick. "Source Attribution for Querying Against Semistructured Documents,". in Proceedings of Workshop on Web Information and Data Management, ACM Conference on Information and Knowledge Management, 1998. [20] Lee, T. and L. McKnight. "Internet Data Management: Policy Barriers to an Intermediated Electronic Market in Data," in Proceedings of 27th Annual Telecommunications Policy Research Conference 1999. [21] Madnick, S. E., "Database in the Internet Age," Database Programming and Design, 1997, pp. 28-33. [22] Reddy, M. P. and R. Y. Wang. "Estimating Data Accuracy in a Federated Database Environment," in Proceedings of 6th International Conference, CISMOD (Also in Lecture Notes in Computer Science). Bombay, India: pp. 115-134, 1995. [23] Rob, P. and C. Coronel, Database Systems: Design, Implementation, and Management. 3rd ed. Course Technology, Boston, 1997. [24] Strong, D. M., Y. W. Lee and R. Y. Wang, "Data Quality in Context," Communications of the ACM, 40(5), 1997, pp. 103-110. [25] Wang, R. Y., M. P. Reddy and H. B. Kon, "Toward quality data: An attribute-based approach," Decision Support Systems (DSS), 13, 1995, pp. 349-372. [26] Wang, R. Y., V. C. Storey and C. P. Firth, "A Framework for Analysis of Data Quality Research," IEEE Transactions on Knowledge and Data Engineering, 7(4), 1995, pp. 623640. [27] Wang, R. Y. and D. M. Strong, "Beyond Accuracy: What Data Quality Means to Data Consumers," Journal of Management Information Systems (MIS), 12(4), 1996, pp. 5-34. [28] Wang, Y. R. and S. E. Madnick, Connectivity among information systems. Vol. 1. Cornposite Information Systems (CIS) Project, MIT Sloan School of Management, Cambridge, MA, 1988. [29] Wang, Y. R. and S. E. Madnick, "Facilitating connectivity in composite information systems," ACM Data Base, 20(3), 1989, pp. 38-46.

Conclusion

35

[30] Wang, Y. R. and S. E. Madnick. "The inter-database instance identification problem in integrating autonomous systems," in Proceedings of the Fifth International Conference on Data Engineering. Los Angeles, CA: pp. 46-55, 1989. [31] Wang, Y. R. and S. E. Madnick. "A Polygen Model for Heterogeneous Database Systems: The Source Tagging Perspective,". in Proceedings of the 16th International Conference on Very Large Data bases (VLDB). Brisbane, Australia: pp. 519-538, 1990. [32] Wang, Y. R. and S. E. Madnick. "A Source Tagging Theory for Heterogeneous Database Systems," in Proceedings of International Conference on Information Systems. Copenhagen, Denmark: pp. 243-256, 1990. [33] Yuan, Y. , The design and implementation of system P: A polygen database management system (No. CIS-90-07). MIT Sloan School of Management, Cambridge, MA 02139, 1990

This page intentionally left blank.

C H A P T E R

3

Extending the ER Model to Represent Data Quality Requirements

In the previous chapter, we described how data quality can be incorporated into the relational model. To achieve dataquality-by-design, it would be useful to incorporate data quality attributes at the conceptual design stage of a database application. Conventional conceptual data models and their corresponding design methodologies, however, have been developed to capture entities, relationships, attributes, and other advanced concepts such as is-a and component-of relationships. Data quality is not explicitly recognized. The task of incorporating data quality into the design of a database application is left to the designer. Consequently, some database applications capture data quality attributes whereas others don’t. Indeed, Chen [ 1] stated that: “Mostdata models (including the ER model) assume that all data given is correct. We need a scientific way to assess the quality of the data in (or to be put into) the database. We also need to incorporate such quality

38

Extending the ER Model to Represent Data Quality Requirements

Chapter 3

measures into the Entity-Relationship (ER) model as well as to develop algebra- or calculus-based operations to manipulate such data quality descriptions.” This chapter presents a Quality Entity Relationship (QER) model4 proposed by Storey and Wang [6] that incorporates data quality requirements into the database design process. The procedure for doing so involves differentiating application quality requirements anddata quality requirements from application requirements, and then developing the constructs and procedures necessary to represent the data quality requirements in an ER.

MOTIVATING EXAMPLE Consider a training database used to keep track of employee training for career development purposes. Data is needed on courses and the vendors who supply them, students who are enrolled and those who are wait-listed to take the classes, job skills, as well as course evaluations, proficiency levels, and other information. A course can be offered by different vendors, as shown in the ER model in Figure 3.1. The min/max cardinalities [8] indicate that, for each Course, there can be zero-tomany vendors who offer the course and one-to-many courses offered by each vendor. This conceptual model represents the traditional approach to database design, but does not capture quality requirements such as the reputation of the vendor or the standards for the course.

Figure 3.1 : A training database example

4Adapted from the Proceedings of the 1998 Conference on Information Quality. Storey, V. C. and R. Y. Wang. “An Analysis of Quality Requirements in Database Design,” pp. 64-87, with permission from the MIT Total Data Quality Management (TDQM) Program.

Quality Requirements Identification

39

Traditional user requirements as well as user quality requirements must be represented in the database design. To do so, user requirements are divided into three categories: • Application Requirements: Widely accepted and practiced. Traditional application requirements analysis identifies data items such as employees and their ranks and salaries, or departments and their names and locations, that are fundamental to a database application [7]. These are the requirements that are usually discussed and represented when designing a database. • Application Quality Requirements Application quality requirements may arise at the input, process, and output stages associated with the production ofa product [2-4]. Such requirements include measurements against standards and comparisons of final products to initial design specifications. Traditionally, these have not been recognized or captured during the design process. • Data Quality Requirements: Data quality requirements correspond to the quality of the actual data in the database. They can be associated with either application requirements or application quality requirements. As previously mentioned, these requirements are multidimensional in nature. In the training example, Course and Instructor and their relationships are application requirements. One may want to know what standards a course has. This is an application quality requirement. An example of a data quality requirement is the accuracy of the class attendance data. There are various reasons why the three types of requirements are separated. First, quality requirements (both application quality and data quality requirements) differ from application requirements. Since they are often overlooked, this forces the user to identify them specifically during requirements analysis. Second, most existing systems were not built with quality requirements in mind. Therefore, the requirements can be elicited separately and added without requiring a completely new design. This is important for legacy systems. Finally, there is subjectivity in deciding what quality items to include and what measurement scales to use.

QUALITY REQUIREMENT IDENTIFICATION The identification of quality requirements must begin at the requirements analysis phase because quality requirements are both application and user-dependent. During the identification ofapplication requirements, the designer collects requirements and expresses them in English sentences [1]. Application quality requirements and data quality requirements can be identified in a similar manner; that is, by the de-

40

Extending the ER Model to Represent Data Quality Requirements

Chapter 3

signer working with the user to elicit the requirements and possibly suggesting some quality requirements based upon the designer's experience with similar or relatedapplicationdomains. Wang and Strong [12] 0provide a framework that categorizes data quality into four main categories: 1) intrinsic data quality (believability, accuracy, objectivity, reputation), 2) contextual data quality (value-added, relevancy, timeliness, completeness, appropriate amount ofdata), 3), representation data quality (interpretability, ease of understanding, representational consistency, concise representation), and 4) accessibility data quality (accessibility, access security). A designer, possibly with the assistance of an automated design tool, could use this framework during requirements elicitation as a checklist of data quality items, drilling down as appropriate to the underlying elements. Alternatively, the system could use them to try to verify that the user had chosen appropriate data quality measures.

REQUIREMENTS MODELING The quality design process is summarized in Figure 3.2. Since the identification of traditional application requirements is well understood, it is the logical foundation upon which to analyze application quality requirements. The acquisition of quality requirements is shown as Steps 1 and 2. Since both application requirements and application quality requirements will ultimately be captured and stored as data in a database management system, a mechanism must be available to measure the quality of the data that are stored to represent these two types of requirements. This is the task of Step 3: identifying data quality requirements.

Figure 3.2: Quality Requirements Modeling Process

Modeling Data Quality Requirements

41

The arrow from Step 1 to Step 2 signifies that application requirements identification precedes application quality requirements identification. The arrows from Steps 1-2 to Step 3 indicate that application and application quality requirements are modeled before data quality requirements. As in conventional database design and software engineering practices, these steps may be iterated. The distinctions among application requirements, application quality requirements, and data quality requirements are important at both the requirements analysis and conceptual design stages. Both application requirements and application quality requirements are identified to represent a real-world system [5,9]. They are modeled as entities, relationships, and attributes. Data quality requirements are identified to measure the quality of the attribute values (data) that are stored in a database that represents a real-world system. Therefore, new constructs are needed to model data quality requirements in a conceptual design. It might be argued that the distinctions made among application requirements, application quality requirements, and data quality requirements are not necessary because a view mechanism can be applied to provide separate views to the user for application-oriented data, application quality-oriented data, and data qualityoriented data. We reject this argument outright. In order to make use of the view mechanism to provide separate views, an underlying schema containing all the data items that represent application requirements, application quality requirements, and data quality requirements is needed. However, developing such a schema is exactly why the distinctions are made in the first place. On the other hand, once a conceptual schema that incorporates application, application quality, and data quality requirements is obtained and the corresponding database system developed, then view mechanisms can be applied to restrict views that correspond to each of these types ofrequirements. This leads to the following principle: The Data Quality (DQ) Separation Principle: Data quality requirements are modeled separately from application requirements and application quality requirements.

MODELING DATA QUALITY REQUIREMENTS Underlying the Data Quality Separation Principle is the need to measure the quality of attribute values. One might try to model the quality of an attribute value as another attribute for the same entity. This would have undesirable results such as violation of normalization principles in the relational model (e.g., “addressquality” is dependent upon “address”which is dependent upon the “person-id”).Another approach is to model the quality of an attribute as a meta-attribute [10, 11]. However, no constructs in the conceptual design level are explicitly defined for capturing meta-attributes. We, therefore, introduce and define two types of data quality entities: 1) a Data Quality Dimension entity and 2) a Data Quality Measure entity.

42

Extending the ER Model to Represent Data Quality Requirements

Chapter 3

To model the quality of attribute values, an attribute gerund representation is needed.

Data Quality Dimension Entity A user needs to be able to capture different data quality dimensions for some (or all) attributes of an entity; for example, the accuracy and timeliness of the attribute “attendance” or the timeliness of the “re-certification period” (is it up-to-date?). Only one representation is desired, but it needs to be flexible enough to represent different data quality dimensions for different attribute values. A Data Quality Dimension entity is introduced for this purpose. Its primary key is the combination of Dimension Name (D-Name), which can take on values such as ‘timeliness’ and ’completeness’, and Rating, which stores the corresponding value that is obtained on that dimension: DataQualityDimension:[Dimension-Name, Rating] For example: [Accuracy, 1], [Accuracy, 2], [Timeliness, yes] The values that appear in the populated entity are the set of values that exist in the database and will usually be a subset of all possible combinations. They represent the actual dimensions, upon which an attribute’s data quality is assessed, and the values obtained on those dimensions. Because both attributes are part of the key we are able, for example, to capture the fact that the attribute “actual cost” has an accuracy of ‘1’ whereas the accuracy of attribute “attendance” is ‘2’, using only one entity. This representation assumes that the accuracy for all attributes is rated on the same scale. If, for example, the accuracy scale can differ depending on the attribute, then this can be accommodated by adding the attribute name to the key as follows: Data QualityDimension: [Dimension-Name,Attribute, Rating] For example: [Accuracy, class attendance, 1], [Accuracy, actual cost, good]

Data Quality Measure Entity To completely capture the quality aspects, it is necessary to store information on the values that are assigned to a data quality dimension, e.g., the interpretation of “rating”. A Data Quality Measure entity is introduced to capture the interpretation of the quality attribute values. It consists, for example, of a primary key, “Rating”, that could have values ‘1’, ‘2’, and ‘3’, and a textual non-key attribute, “description” which stores a description of what the rating values mean (e.g., 1 is excellent, 2 is acceptable, 3 is poor). The Data Quality Measure entity enables different data qual-

Modeling Data Quality Requirements

43

ity dimensions with heterogeneous measurement scales to be modeled (e.g., accuracy is measured between ‘ 1’ and ‘3’, and timeliness is measured as ‘yes,’ or ‘no’): Data Quality Measure: [Rating, description] For example:

[ 1, excellent] [yes, up-to-date]

(for accuracy) (for timeliness)

There is another reason for introducing this separate Data Quality Measure entity: we want to be able to capture all of the descriptions of “rating” whether or not they currently appear in the database, If “description” were a non-key attribute of the Data Quality Dimension entity, then only those that are instantiated would appear. The purpose of the Data Quality Measure entity is, in essence, for its corresponding relation to serve as a complete “look-up” table. Furthermore, including “description”in the Data Quality Dimension entity would lead to a second normal form violation in the relational model because “description” would depend on only part of the key (“rating”). Analogous to the above, the Data Quality Measure entity could be expanded to model different scales for different attributes for the same data quality measure. For example, suppose the accuracy rating for “attendance” has a different interpretation than the accuracy for “basecost”, then the key would be expanded to include both the dimension and attribute names: Data Quality Measure: [Dimension-Name, Attribute, Rating, description] For example: [Accuracy, attendance, 1, excellent], [Accuracy, base cost, 90%, 90% correct] [Timeliness, attendance, yes, up-to-date]

Attribute Gerund Representation As shown in Figure 3.3, the application entity Class has “attendance” as an attribute for which the user might be interested in its data quality ratings. In the entityrelationship model there is no direct mechanism to associate an attribute (“attendance”) of one entity (Class) with another entity (e.g., a Data Quality Dimension entity or even a Data Quality Measure entity). Furthermore, “attendance” is an application attribute and by the DQ Separation Principle, an application requirement must remain part of the application specification, and not become an entity or attribute below it. This problem is resolved using the Attribute Gerund Representation, First the relationship, Class attendance-data-has Data Quality Dimension, is created that includes “attendance” as part of its name. This data quality relationship can be represented as a gerund entity [1]. Then, a relationship between the gerund and Data Quality Measure entity provides a mechanism for retrieving the descriptions ofthe data quality dimension values.

44

Extending the ER Model to Represent Data Quality Requirements

Chapter 3

An alternative representation would be to make “rating” an attribute of the relationship Class attendance-data-has Data Quality Dimensions. This representation, however, does not capture additional data on rating such as descriptive data on its value. It would only be appropriate if the interpretation of rating is obvious, but this is not usually the case. Further analysis of this and alternative representations is provided in Storey and Wang [6]. In general, the Attribute Gerund Representation can be applied when an application entity attribute needs to be associated directly with a data quality entity that has various quality categories, each of which has a corresponding Rating and “description” (or other data).

Figure 3.3: Attribute Gerund Representation

Conceptual Design Example This section extends the training database example to incorporate data on course standards along with other quality requirements. A user might want to know what standards (an application quality entity) a course (an application entity) might have. This can be modeled by the relationship, Course has Standard, as shown in Figure 3.4. The application quality entity Standard Rating is created with primary key [Std-Name, Value] and a textual non-key attribute ‘‘interpretation.’’

Modeling Data Quality Requirements

45

Figure 3.4: Course Standards

Each occurrence of the relationship Course has Standard has a standard rating value. However, in the entity-relationship model, an entity can only be associated with another entity. Therefore, this relationship must first be converted to an entity gerund [ 1], called Course-Standard. SinceCourse has Standard is a many-tomany relationship, the primary key of the gerund is the concatenation of the keys of Course and Standard. Then, a new relationship is created between this gerund and Standard-Rating; that is, Course-Standard has Standard-Rating. The final entityrelationship model is shown in Figure 3.5. The steps involved in the development ofthe final entity-relationship are outlined in Figure 3.6 and are as follows. In the first step, identify user requirements. The user, with assistance from the designer, would be asked to identify the entities, attributes, and relationships of interest as well as any data quality aspects that need to be captured. Note, for example, that, analogous to the “attendance”attribute of Class, the “value” attribute of Standard Rating could have a data quality dimension. The attribute gerund representation would again be used. First the relationship, Standard Rating value-data-has DQ Dimension would be created. This would be represented by the entity gerund Standard-Rating-Value-DQ-Dimension which, in turn, would be associated with the DQ Measure quality entity.

46

Extending the ER Model to Represent Data Quality Requirements

Figure 3.5: An extended conceptual design of the course database

Chapter 3

Concluding Remarks

47

Step 1: Identify user requirements What courses are available, their certification data and cost. What classes are offered for a course? Accuracy and timeliness measures of class attendance, along with a description of What the accuracy and timeliness values mean. (Data quality requirements) The instructor of a class. What standards are required for a course and a course’s rating? (Application quality requirments) Step 2: Identify application entities: Course: [CourseId, base cost, re-certification period, description] Class: [ClassId, attendance, actual cost] Instructor: [InstructorId, phone] Step 3: Identify corresponding application quality entitites: Standard: [Std-Name, description] Standard rating: [Std-Name, Value, interpretation] Step 4: Identify data quality entities: DQ Dimension: [D-Name, Rating] DQ Measure: [Rating, description] Step 5: Associate application entities with application quality entities: Course-Standard: [CourseID, Std-Name] Step 6: Associate application attributes with data quality entities: Class-Attendance-DQ-Dimension: [ClassID, D-Name] Step 7: List the application relationships: Course has Classes [0,N] [1,1] Instructor teaches Class [1,N] [1,N] Step 8: List the application quality relationships: Course has Standards [1,N] [0,N] Course-Standard has Standard Rating [1,1] [0,N] Step 9: List the data quality relationships: Class attendance-data-has DQ Dimension [1,N] [1,N] Class-Attendance-DQ-Dimension has DQ Measure [1,1] [0,N]

Figure 3.6: Conceptual design for the course database

CONCLUDING REMARKS We have presented how a DQ model can be developed through an extension of the entity-relationship model at the conceptual level. The reader is referred to Storey and Wang [6] for a detailed quality database design example adapted from a human

48

Extending the ER Model to Represent Data Quality Requirements

Chapter 3

resources system. This example shows the logical design and the corresponding relational database developmentgiven a quality-entity-relationship model.

References [1] Chen, P. S., "The Entity-Relationship Approach," in Information Technology in Action: Trends and Perspectives, R. Y. Wang, Editor 1993, Prentice Hall, Englewood Cliffs, 1993. [2] Hauser, J. R. and D. Clausing, "The House of Quality," Harvard Business Review, 66(3), 1988, pp. 63-73. [3] ISO ISO9000 International Standards for Quality Management. In International Standard Organization, Geneva, 1992. [4] Juran, J. M., Juran on Quality by Design: The New Steps for Planning Quality into Goods and Services. Free Press, New York, 1992. [5] Kent, W., Data and Reality. North Holland, New York, 1978. [6] Storey, V. C. and R. Y. Wang. "An Analysis of Quality Requirements in Database Design,". in Proceedings of the 1998 Conference on Information Quality. Massachusetts Institute of Technology: pp. 64-87, 1998. [7] Teorey, T. J., Database Modeling and Design: The Entity-Relationship Approach. Morgan Kaufman Publisher, San Mateo, CA , 1990. [8] Tsichritzis, D. and F. Lochovsky, Data Models. Prentice Hall, Englewood Cliffs, N.J., 1982, [9] Wand, Y. and R. Weber, "On the Ontological Expressiveness of Information Systems Analysis and Design Grammars," 3(3), 1993, pp. 217-237. [ 10] Wang, R. Y., H. B. Kon and S. E. Madnick. "Data Quality Requirements Analysis and Modeling," in Proceedings of the 9th International Conference on Data Engineering. Vienna: pp. 670-677, 1993. [11] Wang, R. Y., M. P. Reddy and H. B. Kon, "Toward quality data: An attribute-based approach," Decision Support Systems (DSS), 13, 1995, pp. 349-372. [12] Wang, R. Y. and D. M. Strong, "Beyond Accuracy: What Data Quality Means to Data Consumers," Journal of Management Information Systems (JMIS), 12(4), 1996, pp. 534.

C H A P T E R

4

Automating Data Quality Judgment

In the previous two chapters, we have presented models to represent data quality requirements. At the conceptual level, Storey and Wang [4] propose a Quality Entity-Relationship (QER) model that incorporates data quality requirements. At the logical level, the Polygen Model [8, 9] and the Attribute-Based Model [6] extend the relational model to capture data quality attributes and data quality indicators. In this chapter, we present a knowledge-based model that derives an overall data quality value from local relationships among quality parameters.5 Because the model provides an overall measure of data quality, information consumers can thus use it for data quality judgment. This model was first proposed in 1991 by Jang and Wang [2]. Although inconclusive, this effort represented a first attempt at automating data quality judgment. It can provide a basis for implementing data quality in Adapted from the MIT CISL working paper (No. CISL-91-08) Jang, Y. and Y. R. Wang . Data Quality Calculus: A data-consumer-based approach to delivering quality data., with permission from the MIT Total Data Quality Management (TDQM) Program. 5

50

Automating Data Quality Judgment

Chapter 4

environments where the quality of data must be judged automatically by the system. The model was later extended in 1995 by Wang, Reddy, and Kon [6].

INTRODUCTION A great deal of research in the area of data quality has concentrated on providing the data consumers with quality indicators (see Table 1) they can use in judging data quality. However, the data consumer must still grapple with the task of analyzing the “metadata”provided and making the appropriate conclusions as to the quality of the data. The knowledge-based model, on the other hand, was developed to assist data consumers in judging if the quality of data meets their requirements, by reasoning about information critical to data quality judgment. It does this by using a data quality reasoner to help in assessing, from the user’s perspective, the degree to which datameets certain requirements. Because requirements of data depend to a large extent on the intended usage of the data, the model addresses the issue of how to deal with user- or applicationspecific quality-parameters with a knowledge-based approach. It also addresses the issue of how to represent relationships among quality parameters and how to reason with such relationships to draw insightful knowledge about overall data quality. To do so, it assumes that data quality parameters, such as those shown in Table 2, are available for use. Table 1 : Data Quality Indicators Indicator Source Creation-time Updatefrequency Collectionmethod

data #1 DB#1 6/11/92

data #2 DB#2 6/9/92

Data #3 DB#3 6/2/92

Daily

weekly

Monthly

barcode

entry clerk

radio freq.

Table 2: Data Quality Parameters Parameter Credibility

Data #1 High

Timeliness Accuracy

High High

data #2 Medium Low Medium

Data #3 Medium Low Medium

The data quality reasoner is a simple data quality judgment model based on the notion of a "census of needs." It applies a knowledge-based approach in data quality judgment to provide flexibility advantages in dealing with the subjective,

Overview

51

decision-analytic nature of data quality judgment. The data quality reasoner provides a framework for representing and reasoning with local relationships among quality parameters to produce an overall data quality level to assist data consumers in judging data quality, particularly when data involved in decision-making come from different and unfamiliar sources.

QUALITY INDICATORS

AND

QUALITY PARAMETERS

The essential distinction among quality indicators and quality parameters is that quality indicators are intended primarily to represent objective information about the data manufacturing process [7]. Quality parameters, on the other hand, can be userspecific or application-specific, and are derived from either the underlying quality indicators or other quality parameters. The model presented in this chapter uses a “quality hierarchy” whereas a single quality parameter can be derived from n underlying quality parameters. Each underlying quality parameter, in turn, could be derived from either its underlying quality parameters or quality indicators. For example, a user may conceptualize quality parameter Credibility as one depending on underlying quality parameters such as Source-reputation and Timeliness. The quality parameter Source-reputation, in turn, can be derived from quality indicators such as the number of times that a source supplies obsolete data. The model assumes that such derivations are complete, and that relevant quality parameter values are available.

OVERVIEW In general, several quality parameters may be involved in determining overall data quality. This raises the issue of how to specify the degree to which each quality parameter contributes to overall data quality. One approach is to specify the degree, in some absolute terms, for each quality parameter. It may not, however, be practical to completely specify such values. Rather, people often conceptualize local relationships, such as "Timeliness is more important than the credibility of a source for this data, except when timeless is low." In such a case, if timeliness is high and sourcecredibility is medium, the data may be of high quality. The model presented in this chapter provides a formal specification of such local “dominance relationships”between quality parameters. The issue is, then, how to use these local dominance relationships between quality parameters, and what can be known about data quality from them. Observe that each local relationship between quality parameters specifies the local relative significance of quality parameters. One way to use local dominance relationships would be to rank and enumerate quality parameters in the order of significance im-

52

Automating Data Quality Judgment

Chapter 4

plied by local dominance relationships. Finding a total ordering of quality parameters consistent with local relative significance, however, can be computationally intensive. In addition, a complete enumeration of quality parameters may contain too much information to convey to data consumers any insights about overall data quality. This chapter provides a model to help data consumers raise their levels of knowledge about the data they use, and thus make informed decisions. Such a process represents data quality filtering.

RELATED WORK The decision-analytic approach (as summarized in [3]) and the utility analysis under multiple objectives approach (as summarized in [1]) describe solution approaches for specifying preferences and resolving multiple objectives. The preference structure of a decision-maker or evaluator is specified as a hierarchy of objectives. Through a decomposition of objectives using either subjectively defined mappings or formal utility analyses, the hierarchy can be reduced to an overall value. The decision-analytic approach is generally built around the presupposition of the existence of continuous utility functions. The approach presented in this chapter, on the other hand, does not require that dominance relations between quality parameters be continuous functions, or that their interactions be completely specified. It only presupposes that some local dominance relationships between quality parameters exist. The representational scheme presented in this chapter is similar to those used to represent preferences in sub disciplines ofArtificial Intelligence, such as Planning [10, 11]. That research effort, however, has focused primarily on issues involved in representing preferences and much less so on computational mechanisms for reasoning with such knowledge.

DATA QUALITY REASONER We now discuss the data quality reasoner (DQR). DQR is a data quality judgment model that derives an overall data quality value for a particular data element, based on the following information: 1, A set QP of underlying quality parameters qi that affect data quality: QP = {q1, q2, ...,qn}. 2. A set DR of local dominance relationships between quality parameters in QP.

Data Quality Reasoner

53

In particular, we address the following fundamental issues that arise when considering the use of local relationships between quality parameters in data quality judgment: • How to represent local dominance relationships between quality parameters. • What to do with such local dominance relationships.

Representation of Local Dominance Relationships This section discusses a representation of local dominance relationships between quality parameters. To facilitate further discussion, additional notations are introduced below. For any quality parameter qi, let Vi denote the set of values that qi can take. In addition, the following notation is used to describe value assignment; for quality parameters. For any quality parameter qi, the value assignment qi := v (for example, Timeliness := High) represents the instantiation of the value of qi as v, for some v in Vi. Value assignments for quality parameters, such as qi := v, are called “quality-parameter value assignments”. A quality parameter that has a particular value assigned to it is referred to as an instantiated quality parameter. For some quality parameters q1, q2, ..., qn, and for some integer n ≤ 1, q1 q2 ... qn represents a conjunction of quality parameters. Similarly, q1:=v1 q2:=v2 ... qn:=vn, for some vi in Vi, for all i = 1,2, ... , n, represents a conjunction of quality-parameter value assignments. Note that the symbol used in the above statement denotes the logical conjunction, not the set intersection, of events asserted by instantiating quality parameters. Finally, ' ' notation is used to state that data quality is affected by quality parameters. It is represented as (q1 q2 ... qn) to mean that data quality is affected by qality parameters q1, q2, ..., and qn. The statement (q1 q2 ... qn) is called a quality-merge statement, and is read as “the quality merge of q1, q2, ..., and qn.“The simpler notion, (q1, q2, ..., qn) can also be used. A quality-merge statement is said to be instantiated, if all quality parameters in a quality-merge statement are instantiated to certain values. For example, statement (q1:=v1 q2:=v2 ... qn:=vn) is an instantiated quality-merge statement of (q1, q2, ..., qn), for some vi in Vi and for all i = 1,2 ,..., n. The following defines a local dominance relationship among quality parameters. Definition 1 (Dominance relation): Let E1 and E2 be two conjunctions of qualityparameter value assignments. E1 is said to dominate E2, denoted by E1 >dE2, if and

54

Automating Data Quality Judgment

Chapter 4

only if (E1, E2, +) is reducible to (E1, +), where "+" stands for the conjunction of value assignments for the rest of the quality parameters, in QP, which are shown neither in E1 nor in E2. Note that, as implied by "+," this definition assumes the contextinsensitivity ofreduction (E1,E2,+) can be reduced to (E1, +), regardless of the values of the quality parameters in QP that are not involved in the reduction. Moreover, "+"implies that these uninvolved quality parameters in QP remain unaffected by the application of the reduction. For example, consider a quality-merge statement that consists of quality parameters Source-credibility, Interpretability, Timeliness, and others. Suppose that when Source-credibility and Timeliness are High, and Interpretability is Medium, Interpretability dominates the other two. This dominance relationship can be represented as follows: "Interpretability := Medium >d Source-Credibility := High Timeliness := High" Then: (Source-Credibility := High, Interpretability := Medium, Timeliness := High, +) isreducibleto quality-merge statement (Interpretability :=Medium, +). The evaluation of the overall data quality for a particular data element requires information about a set of quality parameters that play a role in determining the overall quality, QP = {q, qi ..., qn},and a set DR of local dominance relationships between quality parameters in QP. The information provided in QP is interpreted by DQR as follows: "The overall quality is the result of quality merge of quality parameters q1, q2, ..., and qn, i.e., (q1,q2,...,qn).” Local dominance relationships in DR are used to derive an overall data quality value. However, it may be unnecessary or even impossible to explicitly state each and every plausible relationship between quality parameters in DR. Assuming incompleteness of preferences in quality parameter relationships, the model approaches the incompleteness issue with the following default assumption: For any two conjunctions of quality parameters, if no information on dominance relationships between them is available, then they are assumed to be in the indominance relation. The indominance relation is defined as follows: f

Definition 2 (Indominance relation): Let E1 and E2 be two conjunctions of qualityparameter value assignments. E1 and E2 are said to be in the indominance relation, if neither E1 >d E2 nor E2 >dE1. When two conjunctions of quality parameters are indominant, a data consumer may specify the result of their quality merge according to his or her needs.

Reasoning Component of DQR The previous section discussed how to represent local relationships between quality parameters. The next question that arises is then how to derive overall data quality from such local dominance relationships, i.e., how to evaluate a quality-merge state-

Data Quality Reasoner

55

ment based on such relationships. This task, simply referred to as the "data-qualityestimating problem," is summarized as follows:

Data-Quality-Estimating Problem: Let DR be a set of local dominance relationships between quality parame-

ters q1,q2, ..., and qn. Compute (q1,q2,...,qn), subject to local dominance relationships in DR. An instance of the data-quality-estimating problem is represented as a list of a quality-merge statement and a corresponding set of local dominance relationships, i.r., ( (q1,q2,...,qn), DR). The rest of this section presents a framework for solving the data-qualityestimating problem, based on the notion of "reduction". The following axiom defines the data quality value when only one quality parameter is involved in quality merge. Axiom 1 (Quality Merge): For any quality-merge statement (q1,q2,...,qn), if n = 1, then the value of (q1,q2,..., qn) is equal to that of q1. Quality-merge statements with more than one quality parameter are reduced to statements with a smaller number of quality parameters. As implied by Definition 1 and the default assumption, any two conjunctions of quality-parameter valueassignments can be in either the dominance relation or in the indominance relation. Below, we define axioms which provide a basis for the reduction. Axiom2 specifies that any two conjunctions cannot be both in the dominance relation and in the indominancerelation. Axiom 2 (Mutual Exclusivity): For any two conjunctions E1 and E2 of qualityparameter value assignments, E1 and E2 are related to each other in exactly one ofthe followingways: 1. E1 >d E2 2. E2 >d E1 3. E1 and E2 are in the indominance relation. Axiom3 defines the precedence of the dominance relation over the indominance relation. This implies that while evaluating a quality-merge statement, quality parameters in the dominance relation are considered before those not in the dominancerelation.

56

Automating Data Quality Judgment

Chapter 4

Axiom 3 (Precedence of >d): The dominance relation takes precedence over the indominancerelation. Reduction-Based Evaluation: A reduction-based evaluation scheme is any evaluation process where the reduction operations take precedence over all other evaluation operations. Definition 1 and axiom 3 allow the reduction-based evaluation strategy to be used to solve the data-quality-estimating problem for quality-merge statements with more than one quality parameter. The use of dominance relationships to reduce a quality-merge statement raises the issue ofwhich local dominance relationships should be applied first, i.e., regarding the order in which local dominance relationships are applied. Unfortunately, the reduction of a quality-merge statement is not always well defined. In particular, a quality-merge statement can be reduced in more than one way, depending on the order in which the reduction is performed. For example, consider an instance of the data quality-estimating problem, (_(q1,q2,q3,q4,q5,q6), DR), where DR consists of the followinglocaldominancerelationships: q1 > d q2, q2 > d q3, q4 > d q5, q5 > d q6, (q1 q4) > d (q q6), (q2 q5) > d (q1 q4). Then, the quality-merge statement (q1,q2,q3,q4,q5,q6) can be reduced to more than one irreducible quality-merge statement as shown below: • In case q2>d q3, q5>d q6, and (q2 q5) >d (q1 q4) are applied in that order, (q1,q2,q3,q4,q5,q6) is reducible to (q2,q5), as follows: (q1,q2,q3,q4,q5,q6) = (q1,q3,q4,q5,q6), by applying q1>d q2, = (q1,q3,q4,q6), by applying q4 >dq5, = (q1 q4), by applyng (q1 q4) >d (q3 q6). • In case q2>d q3, q5>d q6, and (q2 q5) >d (q1 q4) are applied in that order, (q1,q2,q3,q4,q5,q6) is reducible to (q2,q5), as follows: = = =

(q1,q2,q3,q4,q5,q6) (q1,q2,q4,q5,q6), by applying q2 >d q3, (q1,q2,q4,q5), by applying q5 >d q6, (q2,q5), by applying (q2 q5) >d (q1 q4).

As illustrated in this simple example, the reduction of a quality-merge statement is not always well defined.

First-Order Data Quality Reasoner

57

FIRST-ORDER DATA QUALITY REASONER This section investigates a simpler data quality reasoner that guarantees the welldefined reduction of quality-merge statements, by making certain simplifying assumptions. To facilitate the next step of derivation, an additional definition is introduced.

Definition 3 (First-Order Dominance Relation): For any two conjunctions E1 and E2 of quality parameters such that E1 and E2 are in the dominance relation, E1 and E2 are said to be in the first-order dominance relation, if each of E1 and E2 consists of one and only one quality parameter. The first-order data quality reasoner, in short called DQR1, is a data quality judgment model that satisfies the following: First-order Data Quality Reasoner (DQR1) 1. Axioms 1, 2, and 3 hold. 2. Only indominance and first-order dominance relationships are allowed. 3.
In the first-order data quality reasoner, higher-order dominance relationships, such as q1 q2 q3 >d q4 or (q4 q5) >d (q1 q2), are not allowed. In addition, the firstorder data quality reasoner requires that the dominance relation be transitive. This implies that for any conjunctions of quality-parameter value assignments, E1, E2, and E3, if E1 >dE2 and E2>dE3, then E1 >dE3. transitivity of the dominance relation impIies the need for an algorithm to verify that, when presented with an instance of the quality-estimating proble (q1,q2,...,qn), DR), dominance relationships in DR do not conflict with each other. Well-known graph algorithms can be used for performing this check [5]. Quality-merge statements can be classified into groups, with respect to levels ofthe reducibility, as defined below.

Definition 4 (Irreducible Quality-Merge Statement): For any instantiated qualitymerge statement e = (q1:=v1,q2:=v2,...,qn:=vn) such that n ≤ 2, for some vi in Vi and ∀i = 1,2, ..., n; e is said to be irreducible if for any pair of quality-parameter value assignments in e (say qi:=vi and qj:=vj) qi:=vi and qj:=vj are in the indominance relation. Similarly, any quality-merge statement that consists of one and only one quality parameter is said to be irreducible. Definition 5 (Completely-Reducible Quality-Merge Statement): For any instantiated quality-merge statement e = (q1:=v1, q2:=v2,...,qn:=vn) such that n ≤ 2, for

58

Automating Data Quality Judgment

Chapter 4

some vi in Vi, ∀i = 1,2,..., and n, e is said to be completely reducible, if for any pair of quality-parameter value assignments in e, say qi:=vi and qj:=vj, qi:=vi and qj:=vj are in the dominance relation. The next two sections discuss algorithms for evaluating a quality merge statement in DQR1. This process is diagrammed in Figure 4.1. Algorithm Q-Merge is the top-level algorithm that receives as input a quality-merge statement and the corresponding quality parameter relationship set. Within Algorithm Q-Merge, there is a two-stage process. First, the given quality-merge statement is instantiated accordingly. It then calls Algorithm Q-Reduction to reduce the Quality Merge Statement into its corresponding irreducible form.

Figure 4.1. The Quality-Merge Statement (QMS) Evaluation Process

The Q-Reduction Algorithm This section describes an algorithm, called Q-Reduction, for reducing a qualitymerge statement into an irreducible quality-merge statement, according to local dominance relationships between quality parameters. We continue to assume that all dominancerelationshipsare first-order.

First-Order Data Quality Reasoner

59

Figure 4.2: Q-Reduction Algorithm

Algorithm Q-Reduction in Figure 4.2 takes as input an instantiated qualitymerge statement e and DR, and returns as output an irreducible quality-merge statement of e. The instantiated quality-merge statement e is reduced as follows: For expository purposes, suppose that e = (q1:=v1,q2:=v2,...,qn:=vn), for some vi in Vi, for all i = 1,2, ... , and n, and let Ω be a dynamic set of qualityparameter value assignments, which is initialized to {q1:=v1,q2:=v2,...,qn:=vn}. For any pair of quality-parameter value assignments qi:=vi and qj:=vj in e, if qi:=vi >d qj:=vj is a member of DR, then e is reducible to a quality-merge statement with the quality parameters in Ω, less qj:=vj (by Definition 1.) This allows us to remove qj:=vj from Ω, if both qi:=vi and qj:=vj are elements in Ω. the process of removing dominated quality parameters continues until no pair of the quality parameters in Ω is related in the dominance relation. Let Ω denote the modified Ω produced at the end of this removal process. The quality merge of the quality parameters in Ω' is the corresponding irreducible quality-merge statement of e. The algorithm returns (Ω '). It is proven in [2] that the Q-Reduction algorithm shown in Figure 4.2 always results in a unique output in the first-order data-quality reasoner provided that all dominancerelations are first-order.

60

Automating Data Quality Judgment

Chapter 4

Q-Merge Algorithm When presented with an instance of the quality-estimating problem ( (q1,q2,...,qn), DR) for some integer n, the Q-Merge algorithm first instantiates the given qualitymerge statement. Then, the instantiated quality-merge statement is reduced until the reduction process results in another instantiated quality-merge statement that cannot be reduced any further (using the Q-Reduction algorithm.) This raises the issue of how to evaluate an irreducible quality-merge statement. Unfortunately, the evaluation of an irreducible quality-merge statement is not always well defined. When evaluating an irreducible quality-merge statement, the number of orders in which the quality merge operation can be applied grows exponentially with the number of quality parameters in the statement. In particular, certain quality-merge statements may be merged in more than one way, depending on the order in which the merge is performed. It is possible that this set might include every element of Vi’s This model evades the problem by presenting quality-parameter value assignments in the irreducible quality-merge statement returned by the Q-Reduction algorithm so that a user may use the information presented according to his or her needs. Figure 4.3 summarizes the Q-Merge algorithm. Q-Merge Algorithm

Input: (e, DR), where e = (q1,q2,...,qn), for some integer n and DR is a set of local dominance relationships between quality parameters q1, q2, ..., and qn. Output: Overall data quality value produced by evaluating e. 1.

Instantiate e. ;; Suppose that q1,q2 .. ., and qn are instantiated as v1, v2,... ., and vn, respectively, for some v1in Vt ;; for all i = 1,2, ..., and n.

2. 3.

e' ← IF (n = 1) THEN e ELSE Q-Reduction ( (q1:=v1, q2:=v2,...,qn:=vn) , DR) Present quality-value assignments in e’. Figure 4.3: Q-Merge Algorithm

Conclusion

61

CONCLUSION In this chapter, we have presented a knowledge-based model that tailors data quality judgment to the data consumers’ needs. The model provides flexibility in representing the specific requirements of data consumers by allowing them to specify local dominance relationships between quality parameters. It will then perform the reduction of quality-merge statements and derive a value for overall data quality, i.e., a measurement ofdata quality. The model presented has limitations that have to be overcome in order for it to be of practical use. The following is a list of some of the limitations of the model and suggestions for its enhancement. Higher-order data quality reasoner: The first-order data quality reasoner evades the problem of ill-defined reduction by prohibiting higher order relationships. Realworld problems, however, often involve more complex relationships than first-order relationships between quality parameters. In order to deal with higher-order relationships, both the representational and algorithmic components of the first-order data quality reasoner would need to be extended. Data Acquisition/A hierarchy of quality indicators and parameters: This model assumed that values of quality parameters are available so that quality-merge statements can be instantiated properly. Issues of how to represent and how to streamline such values to the data quality reasoner, however, must be addressed. One approach to these issues would be to organize quality parameters and quality indicators in a hierarchy. Then, to each data element or type can be attached information about how to compute a value of a quality parameter. Such a hierarchy would allow the derivation of a quality parameter value from its underlying quality parameters and quality indicators. A tool for automatically constructing such a hierarchy and computing quality-parameter values would enhance the utility of the data quality reasoner. Knowledge Acquisition: The capability of using local dominance relationships, which are typically user-specific or application-specific, allows us to build systems more adaptable to customers’ needs. As application domains are complex, however, it becomes increasingly difficult to state all the relationships that must be known. Such knowledge acquisition bottlenecks could be alleviated through development of a computer program for guiding the process of acquiring relationships between quality parameters. User interface for cooperative problem solving: The Different orders in which quality parameters in an irreducible quality-merge statement are evaluated may result in different values. This model dealt with the need to evaluate an irreducible qualitymerge statement by simply presenting information on quality parameters in an irreducible quality-merge statement. Development of a user interface, which allows

62

Automating Data Quality Judgment

Chapter 4

evaluating irreducible quality-merge cooperatively with a data consumer, could lessen the problem. The model presented in this chapter provides a first step toward the development of a system that can assist data consumers in judging whether data meets their requirements. This is particularly important when decision-making involves data from different, foreign sources.

References [1] Chankong, V. and Y. Y. Haimes, Multiobjective Decision Making: Theory and Methodology. North-Holland Series in System Science and Engineering, ed. A. P. Sage. Elsevier Science, New York, N.Y., 1983. [2] Jang, Y. and Y. R. Wang , Data Quality Calculus: A data-consumer-based approach to delivering quality data (No. CISL-9 1-08). Composite Information Systems Laboratory, Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA, 02139, 1991 [3] Keeney, R. L. and H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs. John Wiley & Son, New York, 1976. [4] Storey, V. C. and R. Y. Wang. "An Analysis of Quality Requirements in Database Design," in Proceedings of the 1998 Conference on Information Quality. Massachusetts Institute of Technology: pp. 64-87, 1998. [5] T H Cormen, T. H., C. E. Leiserson and R. L. Rivest, Introduction to Algorithms. MIT Press, Cambridge, MA, USA, 1990. [6] Wang, R. Y., M. P. Reddy and H. B. Kon, "Toward quality data: An attribute-based approach," Decision Support Systems (DSS), 13, 1995, pp. 349-372. [7] Wang, R. Y., P. Reddy and H. B. Kon, "An Attribute-based Model of Data for Data Quality Management," 1992. [8] Wang, Y. R. and S. E. Madnick. "A Polygen Model for Heterogeneous Database Systems: The Source Tagging Perspective," in Proceedings of the 16th International Conference on Very Large Data bases (VLDB). Brisbane, Australia: pp. 519-538, 1990. [9] Wang, Y. R. and S. E. Madnick, 'Where Does the Data Come From: Managing Data Integration with Source Tagging Capabilities," 1990. [10] Wellman, M. P., Formulation of Tradeoffs in Planning Under Uncertainty. Pitman and Morgan Kaufmann, 1990. [ 1 1] Wellman, M. P. and J. Doyle. "Preferential Semantics for Goals," in Proceedings of the 9th National Conference on Artificial Intelligence, 1991.

C H A P T E R

5

Developing a Data Quality Algebra

W hen accessing various data sources, it is important to know the meaning of the data [14, 15] and the quality of the data retrieved [11]. Despite the elegance of the relational theory [4, 5] and mechanisms such as integrity constraints [6, 7] to ensure that the database state reflects the real worldstate[9],manydatabases contain deficient data [ 16]. If the data in the underlyingbaserelationsaredeficient, then the query results obtained from these base relationsmayalsocontaindeficient data even if the query processing mechanism is flawless. These deficient data, in turn, may lead to erroneous decisions that can result in significant social and economic impacts. In this chapter, we present a mechanism bsed on the relational algebra to estimate the accuracy of data derived from base relations. We willintroduce a data

64

Developing a Data Quality Algebra

Chapter 5

quality algebra6 that estimates the quality of query results given the quality characteristics of the underlying base relations, in the context of a single relational database environment. We will focus on the accuracy dimension. Thus, the accuracy dimension will be used in this chapter to refer to data quality although, as we saw in chapter 1 , various dimensions such as interpretability, completeness, consistency, and timeliness have been identified as typical data quality dimensions [1, 2].

Assumptions We make the following assumptions: Assumption A1: Query processingisflawless. Assumption A2: Query processing operations are based on the relational algebra. Assumption A3: Accuracy estimates of relevant base relations are given at the relational level. Assumption A4: The accuracy of tuples in base relations can be validated. The first two assumptions define the scope of the problem to be one that focuses on developing a mechanism based on the relational algebra for estimating the accuracy of data derived from base relations. The last two assumptions are elaborated on below. In principle, accuracy information for the relevant base relations should be provided as the input, at a certain level of granularity, for the estimation of the accuracy of data derived from these base relations. In the relational model, a base relation must be at least in the first normal form, which means every attribute value is atomic. Each atomic attribute value, however, can be either accurate or inaccurate and can be tagged accordingly. There are also emerging commercial tools for estimating the quality characteristics of a base relation; see, for example, QDB Solutions' QD B/Analyze. The attribute-based model presented in chapter 2 can be used to store data accuracy value [17-19]; however, measurement and storage ofthe accuracy of each cell value is an expensive proposition. Similarly, it is also relatively expensive to measure and store accuracy information at the tuple level. Therefore, Assumption A3 states that accuracy estimates are available at the relational level, and in addition, these estimates are based on some tuple level accuracy measures. To measure the Adapted from the M.I.T. TDQM working paper, A Data Quality Algebra for Estimating Query Result Quality, co-authored by M.P. Reddy and Richard Wang, with permission from the MIT Total Data Quality Management (TDQM) Program.

6

65

accuracy of a tuple, some validation procedure should be available such as those described by Janson [8] and Paradice & Fuerst [12].

Notation The notation used is summarized below. At N M P AR IMR IAR Ska Ski Skim Skia

The deterministic accuracy of a tuple ‘t’ The number of accurate tuples The number of mismember tuples The number oftuples with at least one inaccurate attribute value The ratio accurate tuples in relation R to the total number of tuples in R The ratio of mismember tuples in relation R to the total number of tuples in R The ratio of tuples in R with at least one inaccurate attribute value to the total number of tuples in R The accurate portion ofrelation Sk The inaccurate portion ofrelation Sk The inaccurate portion ofrelation Sk due to inaccurate membership The inaccurate portion ofrelation Sk due to inaccurate attribute value

(The following notation is adopted from the relational model) |R| Cardinality of relation R σc Selection operation with selection condition c Πp Projection operation with projection list given by the set P ∞ Cartesian Product operation ∪ Union operation Difference operation

Definitions For exposition purposes, let DEPT1_EMP (Table 1) and DEPT2_EMP (Table 2) be two employee relations for two departments, DEPT1 and DEPT2. Each employee relation consists of three attributes: EMP_ID, EMP_NAME, and EMP_SAL.

66

Developing a Data Quality Algebra

Table 1: DEPT1_EMP

EMP ID EMP NAME Henry 1 Jacob 2 3 Marshall Alina 4 5 Roberts 6 Ramesh 7 Patel 8 Joseph 9 John Arun 10

EMP SAL 30,000 32,000 34,000 33,000 50,000 45,000 46,000 55,000 60,000 50,000

Chapter 5

Table 2: DEPT2_EMP

EMP ID EMP NAME Henry 1 2 Jacob 5 Roberts 6 Ramesh 9 John Nancy 11 12 James 13 Peter Ravi 14 Anil 15

EMP SAL 30,000 32,000 55,000 45,000 60,000 39,000 37,000 46,000 55,000 45,000

In the relational model, a relation represents a time-varying subset of a class of instances sharing the same domain values that correspond to its relational scheme. For example, the relation DEPT1_EMP is a subset or a complete set of the employees in DEPT1. Every tuple is implicitly assumed to correspond to a member of the class of instances of the same entity type as defined by a relational scheme. In reality, from a data quality perspective, this implicit assumption may not hold. For example, some tuples in DEPT1_EMP may not belong to DEPT1 (For example, a database administrator may, by mistake, insert into DEPT1 an employee tuple that belongs to another department, or may not have had time to delete a tuple that corresponded to a former employee who has left DEPT1). A relation containing one or more tuples that do not meet the above implicit assumption is said to have tuple mismembership. A tuple that does not satisfy the implicit assumption is referred to as a mismember tuple; otherwise, it is referred to as a member tuple. In order to determine the accuracy of an attribute value in a tuple, one must know if the attribute value reflects the real-world state. We refer to an attribute value as being accurate if it reflects the real-world state; otherwise, it is inaccurate. To validate that the gender, age and income data for a particular person are accurately stored in a tuple, it is necessary to know that the tuple indeed belongs to this person, and that the attribute values in the tuple reflect the real-world state. The concepts of member tuple and accurate attribute values lead to Definition D1 below. Definition D1 - Deterministic Tuple Accuracy: A tuple t in a relation R is accurate if and only if it is a member tuple and every attribute value in t is accurate; otherwise, it is inaccurate. Let At denote the accuracy oft. If t is accurate, then the value of At is 1 ; if t is inaccurate, then the value ofAt is 0. By Definition D1, a tuple can be inaccurate either because it is a mismember tuple or because it has at least one inaccurate attribute value. For example, if an em-

67

ployee belongs to the department, and the employee’s ID number, name, and salary are accurately stored in the tuple denoted by emp_tuple, then emp_tuple is said to be accurate, or Aemp_tuple = 1. On the other hand, Aemp_tuple = 0 if the employee does not belong to the department or if any of the ID number, name, or salary is inaccurate. Suppose that in the relation DEPT1_EMP, the tuple for Employee 2 contains inaccurate salary, the tuple for Employee 8 contains inaccurate name, and Employee 6 no longer works for DEPT1; in DEPT2_EMP, the tuples for Employees 2 and 5 contain inaccurate salary, while the tuple for Employee 15 contains inaccurate name. These tuples are shaded in Tables 1 and 2. By Definition D1, the tuple for Employee 6 is a mismember tuple, and other tuples contain inaccurate attribute values. Definition D2 - Relation Accuracy: Let AR denote the accuracy of relation R, Nthe number of accurate tuples in R, and |R| the cardinality of relation R. Then N AR = |R| Let IMR denote the inaccuracy due to mismember tuples, and IAR denote the inaccuracy due to attribute value inaccuracy for the relation R. Let M denote the number of mismember tuples in which every attribute value is accurate. (If a mismember tuple in the relation R has an inaccurate attribute value, then it will be counted under IAR but not under IMR.) If P denotes the number of tuples with at least one inaccurate attribute value, then it follows that M IMR = |R| P IAR = |R| AR + IMR + IAR = 1 As mentioned earlier, an accuracy tag could be associated with each tuple in a base relation. For a large database, however, measurement and storage ofthese accuracy tags is expensive. As such, in this paper we adopt a statistical approach to provide accuracy estimates for base relations. In this approach, we measure four parameters for each base relation: (i) the number of tuples in the relation, (ii) the accuracy of the relation, (iii) the inaccuracy of the relation due to mismembership, and (iv) the inaccuracy of the relation due to inaccurate attribute value. The collection of these four parameters for each relation in a component database constitutes the quality profile ofthe database. Let T denote the relation that contains a random sample of tuples of a relation R. In T, let NT denote the number of accurate tuples, MT the number of mismember tuples, and PT the number of tuples with at least one inaccurate attribute value. Using Definition D2, we get

68

Developing a Data Quality Algebra

AR =

Chapter 5

NT |T| |R|

PT |R| InTable 1, AR=0.7, IMR=0.1 and IAR=0.2. In Table 2, AR=0.7, IMR= 0, and IAR = 0.3. Suppose that DEPT1_EMP and DEPT2_EMP (Tables 1-2) are two relations in a database. The quality profile of the database would appear as follows: IAR =

Relation Name DEPT1_EMP DEPT2_EMP

Size 10 10

AR 0.7 0.7

IMR 0.1 0.0

IAR 0.2 0.3

Definition D3 - Null Relation Accuracy: In D2, if the cardinality |R| is zero, then we define AR = 1. In order to proceed with the analysis, we make the following additional assumption and use the definitions listed below: Assumption A5: At the relational level, the error is uniformly distributed across tuples andattributes. Definition D4 - Probabilistic Tuple Accuracy: At the relational level, all the tuples in a relation have the same probability of being accurate regardless of their deterministic tuple accuracy. Let AR denote the accuracy of relation R, then the probabilistic accuracy of each tuple in R is AR. In Table 1, the deterministic tuple accuracy of Tuple 2 is 1 and Tuple 8 is 0. Further, AR = 0.7, and the probabilistic tuple accuracy for both Tuples 4 and 8 (in fact, for every tuple in DEPT1_EMP) is 0.7. Definition D5 - Probabilistic Attribute Accuracy: All attributes in a base relation have the same probability of being accurate regardless of their respective deterministicaccuracies. Based on the above definition, one can compute the probabilistic attribute accuracy as follows. Let AR denote the accuracy of relation R, D the degree of the relation R, and X an attribute in relation R. Then, the probability that attribute X is accurate is given by

A Data Quality Algebra

69

In Table 1, the accuracy of relation DEPT1_EMP is 0.7 and the probabilistic attribute accuracy ofthe attribute EMP_NAME is Assumption A5 is relatively strong and will not hold in general. From the theoretical viewpoint, however, it provides an analytic approach for estimating the accuracy of data derived from base relations, especially because the detailed accuracy profile of the derived data is rarely available on an a-priori basis.

A DATA QUALITY ALGEBRA In this section, we compute accuracy measures for reports based on the accuracy estimates of input relations for the five orthogonal algebraic operations: Selection, Projection, Cartesian product, Union, and Difference. Other traditional operators can be defined in terms ofthese operators [10]. We make assumption A6 below: Assumption A6 The cardinality of the resulting relation |R| is available. In practice, |R| can be either counted after an operation is performed or estimated before an operation is performed. Algorithms for estimating |R| are given in [3]. Let AS, AS1, and AS2 denote the accuracy of Relations S, S1, and S2 respectively. Let R denote the resulting relation of an algebraic operation on S (if unary operation) on S1 and S2 (if binary operation). Let AR denote the accuracy of R.

Accuracy Estimation of Data Derived by Selection Selection is a unary operation denoted as R = σcS where ‘ σc ’ represents the selection condition. Furthermore, conditions: (i) (ii)

As=1 As=0

S and AR must satisfy the following boundary

AR=1 AR=0

(S1) (S2)

Using Definition D4, the probabilistic accuracy of each tuple in the relation S is given by AS. Since the selection operation selects a subset oftuples from S, it follows that the estimated number of accurate tuples in R is given by |R| * AS By Definition D2,

70

Developing a Data Quality Algebra

AR =

|R|*AS = AS |R|

Chapter 5

(1)

Equation (1) satisfies both boundary conditions (S1) and (S2). By the same token, IMR, the percentage of mismember tuples in R, and IAR, the percentage of tuples in R with at least one inaccurate attribute value can be estimated as follows: IMR = IMS IAR = IAS where IMs denote the percentage of mismember tuples in S, and IAS the percentage of tuples in S with at least one inaccurate attribute value. Equation (1) is derived using Assumption A6, which assumes errors to be uniformly distributed. When the error distribution is not uniform, the worst and best cases for the selection operation can be analyzed as shown in the following subsections.

Worst case when error distribution is non-uniform From Definition D2, the worst case occurs when the selection operation selects the maximum number ofinaccurate tuples into the resulting relation R. Let Si denote the number of inaccurate tuples in S. Two cases need to be considered. In the first case, we have |Si| ≥ |R |. this scenario, the worst case occurs when all the tuples selected into R are inaccurate and thus, AR = 0. In the second case, we have |Si| < |R |. Under this scenario, the worst case occurs when all the inaccurate tuples in S are included in R. As such, the number of accurate tuples in R is given by |R|- (|S| * (1-AS)). Using Definition D2, we get: (2) Equation (2) satisfies both boundary conditions (S1) and (S2), as shown below. In boundary condition (S1), AS = 1. By substituting AS with 1 in Equation (2), we get AR = 1. Therefore, boundary condition (S1) holds for Equation (2). In boundary condition (S2), AS = 0. It follows from Equation (2) that |S| (2a) AR = 1 |R| Since |S| cannot be less than |R| by the definition of the selection operation, it follows from Equation (2a) that AR ≥ 0. Since by definition, AR ≤ 0, therefore AR = 0. Thus, Equation (2) satisfies boundary condition (S2).

A Data Quality Algebra

71

Best case when error distribution is non-uniform From Definition D2, the best case occurs when the maximum number of accurate tuples is selected into the resulting relation R. Let Sa denote the number of accurate tuples in S. Two cases need to be considered. In the first case, we have |Sa| ≥ |R|. Under this scenario, the best case occurs when all the tuples selected into R are accurate. Therefore, AR = 1. In the second case, we have Sa < R .Under this scenario, the best case occurs when all the accurate tuples are included in R. The number of accurate tuples in relation R is given by AS * |S|. Therefore, by Definition D2, we have |S| (3) AR = AS * |R| Equation (3) satisfies both boundary conditions (S1) and (S2).

Accuracy Estimation of Data derived by Projection Projection selects a subset of attributes from a relation and is denoted as R(A) = ΠA S(B), where A and B are the set ofattributes in R and S respectively, and A B. By definition, we have |R| ≤ |S|. In addition, the accuracy of R must satisfy the following two boundary conditions: ⇒ AR = 0 (P1) AS = 0 AS = 1 ⇒ AR = 1 (P2) We consider projection as a two-step operation. In Step 1, the set of attributes that is not present in R will be eliminated from S to generate a relation Q, in which and Q may have duplicate tuples. In Step 2, all duplicate tuples in Q will be eliminated, resulting in R. Let 〈v1, v2,..., vm〉 be a tuple in the relation S. Let p be the probabilistic at1

tribute accuracy for attribute vi. Using D5, one can write p = (AS)m , where AS is the accuracy of relation S. If n (n ≤ m) attributes are selected from S into Q, then the probability of a tuple in Q being accurate before elimination of duplicates is n

given by(AS)m. As such, the estimated total number of accurate tuples in Q before the elimination of duplicates is given by (AS) mn * |S| . Before elimination of duplicates from Q, we assume that Q and S represent the same class of objects as discussed earlier, except that Q provides less number of attributes than S. Therefore, the number of mismember tuples in Q is the same as that

72

Developing a Data Quality Algebra

Chapter 5

of S. As such, the inaccuracy of Q due to mismember tuples is the same as the inaccuracy of S due to mismember tuples. That is, IMQ = IMS. By definition, the inaccuracy of Q due to inaccuracy in the attribute value is given by (4) Let |R| be the size of the resulting relation after elimination of dupllicates from Q, which implies that, |Q| - |R| tuples are eliminated from Q. The set of tuples eliminated may consist of both accurate and inaccurate tuples. By Definition D4, the number of accurate tuples, mismember tuples, and tuples having at least one inaccurate attribute value can be estimated as (|Q| - |R|)* AQ , (|Q| - |R|)* IMQ , and (|Q| - |R|)* IAQ respectively. by

The total number of accurate tuples retained in the resulting relation R is given

(5) Similarly, we can derive IMR = IMS By definition, IAR can be determined as IAR = 1 - (AR + IMR)

(6) (7)

Worst case when error distribution is non-uniform In the worst case, we consider two sub-cases. In the first sub-case, all tuples in Q are inaccurate, which implies that, the accuracy of R is zero. In the second sub-case, some tuples in Q are accurate. In this sub-case, the worst situation occurs when all accurate tuples in Q collapse to a single tuple in R. In this situation, the accuracy is given by

The above accuracy estimate satisfies boundary conditions both P1 and P2 with an assumption that 1<<| S |, which is a reasonable assumption in industrialstrength relational databases.

Illustrative Example

73

Best case when error distribution is non-uniform In the best case, we consider two sub-cases. In the first sub-case, all tuples in Q are accurate. In this sub-case, the accuracy of R is one. In the second sub-case, some tuples in Q are inaccurate. In this sub-case, the best situation occurs when all inaccurate tuples in Q collapse to a single tuple in R. In this situation, the accuracy is given by

The above accuracy estimate satisfies boundary condition both P1 and P2 with an assumption that 1<<|S|. For details of the derivations of Cartesian Product, Union, and Difference, the reader is referred to [13].

ILLUSTRATIVE EXAMPLE For illustration purposes, we use the relations DEPT1_EMP and DEPT2_EMP and the quality-profiles shown in previous sections and reproduced below for convenience. DEPT 1 -EMP

EMP ID EMP NAME 1 Henry Jacob 2 Marshall 3 Alina 4 Roberts 5 Ramesh 6 7 Patel Joseph 8 John 9 Arun 10

DEPT2_EMP

EMP SAL 30,000 32,000 34,000 33,000 50,000 45,000 46,000 55,000 60,000 50,000

EMP ID 1 2 5 6 9 11 12 13 14 15

EMP NAME Henry Jacob Roberts Ramesh John Nancy James Peter Ravi Anil

Quality Profile

Relation Name DEPT1_EMP DEPT2_EMP

Size 10 10

AR 0.7 0.7

IMR 0.1 0.0

IAR 0.2 0.3

EMP SAL 30,000 32,000 55,000 45,000 60,000 39,000 37,000 46,000 55,000 45,000

74

Developing a Data Quality Algebra

Chapter 5

EXAMPLE: Find the names and salaries of employees who are working either in DEPT1 or DEPT2 and making less than $35,000 per year. An algebraic expression to generate the above query is as follows: ΠEMP-NAME,EMP-SALσEMP-SAL<35,000(DEPT1_EMP DEPT2_EMP) The execution of this query can be decomposed into three steps. Let the subquery be the union of DEPT1_EMP and DEPT2_EMP. Let Temp-1 be the result of this union. The relational algebraic expression for this query and its result are shown below: EMP ID EMP NAMEEMP SAL 30,00 Henry 1 2 32,000 Jacob 34,000 3 Marshall 4 Alina 33,000 Roberts 5 50,000 Roberts 5 55,000 6 45,000 Ramesh 7 Patel 46,000 Joseph 55,000 8 9 John 60,000 Arun 10 50,000 Nancy 39,000 11 James 37,000 12 Peter 46,00 13 55,000 Ravi 14 Anil 15 45,000

Temp-1= DEPT1_EMP ≈ DEPT2_EMP

Query Result from the union (Temp-I)

Since the relational operator involved in this query is the union operator, using Equation (11), the accuracy of Temp-1 can be estimated to be 0.6774. Similarly, other parameters of the quality-profile can be estimated. The quality profile of the results of the above subquery is shown below: Relation Name Temp-1

Size 16

AR 0.674

IMR 0.03334

Quality Profile of Temp-1

IAR 0.28831

Illustrative Example

75

The Selection operator is then applied on Temp-1. Let the result of this operation be Temp-2. The relational algebraic expression for this query and its result are shown below:

Temp-2 =

σEMP-SAL<35,00(Temp-1).

EMPID 1 2

EMP-NAME

EMP-SAL

Henry Jacob

30,000 32,000

3 4

Marshall Alina

34,000 33,000

Query Results from the selection (Temp-2)

Since the relational operator involved in this query is the selection operation, using Equation (1), the accuracy of Temp-2 can be estimated. For the selection operation, IMR = IMS, and IAR = IAS. Thus, the quality profile is: Relation Name Temp-2

Size 6

AR 0.6774

IMR 0.03334

IAR 0.28831

Quality Profile of Temp-2

Next the projection operator is applied to Temp-2. The result are shown below: EMP-NAME Henry Jacob Marshall Alina

Temp-3 = ΠEMP-NAME, EMP-SAL (Temp-2).

EMP-SAL 30,000 32,000 34,000 33,000

Query Results from the Projection (Temp-3)

Relation Name Temp-3

Size 4

AR 0.8230

IMR 0.0333

Quality Profile of Temp-3

IAR 0.1437

76

Developing a Data Quality Algebra

Chapter 5

Using Definition D2, we can see that the accuracy of the above sample report generated is 0.75 and the estimation model computes the accuracy of the report as 0.823. This discrepancy is due to the small sample size.

CONCLUSION We have presented a method for estimating the quality (accuracy) of query results derived from underlying base relations in a relational database through a database query. Since the quality of the derived data is a function of the query, the emphasis is on estimating the quality for the output of every operator that could be present in the relational algebra. By postulating the impact on the quality for each operator in theoretical terms, the quality profile of the output can be generated for any arbitrary query comprised of such operators. Some operators increase the degree of potential quality, while others tend to decrease the quality. This analysis, therefore, provides a basis for further research in formulating queries in terms of preferred operators with the objective of attaining the optimized quality profile for a given set of data sources and their corresponding quality profiles. Clearly, the validity of the computed quality is a function of the validity of the quality figures of the underlying base relations. The latter figures obtained via sampling and other techniques are themselves prone to error. In the absence of more stringent information for each base relation, the data quality profile has been assumed to be uniform; however, the impact of non-uniform accuracy profiles has also been studied. The issue of defining rigorous techniques for estimating the idiosyncrasies of individual data parameters is an area requiring further research.

References [1] Ballou, D. P. and H. L. Pazer, "Modeling Data and Process Quality in Multi-input, Multioutput Information Systems," Management Science, 31(2), 1985, pp. 150-162. [2] Ballou, D. P. and H. L. Pazer, "Designing Information Systems to Optimize the Accuracy-Timeliness Tradeoff," Information Systems Research, 6( 1), 1995, pp. 51-72. [3] Ceri, S. and G. Pelagatti, Distributed Databases Principles & Systems. 1st ed. McGrawHill, 1984. [4] Codd, E. F., "A Relational Model of Data for Large Shared Data Banks," Communications of the ACM, 13(6), 1970, pp. 377-387. [5] Codd, E. F., The Relational Model for Database Management: Version 2. AddisonWesley, Reading, MA, 1990.

Conclusion

77

[6] Date, C. J. "Referential Integrity," in Proceedings of the Proceedings of the 7th International Conference on Very Large Data bases. Cannes, France: pp. 2-12, 1981. [7] Date, C. J., An Introduction to Database Systems. 5th ed. Addison-Wesley Systems Programming Series, Addison-Wesley, Reading, 1990. [8] Janson, M., "Data Quality: The Achilles Heel of End-User Computing," Omega Int. J. of Mgmt. Sci., 16(5), 1988, pp. 491-502. [9] Kent, W., Data andReality. North Holland, New York, 1978. [ 10] Klug, A., "Equivalence of relational algebra and relational calculus query languages having aggregate functions," The Journal of ACM, 29, 1982, pp. 699-717. [11] Madnick, S. E., "Challenges in the "on-and-off the ramps" of the Information Superhighway," Journal of Organizational Computing, 1995. [12] Paradice, D. B. and W. L. Fuerst, "An MIS data quality methodology based on optimal error detection," 5(1), 1991, pp. 48-66. [13] Reddy, M. P. and R. Y. Wang. "Estimating Data Accuracy in a Federated Database Environment," in Proceedings of 6th International Conference, CISMOD (Also in Lecture Notes in Computer Science). Bombay, India: pp. I 15-134, 1995. [14] Siegel, M. and S. E. Madnick. "A metadata approach to resolving semantic conflicts," in Proceedings of the proceedings of the 17th International Conference on Very Large Data Bases (VLDB). Barcelona, Spain: pp. 133-145, 1991. [15] Siegel, M., E. Sciore and A. Rosenthal, Using Semantic Values to Facilitate Interoperability Among Heterogeneous Information Systems (No. 3543-93). Context Interchange Project, MIT Sloan School of Management, 1993. [16] Wand, Y. and R. Y. Wang, "Anchoring Data Quality Dimensions in Ontological Foundations," Forthcoming, Communications of the ACM, 1995. [17] Wang, R. Y., H. B. Kon and S. E. Madnick. "Data Quality Requirements Analysis and Modeling," in Proceedings of the 9th International Conference on Data Engineering. Vienna: pp. 670-677, 1993. [18] Wang, R. Y., M. P. Reddy and H. B. Kon, "Toward quality data: An attribute-based approach," Decision Support Systems (DSS), 13, 1995, pp. 349-372. [19] Wang, Y. R. and S. E. Madnick. "A Polygen Model for Heterogeneous Database Systems: The Source Tagging Perspective," in Proceedings of the 16th International Conference on Very Large Data bases (VLDB). Brisbane, Australia: pp. 519-538, 1990.

This page intentionally left blank.

C H A P T E R

6

The MIT Context Interchange Project7

T he popularity and growth of the World Wide Web have dramatically increased the number of information sources available for use and thus the opportunity for important new information-intensive applications such as massive data warehouses, integrated supply chain management, global risk management, and in-transit visibility. Unfortunately, in order for this opportunity to be realized, we must overcome significant challenges with regard to data extraction anddata interpretation. The data extraction problem refers to the difficulty in extracting, easily and automatically, very specific data elements from Web sites for use by operational systems. New technologies, such as XML and Web Querying/Wrapping, offer ©1999 IEEE. Reprinted from the Proceedings of 1999 IEEE Meta-Data Conference, Stuart Madnick, “Metadata Jones and the Tower of Babel: The Challenge of Large-Scale Semantic Heterogeneity.”

7

80

The MIT Context Interchange Project Chapter 6

possible solutions to this problem. The data interpretation problem refers to the existence of heterogeneous contexts, whereby each source of information and potential receiver ofthat information may operate with a different context, leading to large-scalesemanticheterogeneity. As can be readily seen, the data extraction and data interpretation problems are akin to the accessibility and interpretability data quality dimensions, respectively. In this chapter, examples of important context challenges will be presented and the critical role of metadata, in the form of context knowledge, will be discussed.

INTEGRATING HETEROGENEOUS SOURCES

AND

USES

A context is the collection of implicit assumptions about the context definition (i.e., meaning) and context characteristics (i.e., quality) of the information. As a simple example, whereas most US universities grade on a 4.0 scale, MIT uses a 5.0 scale – posing a problem if one is comparing student GPA's. Another typical example might be the extraction of price information from the Web: is the price in Dollars or Yen; and if it is in dollars, is it US dollars or Hong Kong dollars? Does it include taxes? Does it include shipping? and does that match the receiver's assumptions? Today, as we build "information superhighways" to access information from around the organization and around the world, it is increasingly relevant to ask ourselves the following question: will this endeavor succeed or will it be overcome by a "confusion of tongues"? The effective use of metadata can provide an approach to overcoming the challenges. There have been significant research efforts focused on physical information infrastructure, such as establishing high-speed data links to access information distributed throughout the world. It is increasingly obvious, however, that this kind of "physical connectivity" alone is not sufficient since the transfer of bits and bytes is valuable only when information can be efficiently and meaningfully exchanged. These capabilities are essential to providing the "logical connectivity" that is critically needed for dealing with the challenges of the information age. The need for intelligent information integration is important to all informationintensive endeavors, with broad relevancy for global applications, such as Manufacturing (e.g., Integrated Supply Chain Management), Transportation/Logistics (e.g., In-Transit Visibility), Government / Military (e.g., Total Asset Visibility), and Financial Services (e.g., Global Risk Management). Currently, advances in computing and networking technologies allow huge amounts ofdata to be gathered and shared on an unprecedented scale. Unfortunately, these newfound capabilities by themselves are only marginally useful in cases where: 1. The information cannot be easily extracted and gathered from disparate sources,

The Information Integration Challenge

81

2. The information is represented differently with different interpretations, 3. The information must satisfy differing user needs. Some of the extraction and dissemination challenges arise because the information sources may be traditional databases, web sites, or even spreadsheets or electronic mail. Furthermore, the user may originate his or her request in a variety of ways. Even more challenging to the correct interpretation of information is the fact that the sources and users may each assume different semantics or "context". As a trivial example, one source may be assuming measurements in meters whereas another assumes feet. Therefore, in order to exploit the proliferation of information sources that are becoming available, what is required is not only technology, such as the Internet, that will provide "physical connectivity" to information sources, but also "logical connectivity" so that the information can be obtained from disparate sources and can be meaningfully assimilated. This context knowledge is often widely distributed within and across organizations. With the amount and diversity of information sources available, it is necessary to be able to extract and organize the information not only from structured databases but also semi-structured web sources, spreadsheets, and text sources. In addition, solutions adopted to achieve interoperability must be scaleable and extensible and provide decision makers with the appropriate services in an efficient and timely manner in their environments and their applications. Thus, it is important to support the acquisition, organization, and effective intelligent usage of distributed context knowledge. Components of a Context Interchange (COIN) System have been designed and implemented as a basic prototype at the MIT TDQM program.

THE INFORMATION INTEGRATION CHALLENGE As an illustration of the problems created by the disparities underlying the way information is provided, represented, interpreted, and used, consider the example8 depicted in Figure 6.l below. The users wish to answer a fairly common, but important, type of question: "How much funds are left forp roject A?" The calculation in this case is conceptually quite simple; merely subtract the expenses incurred by the 3 regions from the amount of funds that had been allocated (these are all shown on the left side under the heading labeled "Sources").

8Although this particular example, the reader is encouraged to consider the many other similar situations that exist in all disciplines and among all organizations.

82

The MIT Context Interchange Project

Chapter 6

Information Extraction and Dissemination Challenges Information Extraction: Even if we assume that all the necessary information is available electronically via the Internet, it may be on differing media and has different meanings. In our example, the allocated funds are in an Oracle relational database in Singapore, the expenses for Region 1 (USA) are available from a web site, the expenses for Region 2 (UK) are in an Excel spreadsheet, and the expenses from Region 3 (Japan) are provided via a semi-structured electronic mail message. In order to compute the desired answer, the information must be extracted from these varying sources and gathered together. Information Dissemination: Similarly, the actual request may originate in many ways (these are shown on the right side under the heading labeled "End-User Environments & Applications"). A user in the USA may be making this request from a Web browser, a user in the UK may have this request originating from an "embedded SQL query" in a spreadsheet, and a user in Singapore may be collecting this information for data warehousing purposes. Furthermore, this information may be requested and used as part of calculations for arbitrary application programs (e.g., preparation of budgeting reports, generation of exception reports, etc.)

Information Interpretation Challenges Merely subtracting the numbers shown in the expense sources on the left of Figure 6.1 from the allocated number does not produce the "right" answer because different sets of assumptions underlie the representation of the information in the sources. These assumptions, which we call the meaning or context of the information, are often not explicit. In our example, the source contexts are indicated at the far left in Figure 6.1. For the example shown in Figure 6.1, the allocated funds are expressed in 1000's of Singapore dollars, the expenses in Region 2 are expressed in units (1 's) of British pounds excluding the 10% VAT charges, and Region 3 reports its expenses in 100's of Japanese yens. Likewise, the receivers may have their own unique context, shown at the far right in Figure 6.1. A USA user may expect the answer in units (1 's) of US dollars, whereas the Singapore user may wish the answer to be in 1000's of Singapore dollars. The UK user may want the answer in 100's of British pounds including the 10% VAT charges. Under these circumstances, answering even the "simple" question of Figure 6.1 is not so simple anymore. In fact, auxiliary information sources may be needed, such as currency conversion rates, as well as rules on how such conversions should be done (e.g., as of what date). Contextual issues can be much more complex in other situations. For example, the meaning of "net sales" may vary –with "excise taxes" included for government reporting purposes in one context, but excluded for security analysis purposes in

The Information Integration Challenge

83

another. Also, one context may use information for a fiscal year as reported by the company, while another may use a standardized fiscal year to make all companies comparable. Furthermore, there may be multiple users (see right side of Figure 6.1) that might want an answer to such a question; each with their own desired media and meaning (user context profile). Note that a "user" might be a person, an application program, a database, or a data warehouse.

Figure 6.1 : Example Application Illustrating Modular Architecture

The center portions of Figure 6.1 above illustrate the basic components of the Context Interchange System that has been designed and implemented as a limited prototype at the MIT TDQM program. In one sample application, it makes use of several online databases (e.g., Disclosure, WorldScope, DataStream –historical financial information sources), various web sites (e.g., Security APL –current stock exchange prices, Edgar –USA SEC filings, and Olsen –currency conversion information), and semi-structured documents (e.g., Merrill Lynch analyst reports). The financial information needed to answer a question is extracted from these sources, correctly interpreted (involving automatic conversions). It is then integrated and disseminated in various ways, such as into an Excel spreadsheet application of a financial analyst.

84

OVERVIEW

The MIT Context Interchange Project

OF

Chapter 6

THE CONTEXT INTERCHANGE APPROACH

Context Interchange is a mediation approach for semantic integration of disparate (heterogeneous and distributed) information sources. It has been described in [3]. The Context Interchange approach includes not only the mediation infrastructure and services, but also wrapping technology and middleware services for accessing the source information and facilitating the integration of the mediated results into endusers applications.

Context Interchange Architecture The architecture comprises three categories ofcomponents: the wrappers, the mediation services, and the middleware, interface, andfacilitation services. The wrappers are physical and logical gateways providing a uniform access to the disparate sources over the network. Figure 6.2 illustrates the architecture of the Context Interchange (COIN) system. The set of Context Mediation Services is comprised of a Context Mediator, a Query Optimizer and a Query Executioner. The Context Mediator is in charge of the identification and resolution ofpotential semantic conflicts induced by a query. This automatic detection and reconciliation of the conflicts present in different information sources is made possible by a general knowledge of the underlying application domain, as well as the informational content and implicit assumptions associated with the receivers and sources. These bodies of declarative knowledge are represented in the form of a domain model, a set of elevation axioms, and a set of context theories respectively. The result of the mediation is a mediated query. To retrieve the data from the disparate information sources, the mediated query is then transformed into a query execution plan, which is optimized, taking into account the topology of the network of sources and their capabilities. The plan is then executed to retrieve the data from the various sources; results are composed as a message, and sent to the receiver. The middleware, interface, and facilitation services are the services that give users and application programs access to the mediation services. They rely on an Application Programming Interface (API) and a protocol implemented as a standard subset of the Open Data Base Connectivity (ODBC) protocol tunneled into the HyperText Transfer Protocol (HTTP). Examples of interfaces and facilitation services are the Query-By-Example Web interface, which is a point-and-click interface for the construction of ad-hoc queries [4], and the Context ODBC driver [7], which gives any ODBC-compliant Windows 95 or Windows NT application (such as Excel, Access, etc.) access to the mediation infrastructure.

Overview of The Context Interchange Approach

85

Wrapping Wrappers serve as gateways to external information sources for mediation services engines. While information sources vary widely in interface technology and physical data representation, the wrappers should provide a uniform interface to the sources. Two general classes of information sources are: structured data sources, such as traditional relational DBMS's (Oracle and Ingress), and on-line information services, such as Web sites reached though navigable HyperText Markup Language (HTML) pages. COIN wrappers [6] present a common client interface with the appearance of a relational table to the mediation services engine. The protocol used at the wrapper interface is identical to the protocol for accessing mediation services –ODBC tunneled into HTTP. Requests are presented in SQL; whereas results are returned in the form of standard objects, such as HTML tables or JavaScript objects. Because of the common interface at each stage, a user can, in fact, bypass mediation services and directly access raw data from a source through a wrapper. COIN wrappers for relational DBMS's serve as protocol converters. Queries or other access requests are received from the client in COIN protocol. The SQL is extracted and presented to the DBMS using its own API. Query results are then obtained from the DBMS API and delivered to the client using the COIN protocol via HTTP. A generic Web-wrapping technology capable of extracting semi-structured information from Web-services was also developed. The COIN Web-wrapping technology is unique in that it takes advantage of the Hypertext structure of Web-sources and of the underlying structure provided by HTML. A Web service is treated as a collection of static and dynamic pages connected by transitions. Often, information on the Web is not contained on a single page, but rather distributed over a group of pages linked by static (e.g. ) and dynamic hypertext links (e.g.

). The fact that a "service" is located on a single Web-server, or distributed over a number of independently maintained sites, is transparent to the user. Typically, a user may contact the "home page" of the service, click on hypertext links, retrieve some information, fill in and post HTML forms, obtain another piece of information, and so on. The various pieces of information located on one page can be in a pre-structured format, in a semi-structured format, or in unstructured plain text format. What is meant by pre-structured format is a format that is known in advance to the user. This is typically the case of pages using a data representation compliant with a standard, such as the Open Financial Connectivity standard. Where information producers are able to agree on such standard representations, COIN can take advantage of the format guarantees. Semi-structured format includes data presented in a table, a list, a tree, or other structuring organization, but for which the structure is not fully known in advance and must be parsed and analyzed on the fly to locate the data. There are a large number of information sources on the World Wide Web offering semi-structured format data (e.g., CIA fact book, stock exchange quote services, weather reports and weather forecasts, etc.)

86

The MIT Context Interchange Project

Chapter 6

The COIN Web-wrapping technology is based on a high level declarative language for the specification of wrapper interface and actions. This language specifies what information can be extracted from a source. The generic wrapper engine transforms user requests into a plan for extracting the relevant data according to the specification, executes the plan by accessing the source, and organizes and presents the extracted data. The specification language for the generic Web-wrappers allows the definition of a state transition network. The transitions in the network correspond to the hyperlinks in the hypertext. Additionally, the information initially inputted or collected in the preceding stages is carried and is used to define the transitions, fill the parameters of a form, or choose a link among several on a page. On each page, or state of the transition network, the Web-wrapper specification uses patterns (e.g., regular expressions) to identify the location of data to be extracted, the input fields for a form, or the links to other locations. More recently, it has been possible to move beyond regular expression patterns and take advantage of the structure of the information provided by the use of XML tags in a web page. Furthermore, web sites have differing capabilities. Some sites are collections of static pages while others are dynamically created pages based upon specific interactions. It is necessary to take into consideration the specific capabilities and limitations on data retrieval from sites.

Mediation In a heterogeneous and distributed environment, the mediator transforms a query written using terms known to the user or application program (i.e., according to the user's or programmer's assumptions and knowledge) into one or more queries using the terms of the component sources. The individual sub-queries may still involve several sources. Subsequent planning, optimization, and execution phases are needed. Typically, the planning and execution phases will consider the limitations of the sources and the topology and costs of the network. The execution phase is in charge of the scheduling of the query execution plan and the realization of the complementary operations that could not be handled by the sources individually (e.g. a join across sources). The first mediation phase can be described as the rewriting of the query against a "view definition", the view of the disparate information sources that the mediation service provides to the user or application program. The main quality of the mediation approach will depend on its properties with respect to the strategy for the assimilation and definition of the knowledge needed for the construction of this “view definition." Where a large number of independent information sources are accessed (as is now possible with the global information infrastructure), flexibility, scalability, and non-intrusiveness will be of primary importance. Traditional tight-coupling approaches to semantic interoperability rely on the a priori creation of federated views on the heterogeneous information sources. Although they provide good support for data access, they do not scale-up efficiently given the complexity involved in constructing and maintaining a shared schema for a large number of, possibly independ-

Overview of The Context Interchange Approach

87

ently managed and evolving, sources. Loose-coupling approaches rely on the user's intimate knowledge of the semantic conflicts between the sources and the conflict resolution procedures. This flexibility becomes a drawback for scalability when this knowledge grows and changes because more sources are joining the system or some sources are changing.

Figure 6.2: The Architecture of the Context Interchange System

The Context Interchange (COIN) approach is a middle ground between these two approaches. It allows queries to the sources to be mediated, i.e. semantic conflicts to be identified and solved by a context mediator through comparison of contexts associated with the sources and receivers concerned by the queries. It only requires the minimum adoption of a common Domain Model that defines the domain of discourse of the application. The knowledge needed for integration is formally modeled in a COIN framework [2]. The COIN framework is a mathematical structure offering a sound foundation for the realization of the Context Interchange strategy. The COIN framework comprises a data model and a language, called COINL, of the Frame-Logic (FLogic) family [1, 5]. The framework is used to define the different elements needed to implement the strategy in a given application. These elements are: • The Domain Model which is a collection of rich types (semantic types) defining the domain of discourse for the integration strategy.

88

The MIT Context Interchange Project

Chapter 6

• The Elevation Axioms for each source which identify the semantic objects (instances of semantic types) corresponding to source data elements and define integrity constraints specifying general properties of the sources. • The Context Definitions, which define the different interpretations of the semantic objects in the different sources or from a receiver's point of view. Together with additional generic axioms defining the mediation, the Domain Model, the different sets ofElevation Axioms, and the Context Definitions constitute a COINL program. This program controls the query mediation engine. Figure 6.3 illustrates the Context Interchange formal framework.

Figure 6.3: The Context Interchange Formal Framework

Consider the simple example where a user issues the query Q1 below to a source called "security" which provides historical financial data about a stock exchange. The user and the source have different assumptions regarding the interpretation of the data. These assumptions are captured in their respective contexts C1 and C2. The Domain Model defines semantic types such as money amounts, dates, and company identifications. Query Q1 requests the price of the IBM security on March 12, 1995:

Overview of The Context Interchange Approach

(Q1)

89

select security.Price from security where security.Ticker = "IBM" and security.Date = "12/03/95";

The receiver's context C1 assumes money amounts are in French Francs, dates in the European format, and that currency conversions should use the date of the money amount validity. We see immediately that context information is needed to avoid the confusion between March 12 and December 3, 1995. On the other hand, the source context C2 expresses its money amounts in the local currencies of the company, and dates are in American format. The mediation rewrites the query, incorporating the proper currency conversion (as of March 12, 1995) making use of an ancillary source called "cc" for exchange rates, and the proper date format conversion. The resulting mediated query MQ1 is: (MQ1)

select security.Price * cc.Rate from security, cc where security.Ticker = "IBM" and security.Date = "03/12/95" and cc.Expressed = "USD" and cc.Exchanged = "FRF" and cc.AsOfDay = security.Date;

In this example, the domain model will define the various semantic types corresponding to the concepts associated with the data elements manipulated in the application domain. For instance, semantic types capturing notions like money amounts, company financials, or exchange rates need to be defined. Ifsome relationships exist among these semantic types and are relevant from an ontological point of view (as opposed to the peculiarities ofthe structures hosting the data in the sources), they can be represented in the domain model by means of attributes. The following is an excerpt of the domain model of our example in COINL: MoneyAmount: number; CompanyFinancials: MoneyAmount; ExchangeRate: number [ to currency; from currency; asof date].

The elevation axioms define the semantic image of the relations and the data exported by the sources. Below is an excerpt of the elevation axioms for a source exporting a relation Olsen reporting historical data for currency exchange rates (Olsen is an actual Web site, which can be utilized as if it were a relational database through the use of COIN'S Web Wrapping technology). The first rule defines the semantic relation Olsen_semantic. The second rule, defines an ExchangeRateseman-

90

The MIT Context Interchange Project

Chapter 6

tic object. The third rule is an integrity constraint expressing the reversibility of the rate. Olsen_semantic( f_to(To, From, Date), f_from(To, From, Date), f_date(To, From, Date), f_rate (To, From, Date)) &voτ; Olsen(To, From, Date, Rate). f_rate(To, From, Date): ExchangeRate [to f_to(To, From, Date), from f_from(To, From, Date), date f_date(To, From, Date)]. Olsen(To, From, Date, Ratel), Olsen(From, To, Date, Rate2)→ Ratel = 1/Rate2.

The context associated with the sources and the receivers define the modifiers ofthe semantic objects. The modifiers are special attributes dependent on the context and determine the interpretation of the data. They are used for the identification of conflicts during the query mediation. They can be defined by extension (given a value) or by intention (by means of a rule). Several modifiers corresponding to different notions determining the interpretation of a semantic object are associated with it (e.g., the currency and the as-of date of a money amount). Modifiers are declared for all objects of a given semantic type.

X:moneyAmount [[currency.value "FRF"]; [asofdate V] X[report.date

V] ].

Finally, the conversion functions for each modifier locally define the resolution of potential conflicts. The conversion functions can be defined in COINL but are likely, in practical cases, to rely on external services or external procedures. The relevant conversion functions are gathered and composed during mediation to resolve the conflicts. No global or exhaustive pair wise definition of the conflict resolution procedures is needed. The mediated update problem illustrates the potential advantage of the formal logical approach in COIN over traditional view mechanisms for mediation. For a retrieval, either approach can be made to deliver correct results (with more or less effort). The COIN approach, however, holds the knowledge of the semantics of data in each context and across contexts in declarative logical statements separate from the mediation procedure. An update asserts that certain data objects must be made to have certain values in the updater's context.

Conclusion

91

By combining the update assertions with the COIN logical formulation of context semantics, an update mediation algorithm can determine whether is unambiguous and feasible, and if so, what source data updates must be made to achieve the intended results. Ifambiguous or otherwise infeasible, the logical representation may be able to indicate what additional constraints would clarify the updater's intention sufficiently for the update to proceed. The COIN framework can also be applied to important aspects of the source selection problem to characterize the scope of information available from each source and to efficiently rule out irrelevant data sources, thereby speeding up the selection process. For example, a query requesting information about companies with assets lower than $2 million can avoid accessing a particular source based on knowledge of integrity constraints stating that the source only reports information about companies listed in the New York Stock Exchange (NYSE), and that companies must have assets larger than $10 million to be listed in the NYSE. In general, integrity constraints express necessary conditions imposed on data. However, more generally, a notion of completeness degree of the domain of the source with respect to the constraint captures richer semantic information and allows more powerful source selection. For instance, a source could contain exactly or at least all the data verifying the constraint (information about all the companies listed in the NYSE are exhaustively reported in the source).

CONCLUSION The opportunities to make use of diverse information sources are incredible but the challenges are considerable. The effective use ofmetadata can enable us to overcome the challenges and more fully realize the opportunities. A particularly interesting aspect of the context mediation approach described is the use of metadata to describe the expectations of the receiver as well as the semantics assumed by the sources. If these challenges are not addressed directly and effectively, serious consequences might have to be endured, as illustrated by the historical example below. In 1805, the Austrian and Russian Emperors agreed to join forces against Napoleon. The Russians promised that their forces would be in the field in Bavaria by Oct. 20. The Austrian staff planned its campaign based on that date in the Gregorian calendar. Russia, however, still used the ancient Julian calendar, which lagged 10 days behind. The calendar difference allowed Napoleon to surround Austrian General Mack's army at Ulm and force its surrender on Oct. 21, well before the Russian forces could reach him, ultimately setting the stage for Austerlitz. (Source: David Chandler, The Campaigns ofNapoleon, New York: MacMillan 1966, pg. 390.)

92

The MlT Context Interchange Project

Chapter 6

References [1] Dobbie, G. and R. Topor, On the declarative and procedural semantics of deductive object-oriented systems. Journal of Intelligent Information Systems, 4 1995, 193--2 19. [2] Goh, C. (1996). Representing and Reasoning about Semantic Conflicts In Heterogeneous Information System. Ph. D., M.I.T. [3] Goh, C. H., S. Bressan, S. E. Madnick and M. D. Siegel (1996). Context Interchange: Representing and reasoning about data semantics in heterogeneous systems. Secondary Context Interchange: Representing and reasoning about data semantics in heterogeneous systems. (Sloan School Working Paper #3928). Sloan School of Management, M.I.T. [4] Jakobisiak, M. (1996). Programming the web -- design and implementation of a multidatabase browser (CISL WP #96-04). Sloan School of Management, M.I.T. [5] Kifer, M., G. Lausen and J. Wu, Logical foundations of object-oriented and frame-based languages. JACM, 4 1995, 741-843. [6] Qu, J. F. (1996). Data wrapping on the world wide web (CISL WP #96-05). Sloan School of Management, Massachusetts Institute of Technology. [7] Shum, A. (1996). Open Database Connectivity of the Context Interchange System. Master's, Massachusetts Institute of Technology.

C H A P T E R

7

The European Union Data Warehouse Quality Project

T he Data Warehouse Quality (DWQ) project is a cooperative project in the ESPRIT program of the European Union whose aim is the establishment of a foundation for data warehouse quality through the linking of semantic models of data warehouse architecture to explicit models of data quality. In this chapter, we present the Data Warehouse Quality project to illustrate one approach for implementing data quality in data warehousing systems. INTRODUCTION A data warehouse is a collection of technologies aimed at enabling the knowledge worker [executive, manager, or analyst) to make better and faster decisions. The data

94

The European Union Data Warehouse Quality Project

Chapter 7

warehouse is expected to present the right information in the right place at the right time with the right cost in order to support the right decision. Data warehousing has become an important strategy for integrating heterogeneous information sources in organizations and enabling On-Line Analytic Processing (OLAP). The data warehouse movement is a consequence of the observation by W. Inmon and E.F. Codd, in the early 1990’s, that operational-level on-line transaction processing (OLTP) and decision support applications (OLAP) cannot coexist efficiently in the same database environment for two main reasons, both having to do with trade-offs in data quality: 1. Data characteristics: OLTP databases maintain current data in great detail locally for immediate operational usage, whereas OLAP deals with lightly aggregated and often globally reconciled historical data, covering much more than just the current ones. Mixing both causes complex compromises between different degrees of detail, and varying needs for historical information. 2. Transaction characteristics: OLTP emphasizes efficiency of short update transactions each covering a small part of the database, whereas OLAP requires long queries each surveying a large part of the database. Mixing the two causes concurrency control problems. Because a data warehouse caches selected data of interest to a customer group, access becomes faster, cheaper and more effective. By acting as a long-term buffer between OLTP and OLAP (see Figure 7. 1), a data warehouse faces two main issues: (1) how to reconcile the stream of incoming data from multiple heterogeneous legacy sources? and (2) how to customize the derived data storage to specific OLAP applications? The trade-offs driving the design decisions concerning these two issues change continuously with business needs. Therefore, design support and change management are of greatest importance to the success of data warehouse projects.

Figure 7.1 : From Transaction Processing to Analytic Processing

This is a recognized problem in industry that is not solvable without improved formal foundations. Vendors agree that data warehouses cannot be off-the-shelf products. They must be designed and optimized with great attention to the customer

The DWQ Project

95

situation. Traditional database design techniques do not apply since they cannot deal with DW-specific issues such as data source selection, temporal and aggregated data, and controlled redundancy management. Since the wide variety of product and vendor strategies prevents a low-level solution to these design problems at acceptable costs, only an enrichment of metadata services linking heterogeneous implementations constitutes a promising solution. This requires research in the foundations of data warehouse quality. The goal of the DWQ project is the development of semantic foundations that will allow designers of data warehouses to link their choice of deeper models, richer data structures, and rigorous implementation techniques to quality-of-service factors in a systematic manner. This will greatly improve the design, operation, and most importantly the evolution of data warehouse applications. The rest of this chapter is organized as follows. After a brief overview of the project goals in section 2, section 3 presents an architectural framework for data warehousing that makes an explicit distinction between conceptual, logical, and physical design issues and thus clarifies a number of issues related to data warehouse quality. In section 4, the specific DWQ research questions arising from this framework are outlined, and some pointers to related work and preliminary DWQ results are provided. In Section 5, a demonstration showing the benefits of the tools developed as part of the DWQ approach is presented. Section 6 provides a summary and conclusions.

THE DWQ PROJECT In this section, we briefly review the structure of the DWQ project, the basic components of data warehouse solutions, and the linkage to formal quality models.

DWQ Project Structure DWQ is a three-year cooperative research project (1996-1999) funded by the European Communities under the Reactive Long Term Research Branch of their ESPRIT IV R&D Program. Partners include the National Technical University of Athens, Greece, Aachen University of Technology, Germany, the German AI Research Center DFKI, INRIA Rocquencourt, France, the University ofRome, Italy, and the IRST Research Center, Italy. The project closely cooperates with Software AG, a leading European vendor of data warehouse products and solutions and also with major ongoing data warehouse efforts within such organizations as the Italian Telecom, the French Ministry of Environment, the French Institute for Environment (IFEN), Air France, and the Office of Statistics of the City of Cologne in Germany.

96

The European Union Data Warehouse Quality Project

Chapter 7

DWQ Objectives The research objectives of the DWQ project address three critical areas where quality factors are of central importance to data warehousing. The aim is to: 1. Enrich the semantics of meta-databases with formal models of information quality in order to enable adaptive and quantitative design optimization of data warehouses; 2. Enrich the semantics of information resource models to enable more incremental change propagation and conflict resolution; and 3. Enrich the semantics of data warehouse schema models to enable designers and query optimizers to take explicit advantage of the temporal, spatial, and aggregate nature of data warehouse data. The project is expected to investigate the whole spectrum of data warehouse modeling, design, and development issues, including the DWQ framework and system architectures, as well as the data warehouse meta-modeling, design methods and languages, and optimization. After developing an initial reference model, the results will be delivered in two stages to enable effective project control and coherence. The first group of results, which will include enriched formal meta-models for describing the static architecture of a DW, will demonstrate how these enriched foundations are used in data warehouse operation. The corresponding methodologies and tools include architecture modeling facilities to address data warehouse-specific issues such as resolution of multiple sources and management of partially aggregated multidimensional data, as well as semantics-based methods for query optimization and incremental update propagation. The second group of results focuses on enhancing these enriched models with tools that support the evolution and optimization of data warehouse applications under changing quality goals. The corresponding methodologies and tools include: • Evolution operators which document the link between design decisions and quality factors, • Reasoning methods which analyze and optimize view definitions with multi-dimensional aggregated data, and allow efficient quality control in bulk data reconciliation from new sources, and • Quantitative techniques that optimize data source selection, integration strategies, and redundant view materialization with respect to given quality criteria, especially performance criteria.

The DWQ Project

97

Data Warehouse Components The DWQ project will provide a neutral architectural reference model covering the design, the setting-up, the operation, the maintenance, and the evolution of data warehouses. Figure 7.2 illustrates the basic components and their relationships as seen in current practice. The components used in this figure can be briefly described as follows: • Sources represent any data stores whose content can be materialized in the data warehouse, • Wrappers are used to load the source data into the data warehouse, • Destination databases are the data warehouse(s) and data marts used, • Meta-database is a repository of information about other components, e.g. the schema of the source data, • Agents are used for administration (data warehouse design, scheduler for initiating updates, etc.) and • Clients are used to display the data, for example statistics packages.

Figure 7.2: Sample structure of a Data Warehouse

98

The European Union Data Warehouse Quality Project Chapter 7

The detailed elaboration of these components and relationships can be filled according to many different user requirements and modeling techniques. The DWQ results in this area concern a general architecture-modeling framework. The simplest architecture will only consist of some source databases, a central data warehouse, and several clients. Because data warehouse applications are becoming more complex, data warehouses are being built using multi-tier architectures to increase performance. In such cases, there is not only one central data warehouse but also data marts that allow the data to be placed closer to the end-user.

The Linkage to Data Quality DWQ provides assistance to data warehouse designers by linking the main components of a data warehouse reference architecture to a formal model of data quality. The model has been specialized for use with data warehousing from a quality goal hierarchy proposed in [29]. Main differences to the initial model lie in the greater emphasis on historical as well as aggregated data. Figure 7.3 illustrates the hierarchy of quality factors used.

Figure 7.3: Quality factors in Data Warehousing

A data quality policy is the overall intention and direction of an organization with respect to issues concerning the quality of data products. Data quality management is the management function that determines and implements the data quality policy. A data quality system encompasses the organizational structure, responsibilities, procedures, processes and resources for implementing data quality management.

The DWQ Project

99

Data quality control is a set of operational techniques and activities that are used to attain the quality required for a data product. Data quality assurance includes all the planned and systematic actions necessary to provide adequate confidence that a data product will satisfy a given set of quality requirements. Consequently, the problem that arises is the modeling and measurement of the quality of the data warehouse. Because a data warehouse is composed of several subsystems and associated processes, there must be a mapping between the data warehouse components and the quality model. Furthermore, measures for the quality indicators and a methodology for designing quality for specific data warehouses must be developed. A closer examination of the hierarchy of quality factors reveals several relationships between quality parameters and the designloperational aspects of data warehouses. Some ofthese relationships are shown in Figure 7.4.

Figure 7.4: Linking quality factors to Data Warehouse tasks

The goal of the DWQ project is to investigate these relationships in a systematic way and to propose formal solutions that will help in data warehouse design and development.

100

The European Union Data Warehouse Quality Project Chapter 7

DWQ ARCHITECTURAL FRAMEWORK A detailed look at the literature on data warehousing leads us to interpret figures 7.1 and 7.2 from three different perspectives: a conceptual (or business) perspective, a logical (or data modeling) perspective, and a physical (or distributed information flow) perspective. Any given data warehouse component or substructure can be analyzed from all three perspectives. Moreover, in the design, operation, and especially evolution of data warehouses, it is crucial that these three perspectives are maintained such that they are consistent with each other. Also, quality factors are often associated with specific perspectives or with specific relationships between perspectives. As can be seen in Figure 7.5, there are 9 data store component types, linked by at least 12 kinds ofrelationships maintained by data warehouse agents. Following is a discussion of each perspective. Note that the discussion is purely at the schema level. Each data warehouse component is an instance of all three perspectives.

Figure 7.5: Three aspects of Data Warehouse architecture

DWQ Architectural Framework

101

The Conceptual Perspective From a strategic management point of view, a data warehouse can be used to impose an overall business perspective –a conceptual enterprise model –on the information resources and analysis tasks of an organization. This enterprise model can be thought of as a class structure where the organization is simply an instance that the data warehouse maintains information about. Thus it is not part of the data warehouse itself. Instead, the data warehouse architecture is a network of direct and derived views (i.e., recorded observations) on this reality. The schema of information sources is defined as a collection of pre-materialized conceptual views on the enterprise model, not vice versa! This important observation has already been made in the Information Manifold project at AT&T [ 18]. On the client side, the interests of user groups (such as analysts) can also be described as conceptual views on the enterprise models. However, these client views cannot be satisfied by the direct observation of reality; instead, they must rely on existing materialized views, i.e. the information sources. As a consequence, we have the problem of mapping conceptual client views to conceptual source views –a theorem proving problem, where the proofs correspond to the queries to be asked by client views on information sources. Furthermore, given the quality factors of performance and security, client views are often not allowed to access the sources directly. Therefore, a set of intermediate conceptual views on the enterprise model is often constructed –it is the conceptual schema of the data warehouse. This set must have the formal property that it allows all expected client views to get answers to their queries, preferably the same they would get ifthey would access the source views directly. In terms of data warehouse quality, the enterprise model defines a theory of the organization. Actual observations (i.e. information sources) can be evaluated for quality factors such as accuracy, timeliness, and completeness with respect to this theory. Moreover, the fact that data warehouse views are located between client views and source views can have both a positive and a negative impact on information quality. The positive impact is achieved by using the enterprise model for data cleaning and evidence integration –probably the single most important contribution of the conceptual level in data warehouse practice. The potential negative impact is due to delays in updating the materialized data warehouse views (timeliness of information usually deteriorates in comparison with OLAP data), and by limitations of expressiveness in the conceptual model that may not include all linguistic facilities for all sources.

The Logical Perspective The logical perspective conceives a data warehouse from the viewpoint of the actual data models involved. Researchers and practitioners who follow this perspective consider a data warehouse simply a collection of materialized views on top of each

102

The European Union Data Warehouse Quality Project Chapter 7

other, based on existing information sources. Examples of projects focusing on the logical perspective include the TSIMMIS project at Stanford University [5, 27], the Squirrel project at the University of Colorado [3 1] and the WHIPS project at Stanford University [11, 30]. On the source side, the main problems addressed in this perspective include the wrapping of heterogeneously represented data sources and the integration of data from multiple sources into particular views. These problems are interpreted both in terms of initial query processing (also called bulk loading in this context) and in terms of incremental maintenance. Particular attention has been paid to so-called self-maintainable warehouse views [23] where the data warehouse itself contains sufficient information to maintain all its views, given just the changes in the base data (but no additional back requests to the OLAP system to see how these changes propagate into the views). On the client side, the two questions of view-based query optimization and incremental update propagation remain central but the focus shifts because user data models are different from the source data models. In particular, the introduction of multi-dimensional data models with aggregation and summarization has a strong impact on algorithms and materialization strategies [6, 12, 19, 24]. Data warehouse products differ with regard to where they introduce these complexities, While MOLAP products already keep the data warehouse views themselves as multi-dimensional data structures, ROLAP products use relational databases to store the data warehouse views and make the transformation to multidimensional data models only in the mapping to client views. Quality factors impacted by the logical perspective concern mainly the coverage of data models at the source and client side, and thus the factor of accessibility of information as well as the interpretability on the client side. In addition, logic-level optimization of queries and updates have a strong impact on the amount of data transferred and thus on the performance of the system. For example, in many large organizations, a good incremental view maintenance algorithm can reduce the daily data transfer from a number of gigabytes to a small number of megabytes, thus also reducing the time window in which OLTP systems are unavailable for transaction processing due to data warehouse loading (24x7 availability).

The Physical Perspective The physical perspective interprets the data warehouse architecture as a network of data stores, data transformers, and communication channels, aiming at the quality factors of reliability and performance in the presence of very large amounts of slowly changing data. On the source side, minimizing interference with OLTP systems while maintaining a reasonable degree of actuality in the data is a major concern. Typical research questions include replication policies, recovery from broken bulk transfers, and decisions as to what components should certain functions be allocated. For example, when reconciling data differences, this could be done in the data warehouse

Research Issues

103

itself, or it could be outsourced to mediator agents with local caches. The SourcePoint tool, marketed by Software AG, is an example of a commercial loading environment that allows a lot of customization at the physical level, but could benefit from a more declarative approach in linking to the logical and conceptual perspectives. On the client side, a central question is how to configure a network of prematerialized views in order to minimize average response times for a given mix of updates and queries. As shown in Figure 7.2, this can involve the usage of tailored data marts as well as a central data warehouse. It can also involve client-side caching of intermediate results to minimize data transfer.

Relationships between the Perspectives Generally speaking, the physical perspective has been mostly explored by industry, whereas the conceptual perspective is mainly approached by research. Research and practice do meet at the logical level, but with different emphasis. Given the quest for quality in data warehousing, these streams of interest are bound to merge. The steps from conceptual to logical to physical design are common to all database design tasks; however, they have not yet been thoroughly investigated for the special case of data warehousing. Basically, when going from conceptual to logical, we face two problems: (a) the move from a rich conceptual model to an impoverished data model such as the relational one, and (b) the fragmentation of a coherent conceptual schema into individual logical data structures. In DWQ, what is being explored is the hypothesis that the link between logical and conceptual can be defined in a very similar manner as the link between conceptual source views and the conceptual data warehouse schema, namely through special kinds of views on the conceptual schema. Going from logical to physical requires the allocation of the identified data fragments and transformation tasks to a network of distributed, cooperating software agents. In this step, considerations of reliability and performance must be added, while semantic considerations must be indirectly expressed through mechanisms that enforce them. When studying the specific tasks that need to be addressed in adding quality to data warehousing, techniques from all three perspectives are often needed. The next section presents the DWQ approach to each of these tasks.

RESEARCH ISSUES The DWQ project research is based on the framework shown in Figure 7.2. The project focuses its research effort on the issues described in the rest of this section and investigates the relationships between these issues and data quality factors.

104

The European Union Data Warehouse Quality Project

Chapter 7

Rich Data Warehouse Architecture Modeling Languages Although many data warehouses have been built, there is no common methodology that supports database system administrators in designing and evolving a data warehouse. The problem with architecture models for data warehouses is that practice has preceded research and continues to do so. The sheer amount of data to be manipulated under tight constraints forces the industry to quickly introduce new implementation methods. For example, caching and logging are used to recover from failures in the data transfer between data warehouse components. Such methods are typically implementation-oriented. Consequently, the task of providing an abstract model of the architecture becomes more difficult. The following questions come up: • • • • •

What kinds of components are used in a data warehouse environment? What architectures are used for data warehouses? How do the components exchange (meta-) data? What is the role of metadata in a data warehouse? How can quality factors and architecture be combined?

The strategy for dealing with this dynamic situation in data warehouse architectures is to use a representation language whose features can be adapted to an evolving understanding of what an architecture for data warehouses is about. The framework uses meta-models for encoding the key concepts and their semantics. Such meta-models can be updated in the same way as database objects. Formally, the reference model corresponds to the schema structure of the meta-database that controls the usually distributed and heterogeneous set of data warehouse components. Therefore, it is the essential starting point for design and operational optimization. Expressiveness and services of the meta-data schema are crucial for data warehouse quality. The purpose of architecture models is to provide an expressive, semantically well-defined, and computationally well-understood meta-modeling language. The outcome is a unified terminology that forms the basis for the repository for data warehouse administration. The architectural reference model is formally described in this language and its practical usefulness will be evaluated through a prototype implementation, extending currently existing meta-modeling tools in the consortium. The structural data warehouse framework is linked to the quality hierarchy shown in Figure 7.3 through traceability links, following the recent work in the representation of non-functional requirements in requirements engineering. The starting point for the linguistic framework is the earlier research in description logics and logic-based meta-modeling facilities which allow a data warehouse schema to be treated as a set of interacting knowledge bases. Projects such as COMPULOG, AIMS, and CLASSIC have identified description as a promising candidate for rich schema languages for schema integration and inter-view relationship handling, as well as for query optimization and explanation. However, application experiments in database interoperation (e.g. SIMS at ISI, AIMS and MEDWIS in

Research Issues

105

Europe) and data mining (e.g. IMACS at Bell Labs) have shown that the expressiveness of these formalisms is still unsatisfactory with respect to specific areas. Complementing the traditional multi-database issues which are addressed in existing ESPRIT projects such as IRO-DB, DWQ focuses on modeling and reasoning support for the two central data warehouse mapping problems shown in Figure 7.1, namely extraction and integration, and aggregation and customization. In both cases, the main advance is the ability to reason about planned integration and aggregation tasks at the schema level, thus facilitating the planning of the integration process and making visible the quality of data cleaning and the relationships of overlapping derived views.

Data Extraction and Reconciliation Data extraction and reconciliation are still carried out largely on an intuitive basis in real-life applications. Existing automated tools do not offer choices in the quality of service. Because the integration process is not sufficiently detailed and documented, the integration decisions that are taken are usually very difficult to understand and evaluate. DWQ provides a coherent, methodological, and tool-based support for the integration activity. The idea of declaratively specifying and storing integration knowledge will be of special importance in supporting high quality incremental integration and making all relevant metadata available. First, reconciliation is a task at the schema level, similar to the traditional task of view integration, but with a richer integration language and therefore with more opportunities for checking the consistency and completeness of data [4, 13]. Second, tools based on the proposed language extensions will facilitate the arduous task of instance-level data migration from the sources to the warehouse, such that a larger portion ofinconsistencies, incompatibilities, and missing information can be detected automatically. The integration methodology will provide specific guidelines for controlling the quality of both the integration process and the product of the integration activity.

Data Aggregation and Customization To facilitate complex analyses and visualization, the data in a warehouse are organized according to the Multi-dimensional data model [10]. Multi-dimensional data modeling means the partial aggregation of warehouse data under many different criteria, within tuples but also, for example, across time (e.g. time series analysis) and geographical areas. A generic user interface toolkit for such multi-dimensional models is presented in [8]. By combining and extending earlier results from the fields of statistical databases with description logics and constraint logic programming, DWQ enriches schema languages to allow the representation oftime, space, and numerical/financial domains as well as aggregates over these domains [9]. “Aggregation’refers to the

106

The European Union Data Warehouse Quality Project

Chapter 7

grouping of data by some criteria, followed by the application of a computational function (sum, average, spline, trend,...) to each group. This will enable design-time analysis and rapid adaptability of data warehouses, thus promoting the quality goals of relevance, access to non-volatile historical data, and improved consistency and completeness. The semantic enrichment developed in DWQ not only enhances data warehouse design but can also be used to optimize data warehouse operation by providing reasoning facilities for semantic query optimization (improving accessibility) and for more precise and better controlled incremental change propagation (improving timeliness). While basic mechanisms stem mostly from active databases, reasoning and optimization techniques on top of these basic services use AI-based reasoning techniques together with quantitative database design knowledge.

Query Optimization Data warehouses provide challenges to existing query processing technologies for the following reason: typical queries over data warehouses require costly aggregation over huge sets of data. While OLAP users pose many queries and expect short response times; users that explore the information content of a data warehouse apply sophisticated strategies (“drill down,” “zoom out”) and demand query modes like hypothetical (“what if’) and imprecise (“fuzzy”)querying that are beyond the capabilities of SQL based systems. Commercial approaches fail to make use of the semantic structure of the data in a warehouse, and rather concentrate on parallelism or make heavy use of traditional optimization techniques such as generating indices or choosing low cost access paths. As a support for OLAP, intermediate aggregate results are pre-computed and stored as views. The approach used in DWQ exploits knowledge and reasoning about the structure of the data warehouse to process queries efficiently and to support the user in organizing the space of queries that are of interest to him. There are two kinds of meta-knowledge in the data warehouse that are relevant: integrity constraints expressed in rich schema languages and knowledge about redundancies in the way information is stored. Both kinds of knowledge will be used simultaneously for optimization. The starting point for representing schemas and queries are variants of so-called description logics [1, 2]. Optimization for nested queries with aggregates transforms a query in an equivalent one that is cheaper to compute. Techniques for constraint pushing will prune the set of data to be considered for aggregation. Integrity constraints will be used to establish the equivalence of queries that are not syntactically similar. Other techniques will reformulate queries in such a way that materialized views are used instead of recomputing previous results. To accomplish its task for queries with aggregation, the optimizer will be capable of reasoning about complex relationships between the groupings over which the aggregation takes place. Finally, these basic

Research Issues

107

techniques are embedded into complex strategies to support OLAP users that formulate many related queries and apply advanced query modes.

Update Propagation Updates to information sources first need to be controlled with respect to the integrity constraints specified during the design of the data warehouse and derived views. A constraint may state conditions that must be satisfied by the data in an information source in order for it to be of a quality relevant to the data warehouse. A constraint may also express conditions over several information sources that help resolve conflicts during the extraction and integration process. Thus, the update of an information source database may degrade the quality of information. Violations of constraints must be handled appropriately (e.g., by sending messages or creating alternative time-stamped versions of the updated data.) Updates that meet the quality requirements defined by integrity constraints must then be propagated towards the views defined at the data warehouse and user level. This propagation must be done efficiently in an incremental fashion [22, 25, 32]. The re-computation can take advantage of useful cached views that record intermediate results. The decision to create such views depends on an analysis of both the data warehouse query workload and the update activity at the information sources. This requires the definition of new design optimization algorithms that take into account these activities and the characteristics of the constraints that are considered for optimization. Integrity constraints across heterogeneous information sources are currently managed manually, resulting in unreliable data. At best, a common reference data model enforces the satisfaction of some structural and quality constraints. However, this approach is quite rigid and does not enable an easy integration of other information sources. By contrast, the DWQ approach suggests a flexible and declarative definition of constraints during the design phase. Constraints will be mapped into active rules, which are increasingly accepted as a suitable base mechanism for enforcing such constraints. Building on earlier results concerning the generation and effective management of large rule bases, DWQ will make significant contributions towards the distributed maintenance and execution ofactive rules in a heterogeneous network. A first approach is reported in [21]. A final important aspect of data warehousing is its ability to evolve with the user and organization needs. DWQ will develop methodologies for specifying redesign and evolution parameters and processes to allow a data warehouse to follow the dynamics of the underlying information sources as well the user needs.

Schema and Instance Evolution The static architecture, integration, and aggregation modeling languages will be augmented with evolution operators that support controlled change in the data ware-

108

The European Union Data Warehouse Quality Project

Chapter 7

house structure. This includes the addition, deletion, or re-evaluation of information sources, as well as the efficient migration between different structures of derived aggregated data, and the tracing of such changes across the data warehouse architecture. Taken together, these tools will evolve the vision of a data warehouse from a passive buffer between operational and analytic processing towards that of a competitive information supermarket that carefully selects and prepares its products, packages and displays them nicely, and continuously adapts to customer demands in a cost-efficient manner.

Quantitative Design Optimization The evolution operators provide the means to control and trace the process of evolving a data warehouse application. However, many steps involve options such as what information sources to use, what update policies to select, and what views to materialize. DWQ will help the data warehouse manager evaluate quantitatively these options and find an optimal balance among the different policies involved, prior to executing the evolution steps. In particular, policies will be developed for optimized view materialization based on workload parameters (sets of expected queries and updates). At the same time, the enriched metadata representation and reasoning facilities used in the data warehouse design will be exploited in order to provide rigorous means ofcontrolling and achieving evolution. The tools will exploit the potential of subsumption reasoning in order to determine whether certain information sources can substitute each other and whether certain materialized views can be reused to materialize other views cheaply. The availability of such subsumption reasoners will significantly change the cost trade-offs, making evolution easier than is currently feasible. An initial quantitative model has been proposed in [26]. In addition, a demonstration showing the benefits of the tools developed as part of the DWQ approach is presented in the next section. The DWQ demonstration uses a simplified version of a real-world application to a data warehouse project at Telecom Italia [ 17].

OVERVIEW OF THE DWQ DEMONSTRATION The demonstration will follow roughly the process indicated in Figure 7.6. In this figure, the dark-gray boxes indicate logical data objects, such as relations, queries, or materialized (possibly multi-dimensional) views. The light-gray boxes describe conceptual models. In DWQ, these are externally represented as extended EntityRelationship models and are internally modeled using Description Logic formalisms from artificial intelligence to allow for subsumption reasoning. Steps 1 though 6 represent the extended design approach shown in the demonstration. The two ovals describe related support at the operational level, focusing on the two key problems of aggregate query optimization and view refreshment. The whole process is adminis-

Overview of the DWQ demonstration

109

tered through a metadata repository that has been implemented using the ConceptBase system [15]. In the following subsections, we briefly describe the main steps of the demonstration and point to literature where more details about the underlying theory or industrial applications can be found.

Figure 7.6: Structure of the demonstration

Source Integration (Steps 1, 2) The DWQ approach to source integration is incremental: whenever a new portion of a source is taken into account, the new information is integrated with an “Enterprise Model”, and the necessary new relationships are added. Thus, the Enterprise Model provides a consolidated view of the concepts and the relationships that are important to the enterprise and have been currently analyzed. Such a view is subject to changes and additions as the analysis ofthe information sources proceeds. The main concepts used in the DWQ approach are shown in Figure 7.7:

110

The European Union Data Warehouse Quality Project

Chapter 7

• The Enterprise Model is a conceptual representation of the global concepts and relationships that are of interest to the data warehouse application. It provides a consolidated view of the concepts and relationships that are important to the enterprise and are currently being analyzed. Such a view is subject to changes and additions as the analysis of the information sources proceeds. The Description Logic formalism used is general and powerful enough to express the usual database models, such as the Entity-Relationship Model and the Relational Model. Moreover, there are suitable inference techniques associated with the formalism that allow several reasoning services on the representation to be carried out. A graphical interface hides this formalism from the user of the DWQ tools. • The Source Model for a given information source S is a conceptual representation of the data residing in s. Again, the approach used does not require a source to be fully analyzed and conceptualized. Source Models are expressed by means of the same formalism used for the Enterprise Model. • The notion ofinterdependency is a central one in this approach. Since the sources are of interest in the overall architecture, integration does not simply mean producing the Enterprise Model, but rather being able to establish the correct relationships both between the Source Models and the Enterprise Model, and also between the various Source Models. The notion of interdependency is formalized by means of inter-model assertions, introduced in [4]. An inter-model assertion states that one object (i.e., class, entity, or relation) belonging to a certain Model (either the Enterprise or a Source Model) is always a subset of an object belonging to another Model. This simple declarative mechanism has been shown to be extremely effective in establishing relationships among different database schemas (see also [14]). Again, a logic-based formalism is used to express inter-model assertions, and the associated inference techniques provide a means for reasoning about interdependencies among models. • The logical content of each source S, called the Source Schema, is provided in terms of a set of definitions of relations, each one expressed in terms of a query over the Source Model of S. The logical content of a source represents the structure of data expressed in terms of a logical data model (in the demonstration, the Relational Model is used). In the DWQ framework, specifying the connection with the conceptual representation of the source provides such a structure. In other words, the logical content of a source S, or of a portion thereof, is described in terms of a view over the Source Model associated with S (and, therefore,

Overview of the DWQ demonstration

111

of the Conceptual Data Warehouse Model). Wrappers map physical structures to logical structures. • The logical content of the materialized views constituting the Data Warehouse, called the Data Warehouse Schema, is provided in terms of a set of definitions of relations, each one expressed in terms of a query over the Conceptual Data Warehouse Model. Similar to the sources, each portion of the Data Warehouse Schema is described in terms of a view over the Conceptual Data Warehouse Model. Suitable mechanisms called mediators allow us to specify how a view is actually materialized starting from the data in the sources.

Figure 7.7: Architecture for Data Integration

The demonstration shows the following tasks within steps 1 and 2 of Figure 7.6 for a few Telecom Italia database sources related to contracts: • Enterprise and Source Model construction: the Source Model corresponding to the new source is produced, if not available. Analogously, the conceptual model of the enterprise is produced, if not available.

112

The European Union Data Warehouse Quality Project Chapter 7

• Source Model integration: the Source Model is integrated into the Conceptual Data Warehouse Model. This can lead to changes both to the Source Models, and to the Enterprise Model. Moreover, inter-model assertions between the Enterprise Model and the Source Models and between the new source and the existing sources are added to the Conceptual Data Warehouse Model. As illustrated in the screenshot of Figure 7.8, the designer can specify such inter-model assertions graphically and can invoke various automated analyses supported by the Description Logic formalization. • Source and Data Warehouse Schema specification: the Source Schema corresponding to the new source (or, corresponding to a new portion of the source) is produced. On the basis of the new analyzed source, an analysis is carried out on whether the Data Warehouse Schema should be restructured and/or modified. In all of these tasks, the metadata repository stores the values of the quality factors involved in source and data integration, and helps to analyze the quality of the design choices. The Quality Factors of the Conceptual Data Warehouse Model and the various schemas are evaluated and a restructuring of the Models and the schemas is accomplished to match the required criteria.

Figure 7.8: Screenshot illustrating graphical source integration

Overview of the DWQ demonstration

113

Aggregation and OLAP Query Generation (Steps 3,4) While this approach to source integration formalizes the supply side of the data warehouse, the next two steps ofthe approach consider the client side. The basic idea is symmetric to the one for source integration and will therefore be described in less detail. The conceptual modeling approach underlying the enterprise and source models, including the underlying modeling and reasoning tools, has been extended to the case where concepts are organized into aggregates along multiple dimensions with multi-hierarchy structures [7]. That is, data warehouse designers can define multidimensional and hierarchical views over the enterprise conceptual model in order to express the interests of certain data warehouse client profiles, without losing the advantages ofconsistency and completeness checking as well as the semantic optimization provided by the conceptual modeling approach. The “conceptual data cubes” thus defined can be implemented directly by MOLAP data models. This approach is, for example, followed by the German data warehouse vendor and DWQ cooperation partner GMI with significant success for customers such as Unilever or Deutsche Bank. A second commercialization of the approach, using Lotus Notes as a client-side data representation, has been described in [16]. The demonstration uses a third kind of mapping, the ROLAP mapping to relations. Faithful representation of client views requires a careful design of an OLAP relational algebra, together with the corresponding rewritings to underlying star schemas. The demonstration shows the approach proposed in [28].

Design Optimization and Data Reconciliation (Steps 5, 6) Viewed from the logical level, this conceptually controlled approach to source integration has created a schema of relations implementing the enterprise model. This schema could be implemented as what has been called an operational data store (ODs). In a data warehouse with lots of updates and completely unpredictable queries, this would be the appropriate view materialization. Similarly, the mapping of multi-dimensional aggregates to ROLAP queries creates a set of view definitions (queries). The materialization of these queries would be the optimal solutions (storage space permitting!) in a query-only data warehouse with hardly any updates. Typical data warehouses have less extreme usage patterns and therefore require a compromise between these two view materialization strategies. The DWQ demonstration shows the solution to this combinatorial optimization problem, which has been proposed in [20, 26]. Besides the optimization algorithms themselves, it is interesting to note that this physical-level optimization is fully integrated with the conceptual modeling approaches because it works on their outcomes. Conversely, the resulting optimal design is now implemented by data integration and reconciliation algorithms again derived from the conceptual perspective. The views to be materialized are initially

114

The European Union Data Warehouse Quality Project

Chapter 7

defined over the ODS relations. There can be several qualitatively different, possibly conflicting ways to actually materialize these ODS relations from the existing sources which are generated by a further set of rewritings that can be derived from the source integration definitions [3]. The problem of data integration and reconciliation arises when data passes from the application-oriented environment to the data warehouse. During the transfer of data, possible inconsistencies and redundancies are resolved, so that the warehouse is able to provide an integrated and reconciled view of data of the organization. In the DWQ methodology, data reconciliation is based on (1) specifying how the relations in the Data Warehouse Schema are linked to the relations in the Source Schemas through the notion of Inter-schema Correspondences, and (2) designing suitable mediators for every relation in the Data Warehouse Schema. In step (1), inter-schema correspondences are used to declaratively specify the correspondences between data in different schemas (either source schemas or data warehouse schema). Inter-schema Correspondences are defined in terms of relational tables, similarly to the case of the relations describing the sources at the logical level. A distinction is made between three types of correspondences, namely Conversion, Matching, and Reconciliation Correspondences. By virtue of such correspondences, the designer can specify the different forms of data conflicts that may exist between the sources, and can anticipate methods for solving such conflicts when loading the data warehouse. In step (2), the methodology aims to produce, for every relation in the Data Warehouse Schema, a specification of the corresponding mediator, which determines how the tuples of such a relation should be constructed from a suitable set of tuples extracted from the relations stored in the sources.

SUMMARY

AND

CONCLUSIONS

The Data Warehouse Quality project aims to improve the quality of data warehouse design and operation through the systematic enrichment of the semantic foundations of data warehousing. Logic-based knowledge representation and reasoning techniques were developed to control accuracy, consistency, and completeness via advanced conceptual modeling techniques for source integration, data reconciliation, and multi-dimensional aggregation. This is complemented by quantitative optimization techniques for view materialization, optimizing timeliness and responsiveness without losing the semantic advantages from the conceptual approach. At the operational level, query rewriting and materialization refreshment algorithms exploit the knowledge developed at design time. The DWQ project develops techniques and tools to support the rigorous design and operation of data warehouses. It uses well-defined data quality factors and a rich semantic approach in bringing together enabling technologies. The potential of such an approach has been demonstrated already by successful commercial usage of the ConceptBase metamodeling tool developed in the COMPULOG project.

Summary and Conclusions

115

Data warehouse development time for a given level of quality will be significantly reduced and adaptation to changing user demands will be facilitated. There is a high demand for design tools for distributed databases that is not satisfied by current products. DWQ has the potential to satisfy this demand for an increasingly important market segment. In addition, a host of quality-enhanced query and update services can be derived from DWQ results. Prototypes of specific models and tools developed in the project will be experimented with in various industry and public administration settings, in order to gain reference experiences for industrial uptake of project results. The results of DWQ are demonstrated with a set of prototype tools that are linked together and are embedded into an environment consisting of a real-world application at Telecom Italia. The demonstration shows the interplay between quality factors and design options in data warehousing as well as the benefits gained from the DWQ approach.

References: [1] Baader, F. and P. Hanschke. "A schema for integrating concrete domains into concept languages," in Proceedings of Twelfth International Joint Conference on Artificial Intelligence (IJCAI-91). Sydney, Australia: pp. 452-457, 1991. [2] Buchheit, M., M. A. Jeusfeld, W. Nutt and M. Staudt, "Subsumption of queries over object-oriented databases," Information Systems, 19(1), 1994. [3] Calvanese, D., G. D. Giacomo, M. Lenzerini, D. Nardi and R. Rosati. "A principled approach to data integration and reconciliation in data warehousing," in Proceedings of Int. Workshop on Design and Management of Data Warehouses (DMDW'99). Heidelberg, Germany: pp. 1999. [4] Catarci, T. and M. Lenzerini, "Representing and using interschema knowledge in cooperative information systems," Journal of Intelligent and Cooperative Information Systems, 2(4), 1993, pp. 375-398. [5] Chawathe, S., H. Garcia-Molina, J. Hammer, K. Ireland, Y. Papakonstantinou, J. Ullman and J. Widom. "The TSIMMIS project: Integration of heterogeneous information sources,". in Proceedings of IPSI Conference. Tokyo, Japan: pp. 1994. [6] Dar, S., H. V. Jagadish, A. Y. Levy and D. Srivastava. "Answering queries with aggregation using views," in Proceedings of 22nd International Conference on Very Large Data Bases. Bombay, India: pp. 1996. [7] Franconi, E. and U. Sattler. "A data warehouse conceptual data model for multidimensional aggregation," in Proceedings of Int. Workshop on Design and Management of Data Warehouses (DMDW'99). Heidelberg, Germany: pp. 1999.

116

The European Union Data Warehouse Quality Project

Chapter 7

[8] Gebhardt, M., M. Jarke and S. Jacobs. "A toolkit for negotiation support interfaces to multi-dimensional data," in Proceedings of ACM-SIGMOD. Tucson, AZ: pp. 348-356, 1997. [9] Giacomo, G. D. and M. Lenzerini. "What's in an aggregate: Foundations for description logics with tuples and sets," in Proceedings of Fourteenth International Joint Conference on Artificial Intelligence (IJCAI-95) 1995. [10] Gray, J., A. Bosworth, A. Layman and H. Pirahesh, "Data cube: A relational operator generalizing group-by, cross-tabs, and sub-totals," Data Mining and Knowledge Discovery Journal, 1(1), 1997. [11] Hammer, J., H. Garcia-Molina, J. Widom, W. Labio and Y. Zhuge, "The Stanford Data Warehousing Project," Data Engineering, 18(2), 1995, pp. 41-48. [ 12] Harinarayan, V., A. Rajaraman and J. Ullman. "Implementing data cubes efficiently," in Proceedings of 1996 ACM SIGMOD International Conference on Management of Data. Montreal (Canada): pp. 205-227, 1996. [13] Huhns, M. N., N. Jacobs, T. Ksiezyk, W. M. Shen, M. P. Singh and P. E. Cannata. "Integrating enterprise information models in Carnot," in Proceedings of Int. Conf. on Cooperative Information Systems (CoopIS-93), 32-42, 1993. [ 14] Hull, R. "Managing semantic heterogeneity in databases: A theoreti-cal perspective," in Proceedings of Proceedings of the 16th ACM SIGACT SIGMOD SIGART Symposium on Principles of Database Systems (PODS). Tucson, AZ: pp. 51-61, 1997. [ 15] Jarke, M., R. Gallersdörfer, M. Jeusfeld, M. Staudt and S. Eherer, "ConceptBase - a deductive object base for meta data management," Journal of Intelligent Information Systems, 4(2), 1995, pp. 167-192. [16] Jarke, M., C. Quix, G. Blees, D. Lehmann, G. Michalk and S. Stierl, "Improving OLTP data Quality Using Data Warehouse Mechanisms," Sigmod Record, 28(2), 1999, pp. 536 - 537. [17] Jarke, M., C. Quix, D. Calvanese, M. Lenzerini, E. Franconi, S. Ligoudistianos, P. Vassiliadis and Y. Vassiliou, "Metadata-Driven Management of Data Warehouses: The DWQ Demonstrators," Sigmod 2000, 2000. [18] Levy, A. Y., A. 0. Mendelzon, Y. Sagiv and D. Srivastava. "Answering queries using views," in Proceedings of 14th Symposium on Principles of Database Systems. San Jose, CA: pp. 95-104, 1995. [19] Levy, A. Y. and I. S. Mumick. "Reasoning with aggregation constraints," in Proceedings of International Conference on Extending Database Technology (EDBT-96). Avignon, France: pp. 1996. [20] Ligoudistianos, S., D. Theodoratos and T. Sellis. "Experimental Evaluation of Data Warehouse Configuration Algorithms," in Proceedings of 9th DEXA Workshop (DEXA'98). Vienna, Austria: pp. 1998.

Summary and Conclusions

117

[21] Llirbat, F., E. Simon and D. Tombroff. "Using versions in update transactions," in Proceedings of 23 rd VLDB Conf. Athens, Greece: pp. 1997. [22] Mumick, I. S., D. Quass and B. S. Mumick. "Maintenance of Data Cubes and Summary Tables in a Warehouse," in Proceedings of ACM SIGMOD Conf. Tucson, AZ: pp. 1997. [23] Quass, D., A. Gupta, I. S. Mumick and J. Widom. "Making Views Self-Maintainable for Data Warehousing," in Proceedings of Conference on Parallel and Distributed Information Systems. Florida: pp. 1996. [24] Srivastava, D., S. Dar, H. V. Jagadish and A. Y. Levy. "Answering queries with aggregation using views," in Proceedings of 22. International Conference on Very Large Data Bases (VLDB-96) 1996. [25] Staudt, M. and M. Jarke. "Incremental Maintenace of Externally Materialized Views," in Proceedings of VLDB. Bombay, India: pp. 1996. [26] Therodoratos, D. and T. Sellis. "Data warehouse configuration," in Proceedings of 23rd VLDB Conf. Athens, Greece: pp. 1997. [27] Ullman, J. D. "Information integration using logical views," in Proceedings of 6th Int. Conf. on Database Theory (ICDT-97) 19-49, 1997. [28] Vassiliadis, P. "Modeling multidimensional databases, cubes and cube operations,". in Proceedings of 10th International Conference on Scientific and Statistical Database Management (SSDBM). Capri, Italy: pp. 1998. [29] Wang, R. Y., M. P. Reddy and H. B. Kon, "Toward quality data: An attribute-based approach," Decision Support Systems (DSS), 13, 1995, pp. 349-372. [30] Wiener, J. L., H. Gupta, W. J. Labio, Y. Zhuge, H. Garcia-Molina and J. Widom. "A System Prototype for Warehouse View Maintenance," in Proceedings of the ACM Workshop on Materialised Views: Techniques and Applications. Montreal, Canada: pp. 26-33, 1996. [3 11 Zhou, G., R. Hull and R. King, "Generating Data Integration Mediators that Use Materialization," Journal of Intelligent Information Systems, 6(2), 1996, pp. 199-221. [32] Zhuge, Y., H. Garcia-Molina, J. Hammer and J. Widom. "View maintenance in a warehousing environment," in Proceedings of ACM SIGMOD Int. Conf. Management of Data 316-327, 1995.

This page intentionally left blank.

C H A P T E R

8

The Purdue University Data Quality Project

C onsistency and completeness are considered two important dimensions of data quality [1, 9, 20]. However, they are usually compromised by errors, accidentally or intentionally introduced, in a database system. Such errors result in inconsistent, incomplete, and erroneous data elements. A small variation in the representation of a data object may, for example, produce a unique instantiation of the object being represented. In order to improve the accuracy of the data stored in a database system, the data must be compared either with real-world counterparts or with other data, stored in the same or in a different system. This chapter has been written by Vassilios S. Verykios at Drexel University under the direction of Ahmed Elmagarmid at Purdue University. It profiles one aspect of the on-going Purdue research projects. Specifically, it addresses the problem of matching records, which refer to the same entity, by computing their similarity. In this context, exact record matching has limited applicability since, even simple errors, like character transpositions, cannot be captured in the record linking process.

120

The Purdue University Data Quality Project

Chapter 8

The methodology deploys advanced data mining techniques for dealing with the high computational and inferential complexity of the approximate record matching process.

INTRODUCTION Many of the multiple legacy and information systems that support the operations of a company usually contain a great deal of redundant, summarized and overlapping data objects that are often interdependent [6]. Lack of a common data model, errors in data flows, errors during data entry, or finally situations where updates are not reflected into the database, cause inconsistencies to arise [13,22,23]. Such inconsistencies are common in systems today and they cause significant revenue loss. Elmagarmid et al. [4] reported that up to 25% of customers records are erroneous in a typical billing system. Existing techniques to cope with these problems are listed below: • Data edits are computerized routines, which certify whether data values and their representations satisfy predetermined constraints. Such constraints are sometimes called business rules, and they can be either very simple or quite sophisticated. Data edits may be applied to an entire database as a screen or as a filter within a data flow. • Data correction matches the data against master lists and tags all fields either as good, bad, or automatically correctable. For example, factory management software systems have access to data range tables that are used to filter out-of-range data. • Record matching determines whether two records, probably of different types, represent the same object. This process involves value judgments and requires sophisticated software tools. Customer matching and retail house holding are two examples where record matching can be applied. In the first application, the goal is to locate the customers that have bought another product from the same supplier before, and in the second, to find a group of people who comprise a family, every member of which is a customer. The methodology presented in this chapter improves upon existing techniques used for record matching. A brute-force approach for matching records is of quadratic order with respect to the number of records in the database. This can become a bottleneck for very large databases. The high dimensionality of the problem calls for heuristics to cut down the search space and shortcuts similar to the ones almost unconsciously employed by a human filing clerk. The main contribution is the complete automation of the knowledge acquisition process for record matching. The comparison space, resulting from comparing

Background

121

pairs of records in a sample drawn from the original database, is clustered and the clusters are described in order to produce predictive patterns. The so generated patterns are simplified and used as a rule base for matching pairs of records in the entire database. The results in the initial set of the experiments that we conducted indicate that the proposed methodology improves on the accuracy of the matching process and reduces its high complexity.

BACKGROUND The idea of linking or matching records is very simple. Record matching means bringing together information about the same entity. There are two types of record matching. The first one is the exact matching and it is used whenever the records are error-free and there is unique identifying information in the records. The second type of matching, which is the one that we are investigating in this paper, is the approximate matching. The approximate matching is used whenever the records do not have a unique key, or whenever there are differences in the representation of the records, or when erroneous and incomplete information (missing values) is contained in the database. The two principal steps in the linking process are the searching for potentially linkable pairs of records (searching step) and the decision that must be made as to whether or not a given pair is correctly matched (matching step). For the searching step, the aim must be to reduce the number of failures to bring potentially linkable records together for comparison. The technique to be applied must address this problem without resorting to excessive amounts of additional searching. The problem, as far as the matching step is concerned, is to enable the computer to infer and apply the rules ofjudgment by which a human expert would decide whether or not a pair of records relates to the same entity, when some of the identifying information agrees and some disagrees. In the product space of two tables, a match is a pair that represents the same entity whereas a non-match is a pair that represents two different entities. Within a single database, a duplicate is a record that represents the same entity, as another record, in the same database. Rather than regard all pairs in the product space of two databases, or the space of a single database, it may be necessary to consider only those pairs that agree on certain identifiers or blocking criteria. Blocking criteria are sometimes called sort keys. Missed matches are those false non-matches that do not agree on the set of blocking criteria. Common record identifiers such as names, addresses and code numbers like SSN, are the matching variables that are used to identify matches. The vector, that keeps the values of all the attribute comparisons for a pair of records is called comparison vector. The set of all possible vectors is called comparison space. A record matching decision rule is a rule that designates a pair either as a link, apossible link, or a non-link based on the information contained in the comparison vector. Possible links are those pairs for which identifying information is not sufficient to determine

122

The Purdue University Data Quality Project

Chapter 8

whether a pair is a match or a non-match. Typically, manual review is required to decide upon the matching status of the possible links. False matches are those nonmatches that are erroneously designated as links by a decision rule. False nonmatches are either (i) matches designated as non-links by the decision rule as it is applied to a set of pairs or (ii) matches that are not in the set of pairs to which the decision rule is applied. Generally, link/non-link refers to the designations under decision rules and match/non-match refers to the true status.

RELATED WORK The identification of similar records, in various databases, is a computationally intensive and difficult task. For a large database that stores millions of records, a pairwise comparison of all of its records is not feasible. This implies that a technique for record matching has to be both computationally efficient and intelligent enough. In the following paragraphs, we describe the related work from the viewpoint of the searching and matching steps of the record matching process.

Searching Process In the case of the searching step, errors, in the form of failures to bring potentially linkable pairs of records together for comparison, could be reduced to zero simply by comparing each record with all the other records. However, wherever the files are large, such a procedure would generally be regarded as excessively costly, if there are many wasted comparisons of pairs of records that are not matched. For this reason, it is usual to put the records in the database in order by using identifying information that is common to all of them [14]. The ordering can be done based on the information that is kept in the key or in some other combination of record fields (or subset of parts of fields). In the exact matching, sorting the file can be used to reduce the complexity of identifying duplicate records [2]. In the approximate record matching, various compression codes, i.e., phonetic, can be used to efficiently overcome the failures in bringing potentially linkable pairs of records together for comparison. There are a number of systems for doing this, the most common of which is known as the Russell Soundex code. This is essentially a phonetic coding, based on the assignment of code digits, which are the same for any of a phonetically similar group of consonants. Another technique for cutting down the number of unwanted comparisons in the approximate record matching, is to scan the database by using a fixed size window and check for matches by comparing every pair of records that falls inside that window [8], assuming that the records are already sorted. The same authors have extended this process, (a) by making multiple passes through the database (sorted neighborhood approach), using different sort keys in each pass and (b) by a transitive closure phase for combining the results of independent passes. The transitive closure

Related Work

123

step has been independently identified by Monge and Elkan [12]. Another approach is to scan the database by comparing only those records that agree on a used-defined key, which for example can be the key used for sorting the records. This technique is known as blocking. Another approach proposed in [12], is the use of union/find structures for keeping track ofclusters ofapproximately duplicate records, incrementally. With respect to the searching phase, the proposed methodology makes use of a combination of a multi-pass sorted neighborhood approach based on some preselected keys along with a standardization of the values kept in a subset of fields. This standardization is similar enough to the coding schemes mentioned above. In particular, before sorting starts, a pre-processing phase is applied to the records by using public domain software for standardizing lists containing name and address information.

Matching Process When pairs of records are brought together for comparison, decisions must be made as to whether these are to be regarded as linked, not linked, or possibly linked, depending upon the various agreements and disagreements of items of identifying information. For example, if we are linking person records, a possible measurement would be to compare family names on the two records, and assign the value of “1" for those pairs where there is agreement and 0 for those pairs where there is disagreement. These measurements will yield a vector of observations on each record pair. Most of the work proposed by statisticians has been influenced by the pioneering work of Fellegi and Sunter [5]. Their paper documents the derivation of a test statistic and a critical region for deciding whether or not a pair of records is a match. In addition, it discusses some of the assumptions that are necessary for practical applications and it also describes approaches for estimating the field matching probabilities, which are used to calculate the test statistic. Hernadez and Stolfo [8] suggest the use of an equational theory, manually acquired by domain experts, which can be codified as a rule base of an expert system. Monge and Elkan [12] use a domain-independent algorithm for detecting approximately duplicate records. Cochinwala et al. [3] suggest the training of a prediction model by generating a table of prospective matches to be used as a training set, and the reduction in the complexity of the model by statistical techniques. Our approach improves significantly on the technique presented in [2 I] by using a clustering technique to automatically identify comparison classes in the comparison space of a sample set of records selectively drawn from the original database. The space generated by the comparison vectors, along with the corresponding classes identified during clustering, is automatically reduced by a feature subset selection process that minimizes the number of vector components. These components are used for building the predictive model that determines the linking status of any pair

124

The Purdue University Data Quality Project

Chapter 8

of records in the database. The induced model is automatically transformed into a set of production rules and is simplified for becoming more accurate.

METHODOLOGY The proposed methodology tries to overcome the computational inefficiencies, of all the previous approaches by automatically building a model that captures the most important patterns that exist in the data. Well-established machine learning techniques can be adequately utilized in this framework for building a minimal model that fits the data very well. There are four phases in our methodology that are listed below and are explained in the subsequent subsections. 1. The data pre-processing of the records. This phase deals with various semantic inconsistencies in the data as well as with problems of individual records like missing values, null values, outliers, etc. Data standardization is also a very important task for the preparation of the data for the later phases. 2. The sorting of individual records. The sorting is based on values of a pre-specified key (sorting variable), and on the sampling of the records. The scanning of the records in the sample set and the pair-wise comparison of all records falling inside a constant size window are also included in this phase. The result of this phase is the generation of the comparison vectors for the comparison pairs in the sample set. 3. The use of a clustering algorithm. This algorithm is used so as to identify clusters in the comparison space of the sample set and to use the clusters identified as labels of the comparison vectors for building a model by using a decision tree inducer. 4. The transformation of the decision tree into a rule set and application of a simplification technique for reducing the size of the rule set and/or improving its predictive accuracy. The final rule set will be used as the linkage rule set for deciding upon the linking status of all the pairs of records falling inside a constant size window in the entire sorted data set.

Data Pre-processing The preparation of the data for the later phases is an important and domain specific task. For example, the conversion of the characters, in the data stream into either lowercase or uppercase, helps to avoid any variations that may exist due to case dif-

Methodology

125

ferences between various data elements. In many systems, records are stored as discrete fielded entries. Although this provides a powerful technique for modeling during the analysis, the individual components must be combined in such a way as to represent each entity uniquely. For instance, fields involving last name and first name or street number, city, state, and zip code need to be concatenated. Field values usually contain many different types of extraneous characters. Whenever possible, they should be removed from the data stream. Moreover, it is sometimes useful to reduce the information represented in a field to a minimum number of categorical values. This allows for conveying certain types of information without the overhead of representing every single unique value. A standard unit must be adopted for each field appearing in the data set. For example, if there is a field containing measures of distance, a standard conversion to one unit must be adopted for all values represented in the database. As it is already mentioned, complete processing of the data, by the record linking technique, requires valuable computational time. Based on that, it is beneficial to restrict the data set to a minimal one that exhibits certain quality characteristics. Records containing empty or null fields that are critical to the analysis can usually be removed. Value boundary violations that result in bad or dirty data, can be removed as well. We should point out that the data pre-processing phase is heavily dependent on the domain and the condition of the in-flow data. If, for example, null values are common for the entire data set then it might be better to project on all the other attributes except from the one with the null values, instead of discarding the entire data set. In addition, some boundary violation problems can be easily resolved, if we can find out the semantics of the values represented in the records. For example, a value of 99 for the month field may represent total values for all months in the specified year. Another very important pre-processing task, that must be performed before any attempt is made to link records, is the standardization of the data. The appropriate parsing of the name and the address components is the most crucial part of the computerized record linking. Without it, many true matches would erroneously be designated as non-links, because common identifying information could not be compared. The basic ideas of standardization are (i) to replace the many spelling variations of commonly occurring words with standard spellings such as with a fixed set of abbreviations or spellings and (ii) to use certain key words that are found during standardization as hints for parsing subroutines. In standardizing names, words of little distinguishing power as “Corporation” or “Limited”are replaced with consistent abbreviations such as “CORP”and “LTD’,respectively. First name spelling variations such as “Rob” and “Bobbie”might be replaced with a consistent assumed original spelling such as “Robert”or an identifying root word such as “Robt”.Standardization of addresses operates like standardization of names. Words such as “Road”or “Rural Route”are typically replaced by appropriate abbreviations.

126

The Purdue University Data Quality Project Chapter 8

Sorting, Sampling & the Sorted Neighborhood Approach Before we are able to efficiently process the records, we need to sort them. When we sort the data, we actually try to generate a linearization of the records. Domain experts should sort the data based on a-priori knowledge that certain fields or parts of fields have a good discriminating power for the record linking process. We expect that these parts should, in some general sense, be clean or, to a large extent, not be affected by the various kinds of errors introduced in the system. If this is not the case, multiple passes might be required for achieving good results. For sorting purposes, the reason that multiple passes through the data help in resolving this problem is the following. By using a blocking or a sorted neighborhood technique, records having different values for the blocking variable or falling outside a certain window, will never be considered or tested for linking. If, for example, there is an error in the sort key of a record, then this record will be placed incorrectly far away from its correct place in the sorted file hindering in this way the identification of true matching records. By selecting different sort keys in different passes, there is a high probability that, in one of these passes the record will be located in the “right”place and the identification of the linked records will be achieved. The consistent scanning of the data comes with a heavy duty of decreased performance, and it must always be considered as a function of the available computational resources and the available time. The cost constraints that must be met in a business environment and the balance between the two kinds of errors (Type I and 11) in the statistical sense must also be taken into consideration. The first one identifies a pair of records as a link, even if the records do not actually match, and the second one identifies a pair of records as a non-link, even if the records do match. Depending on the impact of these two errors, we must decide on the selection of sort keys and the number of passes. After the records have been sorted, we can easily extract a contiguous subset of records from the database, as a sample, and use it for further processing. The reason that we do not actually need to apply any complicated algorithm for sampling, after we have sorted the data, is that the most important events in the data are restricted to small neighborhoods of contiguous records. Selecting the appropriate size of the sample set is another issue but, we expect that it can be easily resolved experimentally. The sorted neighborhood technique, suggested by Hernadez and Stolfo in [8], will now be applied to the sample set of records. The records, falling within a range of size w from the location of a reference record in the sorted sample file, will be compared among each other and this process will be repeated for all the records in the sample database. Notice that the value of w for this phase must be bigger than the corresponding value of w when the entire database is searched for linkable records. This is because the training process in the next phase requires both positive (linked) and negative (non-linked) examples for correctly learning the underlying predictive domain model. The comparison vector, generated by a record pair comparison, will be holding information related to the closeness of the various attributes in the pair of records. The comparison of any such pair of records can be viewed as a

Methodology

127

set of outcomes, each of which is the result of comparing a specific attribute from one record with the same attribute in the other record. Outcomes may be defined as precisely as desired; for example, one might define an outcome of a comparison to be simply that the attributes agree or disagree. Or, one might define the agreement outcome more specifically based on the possible values that attribute can take. “Comparison of attributes” is usually interpreted to mean that the same attribute is recorded on each record and that they can be compared directly. However, it is possible to “compare different attributes which are known to be correlated” or to use information related to the record descriptors.

Clustering and Classification As previously mentioned, the comparison space consists of comparison vectors containing information that is related to the differences between fields of a pair of records. The particular components of these vectors contain values that can be either of continuous or of discrete type. A value of a continuous attribute in the comparison vector is always a real number, whereas the value of a discrete attribute is one of a small set of possible values. The next step is how to determine the linking status of a pair of records based on the information kept in its comparison vector. Although we can be sure that every vector must correspond to one of the three possible classes, the link, the non-link and the possible link one, there is no easy way to assign these classes or labels to the comparison vectors. If we are able to assign these classes to the comparison vectors, we can use the comparison vectors along with their corresponding classes to build a model in order to predict the class of any comparison vector. This model can be used as a record linkage rule set. In order to solve the problem of labeling the comparison vectors, we consider a slightly different problem. Instead of trying to immediately assign labels to comparison vectors, we use a technique to identify clusters of comparison vectors. Then we map these clusters to the three labels corresponding to the record linking status. Clustering is a common descriptive task where one seeks to identify a finite set of categories or clusters to describe the data. The categories may be mutually exclusive and exhaustive, or they may consist of a richer representation such as hierarchical or overlapping categories. A clustering technique can be easily applied to the comparison space of the record linking methodology in order to identify clusters ofvectors. By combining the cluster values, identified by the clustering algorithm with their corresponding comparison vectors, we can create a set of training vectors to be given as input to a data mining technique that induces models from labeled vectors. The technique used for inducing the model is a decision tree inducer [16, 19] that generates decision trees, from examples for prediction purposes. A decision tree inducer builds trees that can be used for predicting the label of a pair of previously unseen records, based on the values of the vector components computed for this pair of records. The set of comparison vectors with known classes from which the decision tree will be induced is called the training set. Other collections of comparison vectors not seen while the tree was being developed are known as test sets and are

128

The Purdue University Data Quality Project

Chapter 8

commonly used to evaluate the performance of the tree. The non-leaf nodes of a decision tree are labeled by the name of one of the vector components computed in the training phase and the edges, originating from these nodes, are labeled with values or ranges of values corresponding to the component that labels the node. The leaf nodes are labeled by the linking status that comes from the majority of all the comparison vectors covered by each leaf node. In our case, a non-leaf node may be labeled by the edit distance between the strings corresponding to the first name of each record. The outgoing edges from this node are labeled by possible values or groups of values that can be attained by this component. A leaf node can be labeled by the value “matched-yes”,designating that the majority class of the comparison vectors, covered by the particular leaf node, belongs to the “matched” status. Pruning is one of the traditional techniques used to avoid over-fitting because the initially induced model usually fits either to some randomly present examples or to others that have been corrupted by noise. Pruning produces trees of smaller size than the initial tree with higher predictive accuracy. Feature subset selection is another important technique that affects the complexity ofthe record linking process in two ways. It is well understood that not all vector components have the same discriminating power with respect to the class. By including in the induced model only this subset of components that have the highest correlation with the class label, the induction algorithm applies an implicit form of feature selection. As the computations required for determining the values of each component of the comparison vector is an intensive one, we expect high payoffs from the feature selection mechanism with respect to cutting down the overall processing time. New techniques for feature selection can be used before an induction algorithm is applied, for reducing the dimensionality of the feature space, especially for these kinds of algorithms that use the entire feature (comparison) vector for inferencing purposes like instance based techniques. With this respect, our methodology is not restricted by a particular kind of induction technique. For this reason, many alternatives can be explored and their predictive accuracy can also be tested.

Decision Tree Transformation and Simplification The decision tree induced in the previous phase has to be transformed into a set of production rules that will constitute the rule base of the expert system. A decision tree can be easily transformed into a rule set by converting each path of the tree into a rule as follows: (a) the internal nodes and their output branches are converted into conditions of the antecedent (“if-part”) of the rule; and (b) the leaf nodes are converted into the consequent (“then-part”) of the rule. As a side effect, we try to increase the accuracy of the induced model and, at the same time, decrease its complexity and size for efficiency reasons. Merely rewriting a tree, as a collection ofthese equivalent production rules, would not represent any simplification at all. For further improvement of the quality of the rule set, we adopt the method proposed in [ 17, 18]. The process has two stages: (a) individual production rules are first generated and polished; and (b) the rules produced are

Methodology

129

evaluated as a collection. In the first stage each production rule is examined in order to decide on whether it should be generalized by dropping conditions from its lefthand side. Let Xi be one of the conditions and consider those comparison vectors in the training set that satisfy all the other conditions in the rule but not Xi. With respect only to those cases, the relevance of Xi to determining whether a comparison vector belongs to a certain class c, given that the other conditions are satisfied, can be summarized by the 2x2 contingency table which is shown in Table 8.1. This table used for the simplification ofthe decision trees where sc is the number ofthese cases that satisfy Xi and belong to the class c, sc is the number of these cases that satisfy Xi but belong to some class other than c, and so on. Table 8.1: 2x2 contingency table

Class c

Not Class c

Satisfies Xi

sc

sc

Does not satisfy Xi

sc

sc

A contingency table is a form of presentation of grouped data. In the simplest case, a group of items may be classified into just two groups, according to, say, presence or absence of a certain characteristic. The categories may be qualitative or quantitative. When there is only one characteristic according to which data are classified, we get a one-way table. If there are two ways of classification, the table is called a two-way table. The final step in this first stage is to estimate a certainty factor for the simplified rule. If the left-hand side of a rule is satisfied by V cases in the training set, Wof which belong to the class indicated by the right-hand side, the certainty factor of the production rule is taken as (W- 1/2)/V. By computing a certainty factor, we can estimate whether the eliminated from the antecedent tested condition can be completely dropped without affecting the accuracy of the rule. The number of rules generated this way is always smaller than the number of rules generated by initially transforming the patterns in the decision tree to rules. One of the consequences of dropping conditions from the antecedents of the rules is that an input case can be matched against more than one rule. In this case, the expert system puts all the matched rules into its agenda and fires the one with the highest certainty factor. The second stage of the simplification process looks how well the rules will function as a set. This evaluation depends on the way the rules are used. A simple strategy can be adopted. In order to classify a case, a rule that applies to this case must be found. If there is more than one such rule, the rule with the higher certainty factor is chosen. If there is no applicable rule, then the most frequent class in the training set is assigned to that case by default. For each rule r in the rule set, we now

130

The Purdue University Data Quality Project

Chapter 8

count the mis-classifications produced by the rule set R-{r} in the training set. If the reduced rule set R-{r} gives at least as much accuracy as the entire set R, then the rule r is removed from the rule set R.

EXPERIMENTS

AND

OBSERVATIONS

In order to test our methodology, we make use of a database of records taken from the "Student and Staff Telephone Directory" of Purdue University. Each record contains the first, middle and last name, the home address, the office address, and the telephone number of an individual. Each record contains a system specific key, which automatically assigned by the system, (this is only done for allowing us to compute the accuracy of the proposed methodology). By using software that controls the generation of errors in the database, we can simulate a variety of real life scenarios. Each replicated and possibly erroneous record maintains its original key. For numerical fields, we compute the Euclidean distance and for character string fields we compute the well-known edit-distance metric, by implementing the minimum edit distance algorithm from [11], which is of complexity O(m. n), (m and n are the sizes of the compared strings). The pre-processing phase consists of applying the standardization phase to the records. We can be sure that the records are well conditioned because of the database generation process and for this reason; there is no need to apply any other kind of pre-processihg to these. For the standardization task, we make use of the Bureau of the Census record matching system [24] which includes a module to standardize names, both business and personal, and also street addresses. For sorting the database, we can use the sort Unix command which sorts lines of all the named argument files together and writes the results on the standard output. Comparisons are based on one or more sort keys extracted from each line of input. By default, there is one sort key, the entire input line. After the database has been sorted we can easily extract a small portion of contiguous records, which will be the sample set for generating the training set. We have used the two first characters from the last and first name as well as the two first characters from the main street name, for sorting the database. After that, we can apply the sorted neighborhood approach to the sample set. The size of the window depends among other things on the size of the sample set, but we select it to be very small with respect to the size of the sample set. Usually, a window size of 10 is enough. After the application of the sorted neighborhood approach, we end up with the comparison vectors for all the pairs of records falling within a window size of 10 from each other. The next step in the process is to partition the comparison space into clusters of comparison records that exhibit the same probabilistic behavior for the features used. For this reason, we have used Autoclass, a well-known clustering tool developed by NASA Ames Research Center. Autoclass is applicable to observations of objects that can be described by a set of features or properties, without referring to other

Experiments and Observations

131

objects. This allows the user to represent the observations by a data vector corresponding to a fixed attribute set. Attributes are names of measurable or distinguishable properties of the things observed. The data values, corresponding to each attribute, are thus limited to be either numbers or the elements of a fixed set of attribute specific symbols. AutoClass uses probability models for describing the classes, or else fits probability models to the input data. There are two kinds of models that are supported by Autoclass. The first kind includes probability models, which assume that the attributes are conditionally independent given the class and the second, includes covariant models, which can express mutual dependences within the class. The selection of the particular probabilistic model is very important for ensuring the correctness of the results. The probabilistic model used in this example, for all of the components of the comparison vector, is the multi normal one. This model implements a likelihood term representing a set of real valued attributes, each with constant measurement error and without missing values, which are conditionally independent of other attributes given the class. This is because of the assumption that an error in a record is independent from any other error. The comparison vector given to Autoclass had 15 components. AutoClass came up with a list of possible models; the most probable one contains three “clusters”.This is consistent with the theoretical results of link, non-link and possible link record comparison classes. By comparing the actual status of the pairs of records with the classes identified from the clustering tool, (recall that the correct matching status of the compared pairs of records is known for the training set), we can map each class to the corresponding label. In order to be able to test the predictive accuracy of the induced model, since by having all the records keep the system specific key we know their correct linking status, we use only those linking vectors having as labels the values “matched-yes”and “matched-no”.

Figure 8.1: Decision tree generated using 89 cases as a training set.

For inducing a model for this domain, we make use of these two labels along with the corresponding comparison vectors. This information is given to a decision tree inducer for building a model for the identified classes. We selected the ID3 [16] algorithm, which is a top-down inducer of decision trees. The implementation of this

132

The Purdue University Data Quality Project

Chapter 8

algorithm is from the MLC++ library [10], which also provides a feature subset selection technique for reducing the initial dimension of the comparison vectors. Figure 8.1 demonstrates the decision tree corresponding to a very small comparison space of 89 comparison vectors while Figure 8.2 shows a more complicated decision tree. Figure 8.3 shows a geometric representation of the classes “1” for “matchedyes” and “2” for “matched-no” in the 2-dimensional space of the name-edit-distance in the x-axis and address-edit distance in the y-axis. Table 8.2 shows some preliminary results that indicate the high accuracy attained by the learning algorithms for different sizes of the training and the test set.

Figure 8.2: Decision Tree generated using 602 cases as a training set.

Experiments and Observations

133

Table 8.2: Accuracy achieved by the induced decision trees.

Training Set Test Set

Correctly Classified

Incorrectly Classified

Percentage Error

89

152

146

6

3.947

177

303

288

15

4.950

264

453

433

20

4.415

350

602

580

22

3.654

435

750

726

24

3.200

519

897

874

23

2.564

602

1043

1016

27

2.594

Figure 8.3: Pictorial representation of the distribution of classes in the 2D space.

134

The Purdue University Data Quality Project

Chapter 8

Even though the error decreases, as the number of training set becomes larger and larger, we expect that the complexity of the induced models will become higher and higher. Note the difference regarding the number or parameters in then induced models, between Figure 8.1, which based on the information in Table 8.2 generates an error of 3.947, and Figure 8.2, which generates an error of 2.594. In order to demonstrate the simplification process, we select to proceed with only one of the demonstrated decision trees, and in particular with the one shown in Figure 8.1. By transforming the decision tree patterns from Figure 8.1 into production rules, we get the rule set displayed in Table 8.3. By applying the two-step approach for improving the quality of the rule set, we are left with the reduced rule set displayed in Table 8.4 which has exactly the same predictive accuracy as the original rule set at a certain level of statistical significance. The accuracy attained in this example, is 98%. This must be considered with respect to the number of comparison vectors that were assigned to the possible link class and correspond to 4.5% of the total number of comparison vectors used in this case study. Table 8.3: The rule set produced by translating the information in the induced decision tree.

If name-edit-distance < 4.5 and address-edit-distance <8.5 then class=matched-yes with certainty factor 1.0 if name-edit-distance < 4.5 and address-edit-distance >= 8.5 then class=matched-no with certainty factor 1.0 if name-edit-distane >= 4.5 then class=matched-no with certainty factor 1.0

Table 8.4: The rule set produced by simplifying the initial rule set.

if name-edit-distance < 4.5 then class=matched-yes with certainty factor 0.933 if name-edit-distance >= 4.5 then class=matched-no with certainty factor 1.0

Conclusions

135

The expert system shell we are using as the inference engine for applying the rule set to the entire database, (and in particular to the test set), is CLIPS [7] from NASA Johnson Space Center. The entire system has been coded in Perl [15].

CONCLUSIONS Data mining techniques, like clustering and classification, have been applied to the problem of approximately duplicate record detection. Pruning and certainty factors have also been utilized for increased accuracy in domains where uncertainty is a major problem. Expert system technology is a promising vehicle to provide solutions, even under incomplete, imprecise, or uncertain information. Sampling can also be very handy in domains where an overabundance of information is a fact. The main contribution of the methodology presented in this chapter is the complete automation of the knowledge acquisition for the record matching process. The results of our initial experiments indicate that the proposed methodology improves the accuracy of the matching process and reduces its high complexity. In addition the human intervention is limited to a minimum. Our solution also indicates that great improvements can be made towards the automation of the knowledge acquisition phase for the similarity searching procedure, by combining all these techniques into a single framework.

References

[1] Ballou, D. P. and H. L. Pazer, "Modeling Data and Process Quality in Multi-input, Multioutput Information Systems," Management Science, 31(2), 1985, pp. 150-162. [2] Bitton, D. and D. DeWitt, "Duplicate Record Elimination in Large Data Files," ACM Transactions on Database Systems, 8(2), 1983, pp, 255-265. [3] Cochinwala, M., V. Kurien, G. Lalk and D. Shasha, Efficient Data Reconciliation, Bellcore, 1998. [4] Elmagarmid, A., B. Horowitz, G. Karabatis and A. Umar , Issues in Multisystem Integration for Achieving Data Reconciliation and Aspects of Solutions. Bellcore Research, 1996. [5] Fellegi, I. P. and A. B. Sunter, "A Theory for Record Linkage," 64(328), 1969, pp. 11831210. [6] Georgakopoulos, D., G. Karabatis and S. Gantimahapatruni, "Specification and Management of Interdependent Data in Operational Systems and Data Warehouses," Operational

136

The Purdue University Data Quality Project

Chapter 8

Systems and Data Warehouses Distributed and Parallel Databases, 5(2), 1997, pp. 121166. [7] Giarratano, J. C., CLIPS User's Guide, Version 5.1, NASA Lyndon B. Johnson Space Center, 1991 [8] Hernadez, M. A. and S. J. Stolfo, "Real-world Data is Dirty: Data Cleansing and the Merge/Purge Problem," Journal of Data Mining and Knowledge Discovery, 1(2), 1998. [9] Huang, K., Y. Lee and R. Wang,- Quality Information and Knowledge. Prentice Hall, Upper Saddle River: N.J., 1999. [10] Kohavi, R., D. Sommerfield and J. Dougherty, Data mining using MLC++: A machine learning library in C++. Tools with Artificial Intelligence, IEEE Computer Society Press, 1996. [ 11] Manber, U., Introduction to Algorithms, Addison-Wesley Publishing Company, 1989. [12] Monge, A. and C. Elkan. "An Efficient Domain Independent Algorithm for Detecting Approximate Duplicate Database Records," in Proceedings of SIGMOD Workshop on Research Issues on DMKD, 1997. [13] Motro, A. and I. Rakov, Not all answers are equally good: Estimating the Quality of Database Answers. Flexible Query-Answering Systems, ed. T. Andreasen. Kluwer Academic Publishers, 1997. [14] Newcombe, H. B., "Record Linking: The Design of Efficient Systems for Linking Records into Individual and Family Histories," American Journal of Human Genetics, 19(3), 1967, pp. 335-339. [15] Ousterhout, J. K., "Scripting: Higher-Level Programming for the 21st Century," Computer, 3 1, 1998, pp. 23-30. [16] Quinlan, J. R., "Induction of Decision Trees," Machine Learning, 1(1), 1986, pp. 81-106. [ 17] Quinlan, J. R. "Generating Production Rules From Decision Trees," in Proceedings of 10th International Joint Conference on Artificial Intelligence, 304-307, 1987. [ 18] Quinlan, J. R., "Simplifying Decision Trees," International Journal of Man-Machine Studies, 27, 1987, pp. 221-234. [ 19] Quinlan, J. R., "Learning Decision Tree Classifiers," ACM Computing Surveys, 28(1), 1996, pp. 71-72. [20] Redman, T. C., ed. Data Quality for the Information Age. 1996, Artech House: Boston, MA. 303 pages. [21] Verykios, V. S., A. K. Elmagarmid and E. N. Houstis , Record Matching to Improve Data Quality (No. TR-99-005). Department of Computer Sciences, Purdue University, 1999.

Conclusions

137

[22] Wand, Y. and R. Y. Wang, "Anchoring Data Quality Dimensions in Ontological Foundations," Communications ofthe ACM, 39(11), 1996, pp. 86-95. [23] Wang, R. Y., V. C. Storey and C. P. Firth, "A Framework for Analysis of Data Quality Research," IEEE Transactions on Knowledge and Data Engineering, 7(4), 1995, pp. 623640. [24] Winkler, W. E., Record Linkage Software, US. Bureau of the Census, 1999.

This page intentionally left blank.

C H A P T E R

9

Conclusion

T he widely held assumption in the database community that the data stored in databases are of good quality is not always borne out in the real world. In practice, “dirty” data pervade databases for various reasons [ 15, 17, 22, 24, 35, 46]. Even though they acknowledge the existence of data quality problems, many in the database community view them as simply accuracy and integrity problems. It is now well established that the scope of data quality goes beyond accuracy and integrity to include other aspects such as believability and timeliness which are equally, if not more, important from the end-user’s perspective [1, 6, 40, 47]. Researchers in the data quality field must address these issues. In this book, we have presented an exposé of the state-of-the-art concepts, techniques, and models for data quality in the database area developed over the last decade at the MIT Total Data Quality Management research program. We have also

140

Conclusion

Chapter 9

profiled data quality research projects in other research institutions. Much of the research has been followed up by additional work, as presented below.

FOLLOW-UP RESEARCH In chapter 1, we presented a framework for Total Data Quality Management upon which much of our research has been based. While the material in Chapters 2-5 extends the TDQM framework with research anchored in database theory, we have also substantiated the framework with work from other disciplines. For example, we have applied the management science approach to model information manufacturing systems for delivering high quality data products [9]. We have also developed a statistical sampling procedure for data quality auditing in the Relational Data Base Management Systems environment [1] and techniques for data sampling in industrial-strength RDBMS where relational tables with millions of rows are fairly common [12]. In the management science area, additional work dealing with data quality has been reported by the SUNY, Albany group, as exemplified by Ballou and Pazer [2-6], Giri-Tayi and Ballou [7, 8, 41], and Chengalur-Smith et al. [1, 10]. Collaborations between practitioners and academics using the TDQM framework can be found in the Proceedings of the Conference on Information Quality [11, 23, 39, 44], for example those by Cedars-Sinai Medical Center [13], S.C. Johnson [16], and IRI [21]. The Polygen Model and Attribute-based Model research presented in Chapter 2 has been considered in industry practice. Two approaches have been proposed: the first overlaps the data quality attributes on top of the regular database application that has been developed, while the second considers the implementation of data quality attributes in the data dictionary. The Quality Entity Relationship approach presented in Chapter 3 has generated a significant amount of interest. Peter Chen, for example, states that “Mostdata models (including the ER model) assume that all data given is correct. We need a scientific way to assess the quality of the data in (or to be put into) the database. We also need to incorporate such quality measures into the EntityRelationship (ER) model as well as to develop algebra- or calculus-based operations to manipulate such data quality descriptions.” Chen’s group is investigating how to extend the Entity Relationship model to incorporate data quality characteristics. The work on automating data quality judgment in Chapter 4 has been extended by Jang, Ishii, and Wang [18] to include the notion of a data quality calculus. The data quality calculus is a data quality judgment framework based on a “census needs” and provides a framework for deriving an overall data quality value from a collection of local relationships among quality parameters. By specifying new, or alternate, local relationships, individual data consumers can tailor the data quality calculus framework to reflect their individual requirements. This customizability has the potential to be useful when various pieces of data from various sources, some of which may be unfamiliar, need to be integrated.

Follow-up Research

141

The work on estimating the accuracy of the result from relational algebraic operations in Chapter 5 has been extended by Parssian, Sarkar, and Jacob [29] who discuss a methodology for determining two data quality characteristics, accuracy and incompleteness, that are of critical importance to decision makers. From a different perspective, Ballou, Chengalur-Smith, and Wang [1] extended this work by developing a sampling procedure for data quality auditing. Reddy and Wang [30] also extended the work by estimating data accuracy in a federated database environment. The work on context mediation, presented in Chapter 6, has been extended into research about information aggregation on the Internet with the aim of developing technologies and business models to advance the understanding ofthis phenomenon. An aggregator is an entity that collects and analyzes information from different data sources. Aggregation defines a new landscape in information retrieval for goods and services on the Internet. Aggregators gather, organize, and make sense of valuable, but free, information from different web sites. This information is then presented to the customer without any context difference. For example, aggregators can be used to provide the customer with a comparison tool to decide such things as which goods or services to buy, the status of his/her personal financial information, or the best way to ship a package to the customer. Aggregators today provide information management services that help users manage multiple relationships, consumer education services for making appropriate comparisons of different products, and shopbots for locating the vendor with the lowest price. Aggregators pose significant threats to existing businesses through the consolidation and comparison of information posted on the World Wide Web. By reducing the consumer's search cost and enabling transparent comparisons across different offerings, aggregators eliminate information asymmetry in the marketplace. For details about the aggregation group at MIT, the reader is referred to their web site at http://context2.mit.edu/dev/aggregation. Further publications, white papers, case studies, and technology demonstrators are provided in the site. In Chapter 7, we presented the approach pioneered by the European Union DWQ project to implement data quality in a data warehouse. Many follow up papers were produced. For example, Schafer, Becker, and Jarke [36] developed an integrated data warehouse system, called DB-Prism, for distributed financial and management controlling (data collection, processing, and reporting) at Deutsche Bank. It combines fine-granular availability of historical data with high-actuality reporting and planning facilities. The DB-Prism project was motivated by challenges concerning data quality factors such as integrity, transparency, and traceability of data. The DB-Prism system is in operation and has had a significant impact on the organization by achieving the quality goals of integrity, coherence across controlling pathways and reporting systems, currency and availability of data, evolvability and efficient handling of massive amounts of heterogeneous data both at the source and client side. Thus, DB-Prism provides an interesting example ofa quality-oriented, conceptcentered data warehouse architecture as researched in the European DWQ project. For an extensive listing of all the publications generated by the DWQ project, the reader is referred to the web site located at: http://www.dbnet.ece.ntva,gr/~dwq/publications.html.

142

Conclusion

Chapter 9

Chapter 8 profiles one aspect of the Purdue University’s Data Quality Project whereby an automated methodology for knowledge acquisition in the record matching process is developed. The proposed methodology attempts to overcome the inherent sensitivity of the matching algorithm to the various parameter settings and the computational inefficiencies of existing approaches by (a) automatically and intelligently acquiring the required knowledge from training data, (b) reducing the computational complexity required by the induced algorithm for matching, and (c) ensuring certain levels of accuracy in the decision making process. Furthermore, the decision algorithm is built using a very small, appropriately selected, part of the source data for training. This work has been further extended in [42] where different knowledge induction and searching strategies are evaluated with respect to the completeness and validity of the record matching and searching phases. In the context of the work accomplished in [42], an experimental system has been built to perform controlled studies by generating errors in the data with known probabilities. Various associations between the quality of the input data and the matching model or its parameters can be easily derived in this way. Purdue University and its collaborators in this research area have actively pursued many other aspects of the data quality work. For example, in [25] a real-life case study on assessing and improving the quality of the data in the Italian Public Administration is described. The domain of the study was set on taxpayer’s data maintained by the Italian Ministry of Finances. The Administration was provided with a quantitative reckoning of such specific problems as record duplication and address mismatch and obsolescence. Also, a set of guidelines for setting precise quality improvement goals was suggested along with an illustration of analysis techniques for achieving these goals. Special emphasis was put on the importance ofboth the data flow analysis and the definition of measurable quality indicators. Both the research efforts in [42] and [25] have been accomplished in collaboration with researchers from the Applied Research Department at Telcordia Technologies and the College of Information Science and Technology at Drexel University. As a complementary thread of research, an experiment to build cost and time optimal decision models for record matching is also planned. Currently, almost all decision models for record matching rely on decision rules that minimize the probability of error or the probability of making a wrong decision. A more effective approach would be to build decision models that minimize both the cost of making a decision (i.e., number of comparisons required) and the cost associated with the impact of making this decision (i.e., maybe the cost of making a Type I error is different from the cost of making a Type II error, etc.).

OTHER RESEARCH As mentioned previously, the topics covered in this book are based primarily on the work done at the MIT TDQM research program and the institutions that we profiled.

Future Directions

143

There are other research contributions to the data quality field. Many of the contributions are available in the Proceedings of the Conference on Information Quality [1 1, 23,39,44] and other venues. For example, research has been conducted at George Mason University with the perspective that data integrity equals completeness plus soundness [26-28] and at AT&T in the early 1990’s [31-35]. Other researchers have started to address data quality issues in the context of the Internet and web-based applications. For example, Segev [37] at University of California, Berkeley discusses data quality in the context of B2B Exchange, and Paul Bowen at the University of Queensland, Autstralia investigates time-based threats to data quality in electronic commerce. Measuring the data quality of web sites is also becoming topical; see for example Katerattanakul and Siau [19], and Kopcso, Pipino, and Rybolt [20].

FUTURE DIRECTIONS It is critical to understand that data quality is, by nature, an inter-disciplinary research field. Accordingly, an inter-disciplinary approach is required to develop it into a mature field. To this end, students of data quality from various disciplines, such as computer science, management science, statistics, and management information systems, will be more productive in advancing the data quality field through collaboration with others from different disciplinary backgrounds. Equally important is the understanding that data quality is a cross-industry problem that requires practitioners from different industries to collaborate with academics. Through such collaboration, we would be more likely able to develop pragmatic data quality practice and best practices. These, in turn, will sustain long-term improvement of data quality in organizations. Increasingly, financial management, health care, computer, and telecommunication industries are getting involved in industry-academia collaborations [ 14, 17]. With this understanding, we are actively pursuing a research agenda with an industry-government-academia consortium. This consortium is jointly organized by Boston University’s Systems Research Center and MIT TDQM research program. Although there are many interesting research problems that we are currently investigating, on top of our research agenda are three projects: • Manage information as a product [45] • Model information manufacturing systems in practice with the development of data production maps [9], • Develop a roadmap for data quality journeys to achieve higher productivity [43]. Each of these projects can be investigated from different disciplinary perspectives. For example, we are developing a representation of the manufacture of infor-

144

Conclusion

Chapter 9

mation products [38]. In this research, we explicitly represent business boundaries (e.g., from one department to another) and systems boundaries (e.g., from one database system to another) because our research has suggested these boundaries to be breakdown points in an information manufacturing system. Analogous to data flow diagrams in Oracle's Designer 2000 but explicitly developed in the data quality context, this research has formed the foundation for further work from the technical perspective as we continue to develop some properties such as traceability and timeliness operations. In terms of the development of data production maps, we are collaborating with several TDQM industry consortium sponsors to develop their data production maps for fundamental, mission critical business activities ranging from clinical data products to claims processing. In terms of a roadmap for the data quality journey that organizations inevitably will have to embark on to improve data quality, we are developing a model to show how firms will necessarily go from "the awareness stage to the stumbling stage to the operational stage." A journey it is for data quality practitioners and researchers.

References [1] Ballou, D., I. Chengalur-Smith and R. Y. Wang, A Sampling Procedure for Data Quality Auditing in the Relational Environment (No. TDQM-00-02). Massachusetts Institute of Technology, 2000. [2] Ballou, D. P. and H. L. Pazer, "Modeling Data and Process Quality in Multi-input, Multioutput Information Systems," Management Science, 31(2), 1985, pp. 150-162. [3] Ballou, D. P. and H. L. Pazer, "Process improvement versus enhanced inspection in optimized systems," International Journal of Production Research, 23(6), 1985, pp. 12331245. [4] Ballou, D. P. and H. L. Pazer, "Cost/Quality Tradeoffs for Control Procedures in Information Systems," International Journal of Management Science, 15(6), 1987, pp. 509521. [5] Ballou, D. P. and H. L. Pazer, "Designing inspection strategies for uncertain environments," Decision Sciences, 18(2), 1987, pp. 217-233. [6] Ballou, D. P. and H. L. Pazer, "Designing Information Systems to Optimize the Accuracy-Timeliness Tradeoff,," Information Systems Research, 6( 1), 1995, pp. 51-72. [7] Ballou, D. P. and G. K. Tayi, "Methodology for Allocating Resources for Data Quality Enhancement," Communications of the ACM, 32(3), 1989, pp. 320-329. [8] Ballou, D. P. and G. K. Tayi, "Enhancing Data Quality in Data Warehouse Environment," Communications of the ACM, 42(1), 1999, pp. 73-78.

Future Directions

145

[9] Ballou, D. P., R. Y. Wang, H. Pazer and G. K. Tayi, "Modeling Information Manufacturing Systems to Determine Information Product Quality," Management Science, 44(4), 1998, pp. 462-484. [10] Chengalur-Smith, I., D. Ballou and H. Pazer, "Dynamically Determine Optimal Inspection Strategies for Serial Production Processes," International Journal of Production Researcher, 30(1), 1992, pp. 169-187. [11] Chengalur-Smith, I. and L. L. Pipino. "Proceedings of the 1998 Conference on Information Quality," in Proceedings of. Cambridge, MA: pp. 320, 1998. [12] Chengalur-Smith, I., L. L. Pipino and R. Y. Wang, Sampling Procedures for Data Quality Auditing in Very Large Data Bases (TDQM-2000-06). Total Data Quality Management (TDQM) Research Program, MIT Sloan School of Management, 2000. [13] Chun, A. T. and B. Davidson. "Implementing the Information Quality Survey: A Case Study at Cedars-Sinai Health System," in Proceedings of Conference on Information Quality. Cambridge, MA: pp. 266-284, 1999. [14] Elmagarmid, A. and J. Sacks. "Proceedings of the Workshop on Data Quality: Challenges for Computer Science and Statistics," in Proceedings of Workshop on Data Quality: Challenges for Computer Science and Statistics. N.J.: pp. 2000. [15] English, L. P., Improving Data Warehouse and Business Information Quality. John Wiley & Sons, New York, NY, 1999. [16] Funk, J., Y. Lee and R. Wang. "Institutionalizing Information Quality Practice,". in Proceedings of Conference on Information Quality. Cambridge, MA: pp. 1-17, 1998. [17] Huang, K., Y. Lee and R. Wang, Quality Information and Knowledge. Prentice Hall, Upper Saddle River: N.J., 1999. [18] Jang, Y., A. T. Ishii and R. Y. Wang, "A Qualitative Approach to Automatic Data Quality Judgment," Journal of Organizational Computing, 5(2), 1995, pp. 101-121. [19] Katerattanakul, P. and K. Siau. "Measuring Information Quality of Web Sites: Development of an Instrument," in Proceedings of ICIS 1999. Charlotte, North Carolina: pp. 279-285, 1999. [20] Kopcso, D., L. Pipino and W. Rybolt. "The Assessment of Web Site Quality," in Proceedings of Conference on Information Quality. Cambridge, MA: pp. 2000. [21] Kovac, R., Y. W. Lee and L. L. Pipino. "Total Data Quality Management: The Case of IRI," in Proceedings of the 1997 Conference on Information Quality. Cambridge, MA: pp. 63-79, 1997. [22] Laudon, K. C., "Data Quality and Due Process in Large Interorganizational Record Systems," Communications of the ACM, 29(1), 1986, pp. 4-11.

146

Conclusion

Chapter 9

[23] Lee, Y. W. and G. K. Tayi. "Proceedings of the 1999 Conference on Information Quality," in Proceedings of. Cambridge, MA: pp. 342, 1999. [24] Liepins, G. E. and V. R. R. Uppuluri, ed. Data Quality Control: Theory and Pragmatics. D. B. Owen. Vol. 112. 1990, Marcel Dekker, Inc.: New York. 360 pages. [25] Missier, P., G. Lalk, V. S. Verykios, F. Grillo, T. Lorusso and A. P., "Improving Data Quality in Practice: A Case Study in the Italian Public Administration," Information Systems Journal, Special Issue on Data Extraction, Cleaning and Reconciliation, 2000. [26] Motro, A., "Superviews: Virtual Integration of Multiple Databases," SE-13(7), 1987, . [27] Motro, A., "Integrity = Validity + Completeness," ACM Transactions on Database Systems, 14(4), 1989, pp. 480-502. [28] Motro, A. and I. Rakov, Not all answers are equally good: Estimating the Quality of Database Answers. Flexible Query-Answering Systems, ed. T. Andreasen. Kluwer Academic Publishers, 1997. [29] Parssian, A., S. Sarkar and V. S. Jacob. "Assessing Data Quality for Information Products," in Proceedings of ICIS 1999. Charlotte, North Carolina: pp. 428-433, 1999. [30] Reddy, M. P. and R. Y. Wang. "Estimating Data Accuracy in a Federated Database Environment," in Proceedings of 6th International Conference, CISMOD (Also in Lecture Notes in Computer Science). Bombay, India: pp. 115-134, 1995. [31] Redman, T., "Data Quality for Telecommunications," 12(2), 1995, pp. 7. [32] Redman, T., "The Impact of poor Data Quality on the typical enterprise," Communications of the ACM, 41(2), 1998, pp. 79-82. [33] Redman, T. C., ed. Data Quality: Management and Technology. 1992, Bantam Books: New York, NY. 308 pages. [34] Redman, T. C., "Improve Data Quality for Competitive Advantage," Sloan Management Review, 36(2), 1995, pp. 99-109. [35] Redman, T. C., ed. Data Quality for the Information Age. 1996, Artech House: Boston, MA. 303 pages. [36] Schafer, E., J.-D. Becker and M. Jarke. "DB-Prism: Integrated Data Warehouses and Knowledge Net-works for Bank Controlling," in Proceedings of 26th International Conference on Very Large Data bases (VLDB). Cairo, Egypt: pp. 2000. [37] Segev, A. "On Information Quality and the WWW Impact," in Proceedings of the 1996 Conference on Information Quality. Cambridge, MA: pp. 16-23, 1996. [38] Shankaranarayan, G., R. Y. Wang and M. Ziad. "Modeling the Manufacture of an Information Product with IP-MAP," in Proceedings of Conference on Information Quality. Massachusetts Institute of Technology: pp. 2000.

Future Directions

147

[39] Strong, D. M. and B. K.Kahn, ed. Proceedings of the 1997 Conference on Information Quality. 1997, Cambridge, MA. 372 pages. [40] Strong, D. M., Y. W. Lee and R. Y. Wang, "Data Quality in Context," Communications of the ACM, 40(5), 1997, pp. 103-110. [41] Tayi, G. K. and D. Ballou, "Examining Data Quality," Communications of the ACM, 41(2), 1999, pp. 54-57. [42] Verykios, V. S., M. G. Elfeky, A. K. Elmagarmid, M. Cochinwala and S. Dalal. "On the Accuracy and Completeness of the Record Matching Process," in Proceedings of Information Quality Conference. Sloan School of Management, MIT, Cambridge, Mass.: pp. 2000. [43] Wang, R., J. Funk, Y. Lee and L. Pipino, Journey to Data Quality: A Roadmap to Higher Productivity. Work in Progress, 2001. [44] Wang, R. Y., ed. Proceedings of the 1996 Conference on Information Quality. 1996, Cambridge, MA. 270 pages. [45] Wang, R. Y., Y. L. Lee, L. Pipino and D. M. Strong, "Manage Your Information as Product: The Keystone to Quality Information," Sloan Management Review, forthcoming, 1997. [46] Wang, R. Y., V. C. Storey and C. P. Firth, "A Framework for Analysis of Data Quality Research," IEEE Transactions on Knowledge and Data Engineering, 7(4), 1995, pp. 623640. [47] Wang, R. Y. and D. M. Strong, "Beyond Accuracy: What Data Quality Means to Data Consumers," Journal of Management Information Systems (JMIS), 12(4), 1996, pp. 5-34.

This page intentionally left blank.

B I B L I O G R A P H Y

[1]

Angeles, P. A., Dictionary of Philosophy. Harper Perennial, New York, 1981.

[2]

Arnold, S. E., "Information Manufacturing: The Road to Database Quality," Database, 15(5), 1992, pp. 32.

[3]

Baader, F. and P. Hanschke. "A schema for integrating concrete domains into concept languages," in Proceedings of Twelfth International Joint Conference on Artificial Intelligence (IJCAI-91). Sydney, Australia: pp. 452-457, 1991.

[4]

Ballou, D., I. Chengalur-Smith and R. Y. Wang, A Sampling Procedure for Data Quality Auditing in the Relational Environment (No. TDQM-00-02). Massachusetts Institute of Technology, 2000.

[5]

Ballou, D. P. and H. L. Pazer, "Modeling Data and Process Quality in Multi-input, Multi-output Information Systems," Management Science, 31(2), 1985, pp. 150-162.

[6]

Ballou, D. P. and H. L. Pazer, "Process improvement versus enhanced inspection in optimized systems," International Journal of Production Research, 23(6), 1985, pp. 1233-1245.

[7]

Ballou, D. P. and H. L. Pazer, "Cost/Quality Tradeoffs for Control Procedures in Information Systems," International Journal of Management Science, 15(6), 1987, pp. 509-521.

[8]

Ballou, D. P. and H. L. Pazer, "Designing inspection strategies for uncertain environments," Decision Sciences, 18(2), 1987, pp. 217-233.

[9]

Ballou, D. P. and H. L. Pazer, "A Framework for the Analysis of Error in Conjunctive, Multi-Criteria, Satisficing Decision Processes," Journal of the Decision Sciences Institute, 21(4), 1990, pp. 752-770.

[10] Ballou, D. P. and H. L. Pazer, "Designing Information Systems to Optimize the Accuracy-Timeliness Tradeoff," Information Systems Research, 6(1), 1995, pp. 51-72. [11] Ballou, D. P. and G. K. Tayi, "Methodology for Allocating Resources for Data Quality Enhancement," Communications of the ACM, 32(3), 1989, pp. 320-329. [12] Ballou, D. P. and G. K. Tayi, "Enhancing Data Quality in Data Warehouse Environment," Communications of the ACM, 42(1), 1999, pp. 73-78.

150

Bibliography

[13] Ballou, D. P., R. Y. Wang, H. Pazer and G. K. Tayi, "Modeling Information Manufacturing Systems to Determine Information Product Quality, “ Management Science, 44(4), 1998, pp. 462-484. [14] Batini, C., M. Lenzirini and S. Navathe, "A comparative analysis of methodologies for database schema integration," ACM Computing Survey, 18(4), 1986, pp. 323-364. [15] Bitton, D. and D. DeWitt, "Duplicate Record Elimination in Large Data Files," ACM Transactions on Database Systems, 8(2), 1983, pp. 255-265. [16] Bowen, P. Managing Data Quality in Accounting Information Systems: A Stochastic Clearing System Approach. Unpublished Ph.D. dissertation, University of Tennessee, 1993. [17] Bressan, S., K. Fynn, C. Goh, M. Jakobisiak, K. Hussein, H. Kon, T. Lee, S. Madnick, T. Pena, J. Qu, A. Shum and M. Siegel, "Context Interchange Mediator Prototype," ACM SIGMOD, 1996. [18] Bressan, S., K. Fynn, C. Goh, S. Madnick, T. Pena and M. Siegel, ""Overview of a Prolog Implementation of the Context Interchange Mediator"," Proc. of the Intl. Conf. on Practical Applications of Prolog, 1996. [19] Buchheit, M., M. A. Jeusfeld, W. Nutt and M. Staudt, "Subsumption of queries over object-oriented databases," Information Systems, 19(1), 1994. [20] Bunge, M., Ontology I: The Furniture of the World. Treaties on Basic Philosophy, Vol. 3. D. Reidel Publishing Company, Boston, 1977. [21] Bunge, M., Ontology II: A World of Systems. Treaties on Basic Philosophy, Vol. 4. D. Reidel Publishing Company, Boston, 1979. [22] Calvanese, D., G. D. Giacomo, M. Lenzerini, D. Nardi and R. Rosati. "A principled approach to data integration and reconciliation in data warehousing," in Proceedings of Int. Workshop on Design and Management of Data Warehouses (DMDW'99). Heidelberg, Germany: pp. 1999. [23] Catarci, T. and M. Lenzerini, "Representing and using interschema knowledge in cooperative information systems," Journal of Intelligent and Cooperative Information Systems, 2(4), 1993, pp. 375-398. [24] CEIS. "Commander's Data Quality Assessment Guide for CEIS Beta Test - Region 3," in Proceedings of Corporate Executive Information System. 1216 Stanley Road, Fort Sam Houston, Texas 78234-6127: pp. 1996. [25] Ceri, S. and G. Pelagatti, Distributed Databases Principles & Systems. 1st ed. McGraw-Hill, 1984. [26] Chankong, V. and Y. Y. Haimes, Multiobjective Decision Making: Theory and Methodology. North-Holland Series in System Science and Engineering, ed. A. P. Sage. Elsevier Science, New York, N.Y., 1983.

Bibliography

151

[27] Chawathe, S., H. Garcia-Molina, J. Hammer, K. Ireland, Y. Papakonstantinou, J. Ullman and J. Widom. "The TSIMMIS project: Integration of heterogeneous information sources," in Proceedings of IPSI Conference. Tokyo, Japan: pp. 1994. [28] Chen, P. P., "The Entity-Relationship Model - Toward a Unified View of Data," ACM Transactions on Database Systems, 1, 1976, pp. 166-193, [29] Chen, P. S., "The Entity-Relationship Approach," in Information Technology in Action: Trends and Perspectives, R. Y. Wang, Editor 1993, Prentice Hall, Englewood Cliffs, 1993. [30] Chengalur-Smith, I., D. Ballou and H. Pazer, "Dynamically Determine Optimal Inspection Strategies for Serial Production Processes," International Journal of Production Researcher, 30(1), 1992, pp. 169-187. [31] Chengalur-Smith, I. and L. L. Pipino. "Proceedings of the 1998 Conference on Information Quality,". in Proceedings of. Cambridge, MA: pp. 320, 1998. [32] Chengalur-Smith, I., L. L. Pipino and R. Y. Wang, Sampling Procedures for Data Quality Auditing in Very Large Data Bases (No. TDQM-2000-06). Total Data Quality Management (TDQM) Research Program, MIT Sloan School of Management, 2000. [33] Chun, A. T. and B. Davidson. "Implementing the Information Quality Survey: A Case Study at Cedars-Sinai Health System," in Proceedings of Conference on Information Quality. Cambridge, MA: pp. 266-284, 1999. [34] Cochinwala, M., V. Kurien, G. Lalk and D. Shasha, Efficient Data Reconciliation (No. Bellcore, 1998. [35] Codd, E. F., "A Relational Model of Data for Large Shared Data Banks," Communications of the ACM, 13(6), 1970, pp. 377-387. [36] Codd, E. F., "Relational completeness of data base sublanguages," in Data Base Systems, R. Rustin, Editor 1972, Prentice Hall, 1972. [37] Codd, E. F., "Extending the relational database model to capture more meaning," ACM Transactions on Database Systems, 4(4), 1979, pp. 397-434. [38] Codd, E. F., “ Relational database: A Practical Foundation for Productivity, the 1981 ACM Turing Award Lecture," Communications of the ACM, 25(2), 1982, pp. 109-1 17. [39] Codd, E. F. "An evaluation scheme for database management systems that are claimed to be relational," in Proceedings of the Second International Conference on Data Engineering. Los Angeles, CA: pp. 720-729, 1986. [40] Codd, E. F., The Relational Model for Database Management: Version 2. AddisonWesley, Reading, MA, 1990. [41] Cooper, R. B. "Decision production - a step toward a theory of managerial information requirements," in Proceedings of the Fourth International Conference on Information Systems. Houston, TX: pp. 215-268, 1983.

152

Bibliography

[42] Corey, D., L. Cobler, K. Haynes and R. Walker. "Data Quality Assurance Activities in the Military Health Services System," in Proceedings of the I996 Conference on Information Quality. Cambridge, MA: pp. 127-153, 1996. [43] CRG, Information Quality Assessment (IQA) Software Tool. Cambridge Research Group, Cambridge, MA, 1997. [44] CRG, Integrity Analyzer: A Software Tool for Total Data Quality Management. Cambridge Research Group, Cambridge, MA, 1997. [45] CRG, ed. Cost-Benefit Analysis ofInformation Quality Case Study. 1999, 20 pages. [46] Dar, S., H. V. Jagadish, A. Y. Levy and D. Srivastava. "Answering queries with aggregation using views," in Proceedings of 22nd International Conference on Very Large Data Bases. Bombay, India: pp. 1996. [47] Daruwala, A,, C. H. Goh, S. Hofmeister, K. Hussein, S. Madnick and M. Siegel. "The context interchange network prototype," in Proceedings of The IFIP WG2.6 Sixth Working Conference on Database Semantics (DS-6). Atlanta, GA: pp. 1995. [48] Date, C. J. "Referential Integrity," in Proceedings of the Proceedings of the 7th International Conference on Very Large Data bases. Cannes, France: pp. 2-12, 1981. [49] Date, C. J. "The outer join," in Proceedings of The 2cd International Conference on Databases. Cambridge, England: pp. 76-106, 1983. [50] Date, C. J., An Introduction to Database Systems. 5th ed. Addison-Wesley Systems Programming Series, Addison-Wesley, Reading, 1990. [51] Delen, G. P. A. and B. B. Rijsenbrij, "The Specification, Engineering, and Measurement of Information Systems Quality," 17(3), 1992, pp. 205-217. [52] DeMichiel, L. G., Performing operations over mismatched domains. In pp. 36-45. Los Angeles, CA: 1989. [53] Deming, E. W., Out of the Crisis. Center for Advanced Engineering Study, MIT, Cambridge, MA, 1986. [54] Dobbie, G. and R. Topor, "On the declarative and procedural semantics of deductive object-oriented systems," Journal of Intelligent Information Systems, 4, 1995, pp. 193-219. [55] DQJ, Data Quality Journal, 1999. [56] Elmagarmid, A., B. Horowitz, G. Karabatis and A. Umar, Issues in Multisystem Integration for Achieving Data Reconciliation and Aspects of Solutions (No. Restricted and Proprietary). Bellcore Research, 1996. [57] Elmagarmid, A. and J. Sacks. "Proceedings of the Workshop on Data Quality: Challenges for Computer Science and Statistics," N.J.: pp. 2000.

Bibliography

153

[58] Elmasri, R., J, Larson and S. Navathe, Schema integration algorithms for federated databases and logical database design. [59] Emery, J. C., Organizational Planning and Control Systems: Theory and Technology. Macmillan, New York, 1969. [60] English, L. P., Improving Data Warehouse and Business Information Quality. John Wiley & Sons, New York, NY, 1999. [61] Fellegi, I. P. and A. B. Sunter, "A Theory for Record Linkage." 64(328), 1969, pp. 1183-1210. [62] Franconi, E. and U. Sattler. "A data warehouse conceptual data model for multidimensional aggregation," in Proceedings of Int. Workshop on Design and Management of Data Warehouses (DMD W'99). Heidelberg, Gemany: pp. 1999. [63] Funk, J., Y. Lee and R. Wang. "Institutionalizing Information Quality Practice," in Proceedings of Conference on Information Quality. Cambridge, MA: pp. 1-17, 1998. [64] Gebhardt, M., M. Jarke and S. Jacobs. "A toolkit for negotiation support interfaces to multi-dimensional data," in Proceedings of ACM-SIGMOD. Tucson, AZ: pp. 348-356, 1997. [65] Georgakopoulos, D., G. Karabatis and S. Gantimahapatruni, "Specification and Management of Interdependent Data in Operational Systems and Data Warehouses, '' Operational Systems and Data Warehouses Distributed and Parallel Databases, 5(2), 1997, pp. 121-166. [66] Giacomo, G. D. and M. Lenzerini. "What's in an aggregate: Foundations for description logics with tuples and sets," in Proceedings of Fourteenth International Joint Conference on Artificial Intelligence (IJCAI-95), 1995. [67] Giarratano, J. C., CLIPS User's Guide, Version 5.1, NASA Lyndon B. Johnson Space Center, 1991. [68] Goh, C., S. Madnick and M. Siegel, ""Context Interchange: Overcoming the Challenges of Large-scale interoperable database systems in a dynamic environment"," Proc. of the Intl Conf. On Information and Knowledge Management, 1994. [69] Goh, C. H., S. Bressan, S. E. Madnick and M. D. Siegel, Context Interchange: New Features and Formalisms for the Intelligent Integration of Information. Sloan School of Management, M.I.T., 1996. [70] Goh, C. H., S. Bressan, S. E. Madnick and M. D. Siegel, Context Interchange: Representing and reasoning about data semantics in heterogeneous systems, Sloan School Working Paper #3928. Sloan School of Management, M.I.T., 1996. [71] Gray, J., A. Bosworth, A. Layman and H. Pirahesh, "Data cube: A relational operator generalizing group-by, cross-tabs, and sub-totals," Data Mining and Knowledge Discovery Journal, 1(1), 1997.

154

Bibliography

[72] Hammer, J., H. Garcia-Molina, J. Widom, W. Labio and Y. Zhuge, "The Stanford Data Warehousing Project" Data Engineering, 18(2), 1995, pp. 41-48. [73] Harinarayan, V., A. Rajaraman and J. Ullman. "Implementing data cubes efficiently," in Proceedings of 1996 ACM SIGMOD International Conference on Management of Data. Montreal (Canada): pp. 205-227, 1996. [74] Hauser, J. R. and D. Clausing, "The House of Quality," Harvard Business Review, 66(3), 1988, pp. 63-73. [75] Hernadez, M. A. and S. J. Stolfo, "Real-world Data is Dirty: Data Cleansing and the Merge/Purge Problem," Journal of Data Mining and Knowledge Discovery, 1(2), 1998. [76] Huang, K., Y. Lee and R. Wang, Quality Information and Knowledge. Prentice Hall, Upper Saddle River: N.J., 1999. [77] Huh, Y. U., F. R. Keller, T. C. Redman and A. R. Watkins, "Data Quality," Information and Software Technology, 32(8), 1990, pp. 559-565. [78] Huh, Y. U., R. W. Pautke and T. C. Redman. "Data Quality Control," in Proceedings of ISQE, Juran Institute 7A1-27, 1992. [79] Huhns, M. N., N. Jacobs, T. Ksiezyk, W.-M. Shen, M. P. Singh and P. E. Cannata. "Integrating enterprise information models in Carnot," in Proceedings of Int. Conf. on Cooperative Information Systems (CoopIS-93), 32-42, 1993. [80] Hull, R. "Managing semantic heterogeneity in databases: A theoreti-cal perspective,". in Proceedings of Proceedings of the 16th ACM SIGACT SIGMOD SIGART Symposium on Principles of Database Systems (PODS). Tucson, AZ: pp. 5 1-61, 1997. [81] ICM, ICM Group, Inc., 1999. [82] ISO, ISO9000 International Standards for Quality Management. In International Standard Organization, Geneva, 1992. [83] Jakobisiak, M., Programming the web -- design and implementation of a multidatabase browser CISL WP #96-04. Sloan School of Management, M.I.T., 1996. [84] Jang, Y., A. T. Ishii and R. Y. Wang, "A Qualitative Approach to Automatic Data Quality Judgment, “Journal of Organizational Computing, 5(2), 1995, pp. 101-121. [85] Jang, Y. and Y. R. Wang, Data Quality Calculus: A data-consumer-based approach to delivering quality data, CISL-91-08. Composite Information Systems Laboratory, Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA, 02139, 1991. [86] Janson, M., "Data Quality: The Achilles Heel of End-User Computing," Omega Int. J. of Mgmt. Sci., 16(5), 1988, pp. 491-502.

Bibliography

155

[87] Jarke, M., R. Gallersdörfer, M. Jeusfeld, M. Staudt and S. Eherer, "ConceptBase - a deductive object base for meta data management," Journal of Intelligent Information Systems, 4(2), 1995, pp. 167-192. [88] Jarke, M., C. Quix, G. Blees, D. Lehmann, G. Michalk and S. Stierl, "Improving OLTP data Quality Using Data Warehouse Mechanisms," Sigmod Record, 28(2), 1999, pp. 536 - 537. [89] Jarke, M., C. Quix, D. Calvanese, M. Lenzerini, E. Franconi, S. Ligoudistianos, P. Vassiliadis and Y. Vassiliou. "Metadata-Driven Management of Data Warehouses: The DWQ Demonstrators," in Proceedings of Sigmod 2000, 2000. [90] Juran, J. M., Juran on Quality by Design: The New Steps for Planning Quality into Goods and Services. Free Press, New York, 1992. [91] Kahn, B. K., D. M. Strong and R. Y. Wang. "Information Quality Benchmarks: Product and Service Performance," Communications of the ACM, Forthcoming, 1999. [92] Kakas, A. C., R. A. Kowalski and F. Toni, "Abductive logic programming,"Journal of Logic and Computation, 2(6), 1993, pp. 719-770. [93] Katerattanakul, P. and K. Siau. "Measuring Information Quality of Web Sites." Development of an Instrument," in Proceedings of ICIS 1999. Charlotte, North Carolina: pp. 279-285, 1999. [94] Keeney, R. L. and H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs. John Wiley & Son, New York, 1976. [95] Kent, W., Data and Reality. North Holland, New York, 1978. [96] Khang, S. W., An Accounting Model-Based Approach Toward Integration of Multiple Financial Databases (No. WP # CIS-90-15). Sloan School of Management, MIT, 1990. [97] Khoshafian, S. N. and G. P. Copeland, "Object Identity," in The Morgan Kaufmann Series in Data Management Systems, S. B. Zdonik and D. Maier, Ed. 1990, Morgan Kaufmann, San Mateo, CA, 1990. [98] Kifer, M., G. Lausen and J. Wu, "Logical foundations of object-oriented and framebased languages," JACM, 4, 1995, pp, 741-843. [99] Klug, A., "Equivalence of relational algebra and relational calculus query languages having aggregate functions," The Journal of ACM, 29, 1982, pp. 699-717. [100] Kohavi, R., D. Sommerfield and J. Dougherty, Data mining using MLC++: A machine learning library in C++. Tools with Artificial Intelligence, IEEE Computer Society Press, 1996.

156

Bibliography

[101] Kovac, R., Y. W. Lee and L. L. Pipino. "Total Data Quality Management: The Case of IRI," in Proceedings of the 1997 Conference on Information Quality. Cambridge, MA: pp. 63-79, 1997. [102] Kraimer, K. L., W. H. Dutton and A. Northrup, The Management of Information Systems. Columbia University Press, New York, 198 1. [103] Kriebel, C. H. and O. Mikhail, "Dynamic Pricing of Resources in Computer Networks," 1980. [104] Lai, S. G. Data Quality Case Study - "Optiserv Limited". Master Thesis, MIT Sloan School of Management, 1993. [105] Laudon, K. C., "Data Quality and Due Process in Large Interorganizational Record Systems," Communications of the ACM, 29(1), 1986, pp. 4-11. [ 106] Lee, T. and S. Bressen. "Multimodal Integration of Disparate Information Sources with Attribution," in Proceedings of Entity Relationship Workshop on Information Retrieval and Conceptual Modeling 1997. [107] Lee, T., S. Bressen and S. Madnick. "Source Attribution for Querying Against Semistructured Documents," in Proceedings of Workshop on Web Information and Data Management, ACM Conference on Information and Knowledge Management1998. [ 108] Lee, T. and L. McKnight. "Internet Data Management: Policy Barriers to an Intermediated Electronic Market in Data," in Proceedings of 27th Annual Telecommunications Policy Research Conference1999. [109] Lee, Y. W. and G. K. Tayi. "Proceedings of the 1999 Conference on Information Quality," in Proceedings of. Cambridge, MA: pp. 342, 1999. [110] Levy, A. Y., A. 0. Mendelzon, Y. Sagiv and D. Srivastava. "Answering queries using views," in Proceedings of 14th Symposium on Principles of Database Systems. San Jose, CA: pp. 95-104, 1995. [111] Levy, A. Y. and I. S. Mumick. "Reasoning with aggregation constraints," in Proceedings of International Conference on Extending Database Technology (EDBT-96). Avignon, France: pp. 1996. [112] Liepins, G. E. and V. R. R. Uppuluri, ed. Data Quality Control: Theory and Pragmatics. D. B. Owen. Vol. 112. 1990, Marcel Dekker, Inc.: New York. 360 pages. [113] Ligoudistianos, S., D. Theodoratos and T. Sellis. "Experimental Evaluation of Data Warehouse Configuration Algorithms," in Proceedings of 9th DEXA Workshop (DEXA'98). Vienna, Austria: pp. 1998. [114] Little, J. D. C., On Model Building (No. 3556-93). MIT Sloan School of Management, 1993. [115] Llirbat, F., E. Simon and D. Tombroff. "Using versions in update transactions," in Proceedings of 23 rd VLDB Conf. Athens, Greece: pp. 1997.

Bibliography

157

[116] Madnick, S. and R. Y. Wang, Introduction to Total Data Quality Management (TDQM) Research Program, TDQM-92-01. Total Data Quality Management Program, MIT Sloan School of Management, 1992. [ 117] Madnick, S. E., "Challenges in the "on-and-off the ramps" of the Information Superhighway," Journal of Organizational Computing, 1995. [118] Madnick, S. E., "Database in the Internet Age," Database Programming and Design, 1997, pp, 28-33. [119] Maier, D., The Theory of Relational Databases. 1st ed. Computer Science Press, Rockville, MD, 1983. [ 120] Manber, U., Introduction to Algorithms. Addison-Wesley Publishing Company, 1989. [121] Missier, P., G. Lalk, V. S. Verykios, F. Grillo, T. Lorusso and A. P., "Improving Data Quality in Practice: A Case Study in the Italian Public Administration," Information Systems Journal, Special Issue on Data Extraction, Cleaning and Reconciliation), 2000. [122] Monge, A. and C. Elkan. "An Efficient Domain Independent Algorithm for Detecting Approximate Duplicate Database Records," in Proceedings of SIGMOD Workshop on Research Issues on DMKD1997. [123] Morey, R. C., "Estimating and Improving the Quality of Information in the MIS," Communications of the ACM, 25(5), 1982, pp. 337-342. [124] Motro, A., "Superviews: Virtual Integration of Multiple Databases," SE-13(7), 1987, . [125] Motro, A., "Integrity = Validity + Completeness," ACM Transactions on Database Systems, 14(4), 1989, pp. 480-502. [126] Motro, A. and I. Rakov, Not all answers are equally good: Estimating the QuaIity of Database Answers. Flexible Query-Answering Systems, ed. T. Andreasen. Kluwer Academic Publishers, 1997. [127] Mumick, I. S., D. Quass and B. S. Mumick. "Maintenance of Data Cubes and Summary Tables in a Warehouse," in Proceedings of ACM SIGMOD Conf. Tucson, AZ: pp. 1997. [ 128] Newcombe, H. B., "Record Linking: The Design of Efficient Systems for Linking Records into Individual and Family Histories," American Journal of Human Genetics, 19(3), 1967, pp. 335-339. [129] Nichols, D. R., "A Model of auditor's preliminary evaluations of internal control from audit data," 62(1), 1987, pp. 183-190. [130] Ousterhout, J. K., "Scripting: Higher-Level Programming for the 21st Century," Computer, 31, 1998, pp. 23-30. [131] Paradice, D. B. and W. L. Fuerst, "An MIS data quality methodology based on optimal error detection," 5(1), 1991, pp. 48-66.

158

Bibliography

[132] Parssian, A., S. Sarkar and V. S. Jacob. "Assessing Data Quality for Information Products," in Proceedings of ICIS 1999. Charlotte, North Carolina: pp. 428-433, 1999. [ 133] Pautke, R. W. and T. C. Redman. "Techniques to control and improve quality of data in large databases," in Proceedings of Statistics Canada Symposium 90. Canada: pp. 319333, 1990. [134] Pipino, L., Y. W. Lee and R. Y. Wang, Measurng Information Quality, TDQM-97-04. Total Data Quality Management Program, MIT Sloan School of Management, 1998. [135] Porter, M. and V. E. Millar, "How information gives you competitive advantages," 63(4), 1985, pp. 149-160. [136] Porter, M. E., ed. Competitive Advantage. 1985, Free Press: New York. 270 pages. [137] Qu, J. F., Data wrapping on the world wide web. CISL WP #96-05. Sloan School of Management, Massachusetts Institute of Technology, 1996. [138] Quass, D., A. Gupta, I. S. Mumick and J. Widom. "Making Views Self-Maintainable for Data Warehousing," in Proceedings of Conference on Parallel and Distributed Information Systems. Florida: pp. 1996. [139] Quinlan, J. R., "Induction of Decision Trees," Machine Learning, 1(1), 1986, pp. 81106. [ 140] Quinlan, J. R. "Generating Production Rules From Decision Trees," in Proceedings of 10th International Joint Conference on Artificial Intelligence 304-307, 1987. [ 141] Quinlan, J. R., "Simplifying Decision Trees," International Journal of Man-Machine Studies, 27, 1987, pp. 221-234. [ 142] Quinlan, J. R., "Learning Decision Tree Classifiers," ACM Computing Surveys, 28(1), 1996, pp. 71-72. [143] Reddy, M. P. and R. Y. Wang. "Estimating Data Accuracy in a Federated Database Environment," in Proceedings of 6th International Conference, CISMOD (Also in Lecture Notes in Computer Science). Bombay, India: pp. 115-134, 1995. [144] Redman, T., "Data Quality for Telecommunications," 12(2), 1995, pp. 7. [145] Redman, T., "The Impact of poor Data Quality on the typical enterprise," Communications of the ACM, 41(2), 1998, pp. 79-82. [146] Redman, T. C., ed. Data Quality: Management and Technology. 1992, Bantam Books: New York, NY. 308 pages. [147] Redman, T. C., "Improve Data Quality for Competitive Advantage," Sloan Management Review, 36(2), 1995, pp. 99-109. [148] Redman, T. C., ed. Data Quality for the Information Age. 1996, Artech House: Boston, MA. 303 pages.

Bibliography

159

[149] Rob, P. and C. Coronel, Database Systems: Design, Implementation, and Management. 3rd ed. Course Technology, Boston, 1997. [150] Ronen, B. and I. Spiegler, "Information as inventory: A new conceptual view," 21, 1991, pp. 239-247. [I51] Schafer, E., J,-D. Becker and M. Jarke. "DB-Prism: Integrated Data Warehouses and Knowledge Net-works for Bank Controlling," in Proceedings of 26th International Conference on Very Large Data bases (VLDB). Cairo, Egypt: pp. 2000. [152] Sciore, E., M. Siegel and A. Rosenthal, "Using Semantic Values to Facilitate Interoperability Among Heterogeneous Information Systems," ACM Transactions on Database Systems, 19(2), 1994, pp. 254-290. [153] Segev, A. "On Information Quality and the WWW Impact," in Proceedings of the I996 Conference on Information Quality. Cambridge, MA: pp. 16-23, 1996. [154] Shewhart, W. A., "The Application of Statistics as an Aid in Maintaining Quality of a Manufactured Product," 20, 1925, pp. 546-548. [155] Shum, A. Open Database Connectivity of the Context Interchange System. Master's, Massachusetts Institute of Technology, 1996. [156] Siegel, M. and S. Madnick, "Context Interchange: Sharing the Meaning of Data," 20(4), 1991, pp. 77-79 (December). [ 157] Siegel, M. and S. E. Madnick. "A metadata approach to resolving semantic conflicts," in Proceedings of the proceedings of the 17th International Conference on Very Large Data Bases (VLDB). Barcelona, Spain: pp. 133-145, 1991. [158] Siegel, M., E. Sciore and A. Rosenthal, Using Semantic Values to Facilitate Interoperability Among Heterogeneous Information Systems (No. 3543-93). Context Interchange Project, MIT Sloan School of Management, 1993. [159] Srivastava, D., S. Dar, H. V. Jagadish and A. Y. Levy. "Answering queries with aggregation using views," in Proceedings of 22. International Conference on Very Large Data Bases (VLDB-96)1996. [ 160] Staudt, M. and M. Jarke. "Incremental Maintenace of Externally Materialized Views," in Proceedings of VLDB. Bombay, India: pp. 1996. [161] Storey, V. C. and R. Y. Wang. "An Analysis of Quality Requirements in Database Design," in Proceedings of the 1998 Conference on Information Quality. Massachusetts Institute of Technology: pp. 64-87, 1998. [162] Strong, D. M. and B. K.Kahn, ed. Proceedings of the I997 Conference on Information Quality. 1997, Cambridge, MA. 372 pages. [163] Strong, D. M., Y. W. Lee and R. Y. Wang, "Data Quality in Context," Forthcoming in Communications of the ACM, MIT Working Paper TDQM-96-02, 1996.

160

Bibliography

[164] Strong, D. M., Y. W. Lee and R. Y. Wang, "Data Quality in Context," Communications of the ACM, 40(5), 1997, pp. 103-1 10. [165] T H Cormen, T. H., C. E. Leiserson and R. L. Rivest, Introduction to Algorithms. MIT Press, Cambridge, MA, USA, 1990. [166] Tayi, G. K. and D. Ballou, "Examining Data Quality," Communications of the ACM, 41(2), 1999, pp. 54-57. [167] TDQM, Total Data Quality Management Research Program, 1999. [168] Te'eni, D., "Behavioral Aspects of Data Production and Their Impact on Data Quality," Journal of Database Management, 4(2), 1993, pp. 30-38. [169] Teorey, T. J., Database Modeling and Design: The Entity-Relationship Approach. Morgan Kaufman Publisher, San Mateo, CA , 1990. [170] Therodoratos, D. and T. Sellis. "Data warehouse configuration," in Proceedings of 23rd VLDB Conf. Athens, Greece: pp, 1997. [171] Tsichritzis, D. and F. Lochovsky, Data Models. Prentice Hall, Englewood Cliffs, N.J., 1982. [ 172] TTI, Technology Transfer Institute, Inc. Website, 1999. [ 173] Ullman, J. D. "Information integration using logical views," in Proceedings of 6th Int. Conf. on Database Theory (ICDT-97), 19-49, 1997. [174] Vassiliadis, P. "Modeling multidimensional databases, cubes and cube operations," in Proceedings of 10th International Conference on Scientific and Statistical Database Management (SSDBM). Capri, Italy: pp, 1998. [175] Verykios, V. S., M. G. Elfeky, A. K. Elmagarmid, M. Cochinwala and S. Dalal. "On the Accuracy and Completeness of the Record Matching Process," in Proceedings of Infomation Quality Conference. Sloan School of Management, MIT, Cambridge, Mass.: pp. 2000. [176] Verykios, V. S., A. K. Elmagarmid and E. N. Houstis, Record Matching to Improve Data Quality (TR-99-005). Department of Computer Sciences, Purdue University, 1999. [177] Wand, Y. and R. Y. Wang, "Anchoring Data Quality Dimensions in Ontological Foundations," Forthcoming, Communications of the ACM, 1995. [178] Wand, Y. and R. Y. Wang, "Anchoring Data Quality Dimensions in Ontological Foundations," Communications of the ACM, 39(1 1), 1996, pp. 86-95. [179] Wand, Y. and R. Weber, "On the Ontological Expressiveness of Information Systems Analysis and Design Grammars," 3(3), 1993, pp. 217-237.

Bibliography

161

[180] Wang, R. and C. Firth, Using a Flow Model to Analyze the Business Impact of Data Quality (No. MIT Sloan School of Management, 1993. [181] Wang, R. Y., Proceedings of the 1996 Conference on Information Quality. 1996, Cambridge, MA. 270 pages. [182] Wang, R. Y., "A Product Perspective on Total Data Quality Management," Communications of the ACM, 41(2), 1998, pp. 58-65. [183] Wang, R. Y. and H. B. Kon, "Towards Total Data Quality Management (TDQM)," Information Technology in Action: Trends and Perspectives, Chapter 10, 1993, pp. 179197. [184] Wang, R. Y., H. B. Kon and S. E. Madnick. "Data Quality Requirements Analysis and Modeling," in Proceedings of the 9th International Conference on Data Engineering. Vienna: pp. 670-677, 1993. [185] Wang, R. Y., Y. L. Lee, L. Pipino and D. M. Strong, "Manage Your Information as a Product," Sloan Management Review, 39(4), 1998, pp. 95-105. [186] Wang, R. Y. and M. P. Reddy, A Data Quality Algebra for Estimating Query Result Quality (TDQM-95-11). Total Data Quality Management (TDQM) Research Program, MIT Sloan School of Management, 1995. [187] Wang, R. Y., M. P. Reddy and H. B. Kon, "Toward quality data: An attribute-based approach," Decision Support Systems (DSS), 13, 1995, pp. 349-372. [188] Wang, R. Y., P. Reddy and H. B. Kon, "An Attribute-based Model of Data for Data Quality Management," 1992. [189] Wang, R. Y., V. C. Storey and C. P. Firth, "A Framework for Analysis of Data Quality Research," IEEE Transactions on Knowledge and Data Engineering, 7(4), 1995, pp. 623-640. [190] Wang, R. Y. and D. M. Strong, "Beyond Accuracy: What Data Quality Means to Data Consumers," Journal of Management Information Systems (JMIS), 12(4), 1996, pp. 534. [191] Wang, R. Y. and D. M. Strong, "Beyond Accuracy: What Data Quality Means to Data Consumers," Journal of Management Information Systems, 12(4), 1996, pp. 5-34. [192] Wang, Y. R. and S. E. Madnick, Connectivity among information systems. Vol. 1. Composite Information Systems (CIS) Project, MIT Sloan School of Management, Cambridge, MA, 1988. [193] Wang, Y. R. and S. E. Madnick, "Facilitating connectivity in composite information systems," ACM Data Base, 20(3), 1989, pp. 38-46. [194] Wang, Y. R. and S. E. Madnick. "The inter-database instance identification problem in integrating autonomous systems," in Proceedings of the Fifth International Conference on Data Engineering. Los Angeles, CA: pp. 46-55, 1989.

162

Bibliography

[195] Wang, Y. R. and S. E. Madnick. "A Polygen Model for Heterogeneous Database Systems: The Source Tagging Perspective," in Proceedings of the 16th International Conference on Very Large Data bases (VLDB). Brisbane, Australia: pp. 519-538, 1990. [196] Wang, Y. R. and S. E. Madnick. "A Source Tagging Theory for Heterogeneous Database Systems," in Proceedings of International Conference on Information Systems. Copenhagen, Denmark: pp. 243-256, 1990. [ 197] Wellman, M. P., Formulation of Tradeoffs in Planning Under Uncertainty. Pitman and Morgan Kaufmann, 1990. [ 198] Wellman, M. P. and J. Doyle. "Preferential Semantics for Goals," in Proceedings of the 9th National Conference on Artificial Intelligence 1991. [199] Wiener, J. L., H. Gupta, W. J. Labio, Y. Zhuge, H. Garcia-Molina and J. Widom. "A System Prototype for Warehouse View Maintenance," in Proceedings of the ACM Workshop on Materialised Views: Techniques and Applications. Montreal, Canada: pp. 26-33, 1996. [200] Winkler, W. E., Record Linkage Software (No. US. Bureau of the Census, 1999. [201] Yu, S. and J. Neter, "A Stochastic Model of the Internal Control System," Journal of Accounting Research, 1(3), 1973, pp. 273-295. [202] Yuan, Y., The design and implementation of system P: A polygen database management system (CIS-90-07). MIT Sloan School of Management, Cambridge, MA 02139, 1990. [203] Zhou, G., R. Hull and R. King, "Generating Data Integration Mediators that Use Materialization," Journal of Intelligent Information Systems, 6(2), 1996, pp. 199-221. [204] Zhuge, Y., H. Garcia-Molina, J. Hammer and J. Widom. "View maintenance in a warehousing environment," in Proceedings of ACM SIGMOD Int. Conf. Management of Data 316-327, 1995.

I N D E X

A accessibility, 104 accuracy, 4,65 accuracy estimation, 71 Act, 5 agents, 99 algebraic operations, 29 algorithms, 58 analysis matrix, 11 Analyze, 5 analyze IP, 13 API, 86 application attributes, 28 quality requirements, 38, 39 requirements, 39 artificial intelligence, 52 atomic, 66 property, 32 unit, 31 attribute, 26 gerund representation, 42, 43 value, 26, 31 values, 41 attribute-based model, 25 attributes, 23, 31 Autoclass, 132 automating data quality judgment, 49

B believability, 31 best case, 73 blocking, 125

C Cartesian Product, 23 cell, 22 cell-level tagging, 25 certainty factor, 131 characteristics, 26 Check, 5 check inspection, 9 classification, 129 clients, 99 clustering, 129 clustering algorithm, 126 coalesce, 24 Codd, 26, 96 COIN, 89 comparison space, 123 completely-reducible quality-merge statement, 57 completeness, 9 component-of relationships, 37 ConceptBase, 111 conceptual design, 44 conceptual perspective, 103 consistency, 12 constructs, 27 consumers, 11 context knowledge, 83 context definitions, 90 Context Interchange (COIN), 83 context knowledge, 82 context mediation, 86 contextual information quality, 4 contraint logic, 107 cooperative problem-solving, 6 1 Course, 38

164 course standards, 45 credibility, 9

D data, 2 acquisition, 61 aggregation, 107 authenticity, 31 characteristics, 96 consumers, 50 correction, 122 customization, 107 deficient, 65 edits, 122 extraction, 81, 107 integrity, 19, 31 interpretation, 81 manipulation, 27 manufacturing, 51 pre-processing, 126 reconciliation, 107 sources, 65 data quality, 2, 37, 49 algebra, 71 attributes, 49 filtering, 52 indicators, 49 judgment, 5 1 measure entity, 42 reasoner, 50, 52 requirements, 38, 39, 49 Data Quality algebra, 65 Separation Priciple, 41 data quality blocks, 10 data structure, 26 data units, 10 Data warehouse components, 99 Data Warehouse Quality (DWQ) project, 95 Data Warehouse Schema, 113 data warehousing, 84 database management systems, 19 data-quality-by-design, 37 DB2, 19

Index

decision tree, 126 inducer, 129 simplification, 130 transformation, 130 decision-analytic approach, 52 decision-maker, 52 Define, 5 Define IP, 7 deletion, 32 Deming, 5 designer, 45 destination databases, 99 deterministic tuple accuracy, 68 difference, 24, 31 dimension, 45, 66 dimension name, 42 dimensions, 4, 8, 13 accessibility, 82 interpretability, 82 Do, 5 domain mapping, 22 domain model, 89 dominance relation, 53 dominance relationships, 51 DQ measure, 45 DWQ architectual framework, 102 DWQ objectives, 98 DWQ project, 97

E elevation axioms, 90 Enterprise Model, 112 entities, 37 entity, 42 entity-relationship model, 7, 43 Entity-Relationship model, 110 Entity-Relationship Model, 38, 142 ER diagram, 7 ER model, 37, 142 error distribution non-uniform, 73 European Union, 95 expert system, 137

Index

F facilitation, 86 false matches, 124 first-order data quality reasoner, 57 first-order dominance relation, 57 framework, 40 functionalities, 7

H heterogeneous sources, 82 heterogeneous database systems, 20 higher-order data quality reasoner, 61 HTTP, 86

165

intersection join, 24 IP, 7, 11 characteristics, 9 definition, 11 managers, 11 roles, 13 team, 6, 13, 14 irreducible quality-merge statement, 57

J join, 24

K knowledge acquisition, 61 knowledge-based model, 49

I Improve, 5 improve IP, 14 inaccuracy, 12 incompleteness, 12 indominance relation, 54 information, 2 collectors, 4 consumers, 4 custodians, 4 dissemination, 84 extraction, 84 flow, 14 interpretation, 84 manufacturing, 3, 11 manufacturing system, 4, 6, 10 product, 6, 14 quality, 4 suppliers, 11 insertion, 32 instance evolution, 109 instance matching, 22 instructor, 39 integer programming, 14 integrity constraints, 65, 108 interdependency, 112 interface, 86 interpretability, 54, 104

L linearization, 128 link, 129 local dominance relationships, 52 Local Query Processors, 21 local-dominance relationships, 53 logical perspective, 103

M manufacturers, 11 many-to-many relationship, 45 matching process, 125 matching variables, 123 Measure, 5 measure IP, 12 mediation, 88 metadata, 50 meta-database, 99 methodologies, 37 metric edit-distance, 132 metrics, 12, 13 middleware, 86 Military Treatment Facilities, 13 mismember tuple, 68 modeling data quality requirements, 41

166

modification, 32 modifiers, 92 MOLAP products, 104 multi-dimensional, 4 multi-dimensional aggregation, 1 15 multiple source, 20 mutual exclusivity, 55

N non-link, 129 normal form, 66 null relation accuracy, 70

O ODBC, 86 OnLine Analytic Processing (OLAP), 96 operational data store (ODs), 115 Oracle, 19, 84 orthogonal, 29 outer natural join, 24

P Pareto chart analysis, 13 pattern recognition, 13 performance, 104 physical perspective, 104 Plan, 5 planning, 52 polygen algebra, 23, 25 data structure, 22 domain, 22 model, 20, 25 relation, 22 schema, 22 Polygen Query Processor, 21 possible link, 129 primary key, 42, 44 probabilistic attribute accuracy, 70 probabilistic tuple accuracy, 70 product, 39 product manufacturing, 3

Index

project, 23 projection, 30 projection operator, 73 propagation, 108

Q QER model, 49 QI-compatability, 27 QI-Compatible, 31 QIV-Equal, 27 Q-Merge Algorithm, 60 Q-Reduction Algorithm, 58 quality cell, 26 control, 3 1 database, 26 indicator, 27, 3 1 indicator algebra, 29, 32 indicator tuple, 26 indicators, 50, 51 key value, 26 merge, 55 parameters, 50, 51 requirements identification, 39 schema, 26 scheme set, 26 Quality Entity-Relationship model, 49 quality-estimating problem, 60 quantitative design optimization, 110 query, 24 query optimization, 108 Query Process Architecture, 21 query processing, 66 query results, 65

R rating, 42 receiver, 82 reduction-based evaluation, 56 relation accuracy, 69 relational algebra, 66 relational model, 19, 49 relational scheme, 68 reliability, 104

Index

requirements modeling, 40 Restrict, 23 ROLAP products, 104

S sampling, 128 schema, 22, 109 Inter-schema correspondences, 116 Schema, 8 searching process, 124 security, 12 select, 24 selection, 30 selection operator, 71 semantic reconciliation, 22 semi-structured format, 87 share price, 9 sorted neighborhood, 128 sorting, 126 source, 81 integration, 111 Source Model, 112 source reputation, 51 Source Schema, 112 source tagging, 20, 22, 32 SourcePoint, 105 sources, 99 SQL, 84, 108 SQL server, 19 Squirrel project, 104 standard rating, 44 statistical process control. See SPC

T target payoffs, 13

167

TDQM Cycle, 5,6 TDQM framework, 6, 14 TDQM program, 85 test sets, 129 timeliness, 3, 9, 12 Total Data Quality Management TDQM, xiii Total Quality Management, 3 TQM, 5 training set, 129 transaction characteristics, 96 TSIMMIS project, 104 tuple, 23, 32 tuple membership, 68 tuples, 66

U union, 30 Union, 24 unstructured plain text format, 87 update propagation, 109 user-specific quality parameters, 50

W WHIPS project, 104 work flow, 14 worst case, 72 wrappers, 99 wrapping, 87

X XML, 81