ADVANCES IN INFORMATION RETRIEVAL
Recent Research from the Center for Intelligent Information Retrieval
THE KLUWER INTERNATIONAL SERIES ON INFORMATION RETRIEVAL Series Editor W. Bruce Croft University of Massachusetts, Amherst
Also in the Series: MULTIMEDIA INFORMATION RETRIEVAL: Content-Based Information Retrieval from Large Text and Audio Databases, .. by Peter Schauble; ISBN: 0-7923-9899-8 INFORMATION RETRIEVAL SYSTEMS, by Gerald Kowalski; ISBN: 0-7923-9926-9 CROSS-LANGUAGE INFORMATION RETRIEVAL, edited by Gregory Grefenstette; ISBN: 0-7923-8 122-X TEXT RETRIEVAL AND FILTERING: Analytic Models of Performance, by Robert M. Losee; ISBN: 0-7923-8177-7 INFORMATION RETRIEVAL: UNCERTAINTY AND LOGICS: Advanced Models for the Representation and Retrieval of Information, by Fabio Crestani, Mounia Lalmas, and Cornelis Joost van Rijsbergen; ISBN: 0-7923-8302-8 DOCUMENTCOMPUTING: Technologies for Managing Electronic Document Collections, by Ross Wilkinson, Timothy Arnold-Moore, Michael Fuller, Ron Sacks-Davis, James Thom, and Justin Zobel; ISBN: 0-7923-8357-5 AUTOMATIC INDEXING AND ABSTRACTING OF DOCUMENT TEXTS, by Marie-Francine Moens; ISBN 0-7923-7793-1
ADVANCES IN INFORMATION RETRIEVAL
Recent Research from the Center for Intelligent Information Retrieval
Edited by W. Bruce Croft University of Massachusetts, Amherst
KLUWER ACADEMIC PUBLISHERS New York / Boston / Dordrecht / London / Moscow
eBook ISBN: Print ISBN:
0-306-47019-5 0-792-37812-1
©2002 Kluwer Academic Publishers New York, Boston, Dordrecht, London, Moscow Print ©2000 Kluwer Academic Publishers Massachusetts All rights reserved No part of this eBook may be reproduced or transmitted in any form or by any means, electronic, mechanical, recording, or otherwise, without written consent from the Publisher Created in the United States of America Visit Kluwer Online at: and Kluwer's eBookstore at:
http://kluweronline.com http://ebooks.kluweronline.com
Contents
Preface
ix
xiii Contributing Authors 1 Combining Approaches to Information Retrieval 1 W. Bruce Croft 1 Introduction 1 2 Combining Representations 5 3 Combining Queries 9 4 Combining Ranking Algorithms 11 5 Combining Search Systems 13 6 Combining Belief 16 7 Language Models 25 28 8 Conclusion 2 The Use of Exploratory Data Analysis in 37 Information Retrieval Research Warren R. Greiff 1 Introduction 37 2 Exploratory Data Analysis 39 3 Weight of Evidence 40 4 Analysis of the Relationship between Document Frequency and the 43 Weight of Evidence of Term Occurrence 5 Probabilistic Modeling of Multiple Sources of Evidence 53 6 Conclusions 70 3 Language Models for Relevance Feedback 73 Jay M. Ponte 1 Introduction 73 2 The Language Modeling Approach to IR 75 3 Related Work 78 4 Query Expansion in the Language Modeling Approach 83
vi
ADVANCES IN INFORMATION RETRIEVAL
5
Discussion and Future Work
4 Topic Detection and Tracking: Event Clustering as a Basis for First Story Detection Ron Papka James Allan 1 Topic Detection and Tracking 2 On-lineClustering Algorithms 3 Experimental Setting 4 Event Clustering 5 First Story Detection 6 Discussion of First Story Detection 7 Conclusion 8 Future Work 5 Distributed Information Retrieval Jamie Callan 1 Introduction 2 Multi-Database Testbeds 3 Resource Description 4 Resource Selection 5 Merging Document Rankings 6 Acquiring Resource Descriptions 7 Summary and Conclusions 6 Topic-Based Language Models for Distributed Retrieval Jinxi Xu W. Bruce Croft 1 Introduction 2 Topic Models 3 K-Means Clustering 4 Four Methods of Distributed Retrieval Experimental Setup 5 6 Global Clustering Recall-basedRetrieval 7 8 Distributed Retrieval in Dynamic Environments 9 More Clusters 10 Better Choice of Initial Clusters 11 Local Clustering 12 Multiple-Topic Representation 13 Efficiency 14 RelatedWork 15 Conclusion and FutureWork 7 The Effect of Collection Organization and Query Locality on Information Retrieval System Performance Zhihong Lu Kathryn S. McKinley 1 Introduction 2 Related Work
92 97 98 103 108 110 112 119 120 122 127 127 129 130 131 135 137 145 151 152 154 155 155 158 160 163 165 165 165 166 166 168 168 169 173
174 176
Contents
3 4 5 6 7
SystemArchitectures Configuration with Respect to Collection Organization, Collection Access Skew, and Query Locality SimulationModel Experiments Conclusions
8 Cross-LanguageRetrieval via TransitiveTranslation Lisa A. Ballesteros 1 Introduction 2 Translation Resources 3 Dictionary Translation and Ambiguity 4 Resolving Ambiguity 5 Addressing Limited Resources 6 Summary 9 Building, Testing, and Applying Concept Hierarchies Mark Sanderson Dawn Lawrie 1 Introduction 2 Building a Concept Hierarchy 3 Presenting a Concept Hierarchy 4 Evaluating the Structures 5 Future Work 6 Conclusions Appendix: ANOVA analysis 10 Appearance-Based Global Similarity Retrieval of Images S. Ravela C. Luo 1 Introduction 2 Appearance Related Representations 3 Computing Global Appearance Similarity 4 TrademarkRetrieval 5 Conclusions and Limitations Index
vii
181 185 188 189 197 203 203 205 208 209 212 230 235 235 238 246 251 255 261 262 267 268 272 278 293 299 305
Preface
The Center for Intelligent Information Retrieval (CIIR) was formed in the Computer Science Department of the University of Massachusetts, Amherst in 1992. The core support for the Center came from a National Science Foundation State/Industry/University Cooperative Research Center (S/IUCRC) grant, although there had been a sizeable information retrieval (IR) research group for over 10 years prior to that grant. The basic goal of these Centers is to combine basic research, applied research, and technology transfer. The CIIR has been successful in each of these areas, in that it has produced over 270 research papers, has been involved in many successful government and industry collaborations, and has had a significant role in high-visibility Internet sites and start-ups. As a result of these efforts, the CIIR has become known internationally as one of the leading research groups in the area of information retrieval. The CIIR focuses on research that results in more effective and efficient access and discovery in large, heterogeneous, distributed, text and multimedia databases. The scope of the work that is done in the CIIR is broad and goes significantly beyond “traditional” areas of information retrieval such as retrieval models, cross-lingual search, and automatic query expansion. The research includes both low-level systems issues such as the design of protocols and architectures for distributed search, as well as more human-centered topics such as user interface design, visualization and data mining with text, and multimedia retrieval. The papers in this book contain some of the more recent research results from the CIIR. The first group of papers present new research related to the retrieval models that underly IR systems. The first paper, Combining Approaches to Information Retrieval by Croft, discusses retrieval models and strategies for combining evidence from multiple document representations, queries, ranking algorithms and search systems. This has been an important line of research for more than 10 years, and this paper provides a framework for understanding the many experimental results in this area and indicates how recent work on
x
ADVANCES IN INFORMATION RETRIEVAL
language models contributes to these results. Greiff’s paper, The Use of Exploratory Data Analysis in Information Retrieval Research, introduces a datadriven approach to developing retrieval models and uses this approach to derive a probabilistic ranking formula from an analysis of TREC data. A number of retrieval experiments are used to validate this new model. In the third paper of this group, Language Models for Relevance Feedback, Ponte describes the language modeling approach to IR that he introduced in his thesis work, and then shows how this approach can be used for relevance feedback and filtering environments. A number of experiments demonstrate the effectiveness of this conceptually simple, but potentially very powerful retrieval model. The next paper, Topic Detection and Tracking: Event Clustering as a Basis for First Story Detection by Allan and Papka, describes a relatively new area of research that focuses on detecting significant events in broadcast news. New algorithms and modifications of existing IR techniques are presented and evaluated in the context of this novel task. The next three papers deal with a range of topics related to distributed information retrieval. In Distributed Information Retrieval, Callan gives an overview of this area of research and summarizes the results related to database description and selection, and merging rankings from multiple systems. Xu and Croft, in their paper Topic-Based Language Models for Distributed Retrieval, present recent results from an approach to describing databases that is based on identifying language models through clustering. Lu and McKinley discuss performance-related issues in their paper The Effect of Collection Organization and Query Locality on Information Retrieval System Performance. They show that the use of database replication combined with selection algorithms can significantly improve the efficiency and scalability of distributed retrieval. The next paper, Cross-Language Retrieval via Transitive Translation by Ballesteros, discusses the language resources and techniques that are used for IR in multiple languages. The paper describes a series of experiments using a dictionary-based approach to transitive translation and retrieval through an intermediate language. In Building, Testing and Applying Concept Hierarchies, Sanderson and Lawrie describe research in the important new area of summarization. They focus specifically on a technique for constructing a hierarchy of concepts to summarize the contents of a group of documents, such as those retrieved by a query. The last paper, Appearance-Based Global Similarity Retrieval of Images by Ravela and Luo, presents new research in the important, emerging area of image retrieval. Because many of the techniques used for image indexing and comparison are very different to those used for text retrieval, the paper contains an extensive introduction to those techniques. Retrieval evaluations of a new indexing technique based on image “appearance” are described and discussed.
PREFACE
xi
These papers, like the research in the CIIR, cover a wide variety of topics in the general area of IR. Together, they represent a snapshot of the “state-ofthe-art” in information retrieval at the turn of the century and at the end of a decade that has seen the advent of the World-Wide Web. The papers have been written to provide overviews of their subareas and to serve as source material for graduate and undergraduate courses in information retrieval. Finally, I would liketo acknowledge thefaculty, staff, and students associated with the CIIR since 1992 who have contributed enormously to its success. In particular, Jean Joyce, Kate Moruzzi, and Glenn Stowell havebeen instrumental to the Center’s operation. I would also like to thank Win Aung, ourNSF Program Manager, for his support over the years. BRUCE CROFT
Contributing Authors
James Allan is an Assistant Professor in the Computer Science Department and Assistant Director of the CIIR at the University of Massachusetts, Amherst. He received the Ph.D. in Computer Science from Cornell University in 1995. His research interests are in information retrieval and organization, as well as topic detection and tracking. Lisa Ballesteros is an Assistant Professor in the Computer Science Department at Mount Holyoke College, Massachusetts. She received the B.S. degree in Biology from Union College, Schenectady, N.Y. in 1987 and the M.S. degree in Computer Science from the University of Massachusetts, Amherst in 1996. She is currently a Ph.D. candidate in Computer Science at U.Mass. Her major research interest is cross-language retrieval. Jamie Callan received a Ph.D. in Computer Science from the University of Massachusetts, Amherst, in 1993. He was a Research Assistant Professor at U.Mass. from 1994 to 1999, and was Assistant Director of the CIIR from 1995 to 1999. He has been an Associate Professor at Carnegie Mellon University since the Fall of 1999. Dr. Callan’s research interests include software architectures for IR, distributed information retrieval, information filtering, and use of the Internet and IR tools for K-12 education. W. Bruce Croft is a Professor of Computer Science and Director of the CIIR at the University of Massachusetts, Amherst. He holds a Ph.D. from Cambridge University, England and has been active in information retrieval research since 1975.
xiv
ADVANCES IN INFORMATION RETRIEVAL
Warren Greiff received his Bachelor’s Degree in Mathematics from Case Institute of Technology and Master’s Degrees from the University of Pennsylvania and Antioch College in Computer Science and Education, respectively. After a number of years working in American industry, he spent 12 years as a professor at the University of the Americas in Puebla, Mexico. In 1994, he returned to the United States to pursue a doctoral degree, which he received from the University of Massachusetts, Amherst in 1999. He is currently pursuing his research interests in statistical techniques applied to natural language processing technology at the Mitre Corporation in Bedford, Massachusetts. Dawn Lawrie is a Computer Science graduate student at the University of Massachusetts, Amherst. She received her bachelor’s degree in Computer Science from Dartmouth College in 1997. Currently she is studying the organization of information and text mining at the CIIR. Zhihong Lu received her Ph.D. in Computer Science from the University of Massachusetts, Amherst, and is currently working in AT&T network services. Her dissertation work is on scalable distributed architectures for information retrieval. Chen Luo is a graduate student at the University of Massachusetts, Amherst. He received his B.S. from the Automation Department of Tsinghua University, China. His research interests are image retrieval, pattern recognition and computer vision. Kathryn S. McKinley is an Associate Professor at the University of Massachusetts. She received her Ph.D. from Rice University in 1992. Her research interests include compilers, architectures, parallel and distributed computing with a particular focus on memory hierarchy and program analysis, and information retrieval. Ron Papka is currently the Director of Research and Development at Dataware Technologies. He is also an Adjunct Professor of Computer Science at Smith College. He received his Ph.D. in Computer Science from the University of Massachusetts in 1999, an M.S. from Brown University in 1993, and B.S. from Columbia University in the same discipline. His research interests include Information Retrieval, Machine Learning, and Object-Oriented Programming. Jay M. Ponte received his B.S. in computer science from Northeastern University in 1989. He received his M.S. and Ph.D. degrees in Computer Science
Contributing Authors
xv
from the University of Massachusetts in 1996 and 1998 respectively, where he worked in the CIIR. He is currently a Principal Member of Technical Staff at GTE Laboratories where he continues to specialize in information retrieval and statistical natural language processing. S. Chandu Ravela is a doctoral candidate at the University of Massachusetts, Amherst. Mr. Ravela’s research interests are visual media representations, computer vision, scale-space and multi-sensor data abstractions. Mr. Ravela received his M.S.C.S in 1994 and B.E. in Computer Science and Engineering from the Regional Engineering College, Trichy, India in 1991 and M.S. in Computer Science from the U.Mass. in 1994. Mark Sanderson is a lecturer in the Information Science department at the University of Sheffield. Prior to this, he was a postdoctoral researcher in the CIIR at the University of Massachusetts. He completed his Ph.D. on disambiguation and IR at the University of Glasgow in 1997 and works on summarization, hypertext retrieval, and spoken document retrieval. Jinxi Xu received his Ph.D. in Computer Science from the University of Massachusetts, Amherst in 1997, and was a post-doctoral research associate for two years at the CIIR. He is now a scientist at BBN Technologies in Cambridge, Massachusetts. His research interests include information retrieval and natural language processing.
Chapter 1 COMBINING APPROACHES TO INFORMATION RETRIEVAL W. Bruce Croft Department of Computer Science University of Massachusetts, Amherst
[email protected]
Abstract
1
The combination of different text representations and search strategies has become a standard technique for improving the effectiveness of information retrieval. combination, for example, has been studied extensively in the TREC evaluations and is the basis of the “meta-search” engines used on the Web. This paper examines the development of this technique, including both experimental results and the retrieval models that have been proposed as formal frameworks for combination. We show that combining approaches for information retrieval can be modeled as combining the outputs of multiple classifiers based on one or more representations, and that this simple model can provide explanations for many of the experimental results. We also show that this view of combination is very similar to the inference net model, and that a new approach to retrieval based on language models supports combination and can be integrated with the inference net model.
INTRODUCTION
Information retrieval (IR) systems are based, either directly or indirectly, on models of the retrieval process. These retrieval models specify how representations of text documents and information needs should be compared in order to estimate the likelihood that a document will be judged relevant. The estimates of the relevance of documents to a given query are the basis for the document rankings that are now a familiar part of IR systems. Examples of simple models include the probabilistic or Bayes classifier model (Robertson and Sparck Jones, 1976; Van Rijsbergen, 1979) and the vector space model (Salton et al., 1975). Many others have been proposed and are being used (Van Rijsbergen, 1986; Deerwester et al., 1990; Fuhr, 1992; Turtle and Croft, 1992).
2
ADVANCES IN INFORMATION RETRIEVAL
As these retrieval models were being developed, many experiments were carried out to test the effectiveness of these approaches. Quite early in these experiments, it was observed that different retrieval models, or alternatively, variations on ranking algorithms, had surprisingly low overlap in the relevant documents that were found, even when the overall effectiveness of the algorithms was similar (e.g. McGill et al., 1979; Croft and Harper, 1979). Similar studies showed that the practice of searching on multiple document representations such as title and abstract or free text and manually assigned index terms was more effective than searching on a single representation (e.g. Fisher and Elchesen, 1972; McGill et al., 1979; Katzer et al., 1982). These, and other studies, suggested that finding all the relevant documents for a given query was beyond the capability of a single simple retrieval model or representation. The lack of overlap between the relevant documents found by different ranking algorithms and document representations led to two distinct approaches to the development of IR systems and retrieval models. One approach has been to create retrieval models that can explicitly describe and combine multiple sources of evidence about relevance. These models are typically probabilistic and are motivated by the Probability Ranking Principle (Robertson, 1977), which states that optimal retrieval effectiveness is achieved by ranking documents in decreasing order of probability of relevance and that “probabilities are estimated as accurately as possible on the basis of whatever data has been made available to the system”. The INQUERY system, for example, is based on a probabilistic model that is explicitly designed to combine evidence from multiple representations of documents and information needs (Turtle and Croft, 1991; Callan et al., 1995a). The other approach has been to design systems that can effectively combine the results of multiple searches based on different retrieval models. This combination can be done in a single system architecture (e.g. Croft and Thompson, 1987; Fox and France, 1987) or in a distributed, heterogeneous environment (e.g. Lee, 1995; Voorhees et al., 1995; Callan et al., 1995b). Combining multiple, heterogeneous searches is the basis of the “metasearch’’ engines on the Web (e.g., MetaCrawler1 ) and has become increasingly important in multimedia databases (e.g. Fagin, 1996). The motivation for both these approaches is to improve retrieval effectiveness by combining evidence. Apart from the empirical results, theoretical justification for evidence combination is provided by a Bayesian probabilistic framework (e.g. Pearl, 1988). In this framework, we can describe how our belief in a hypothesis H is incrementally affected by a new piece of evidence e. Specifically, using log-odds: log O(H|E, e) = log O(H|E) + log L(e|H) 1
http://www.metacrawler.com
Combining Approaches to Information Retrieval 3 where E is all the evidence seen prior to e, O(H|E) =
is the posterior odds on H given evidence E,
O(H| E, e) is the odds on H given the new evidence e, and L(e|H) =
is the likelihood ratio of evidence e.
This formulation makes it clear that each additional piece of positive evidence (i.e. with likelihood > 1) increases the odds of the hypothesis being true. A piece of evidence with very strong likelihood can have a substantial impact on the odds. In addition, the effect of a large error in the estimation of the likelihood for one piece of evidence can be reduced by additional evidence with smaller errors. In other words, the average error can be smaller with more evidence. This analysis assumes that the evidence is conditionally independent and, therefore, P(e|E,H) = P(e|H). If, however, the new evidence is correlated with the previous evidence, the impact of that new evidence will be reduced. If the new evidence can be directly inferred from the previous evidence, P(e|E, H) = 1 and the probability of the hypothesis being true does not change. In retrieval models, the hypothesis of relevance (R) is based on the observation of (or evidence about) document (D) contents and a specific query (Q). Estimating P(R|D, Q) could be viewed, then, as accumulating pieces of evidence from the representations of documents and queries, such as additional words or index terms. Accumulating more pieces of evidence should result in more accurate estimates of the probability of relevance, if the evidence is uncorrelated. As we will see, retrieval models often introduce intermediate concepts that make the relationship between observations and hypothesis less direct, but this simple model supports the basic intuition of evidence combination. Similarly, combining the output of ranking algorithms or search systems can be modeled as a combination of classifiers, which has been shown to reduce classification error (Turner and Ghosh, 1999). A search system can be viewed as a classifier for the classes relevant and nonrelevant. For a given document, the search system’s output corresponds to a probability of that document belonging to the relevant class. In this framework, classification errors reduce retrieval effectiveness. Misclassifying a relevant document reduces recall and misclassifying a nonrelevant document reduces precision. The amount of error reduction from combination depends on the correlation of the classifier outputs, with uncorrelated systems achieving the maximum reduction. We will show that this model provides an explanation for many of the phenomena observed in combination experiments (e.g., Vogt and Cottrell, 1998), such as the increased probability of relevance for a document ranked highly by different systems. It also provides a simple prescription of the conditions for optimum combination. Despite the popularity of the combination approach (sometimes called “fusion”), what is known about it is scattered among many papers covering different areas of IR. One of the main goals of this paper is to summarize the research
4
ADVANCES IN INFORMATION RETRIEVAL
in this area. The summary will show how combination has been applied to many aspects of IR systems, and will discuss the successes and limitations of this research. The most obvious limitation is that there is no clear description of how representations, retrieval algorithms, and search systems should be combined for optimum effectiveness. By comparing and analyzing previous research using the terminology of classifier combination and inference nets, we hope to improve this situation. We will also describe how a new approach to probabilistic retrieval based on language models (Ponte and Croft, 1998; Miller et al., 1999; Berger and Lafferty, 1999) provides mechanisms for representing and combining sources of evidence for IR, and that this approach can be integrated with the inference net model to provide an improved framework for combination. Although our focus in this paper is primarily on combination techniques for improving retrieval effectiveness, combination has been applied to a number of related tasks, such as filtering (Hull et al., 1996) and categorization (Lewis and Hayes, 1994; Larkey and Croft, 1996), and has been studied in other fields such as machine learning (Mitchell, 1997). We will refer to work in these areas in a number of sections of the paper. In the following sections, we describe the research that has been done on combination applied to different levels of an IR system. Section 2 describes how different representations of text and documents can be generated and how they have been combined. Section 3 describes research related to combining different representations of information needs (queries). Section 4 describes how ranking algorithms can be combined and the results of that combination. Section 5 describes how the output from different search systems can be combined and the effectiveness of such combinations. In Section 6, we describe some of the retrieval models that have been proposed for combining all the evidence about relevance in a single framework. Finally, in section 7, we describe the language model approach to retrieval and show how it can be used to support combination. In discussions of retrieval effectiveness in this paper, we assume familiarity with the standard recall and precision measures used for evaluations of information retrieval techniques (Van Rijsbergen, 1979). Although specific performance improvements are discussed for some experiments, it is in general difficult to compare the results from multiple studies because of the variations in the baselines and test collections that are used. For example, a combination technique that produces a 20% improvement in average precision in one study may not yield any improvement in another study that uses a more effective search as the baseline. For this reason, we focus on general summaries of the research rather than detailed comparisons.
Combining Approaches to Information Retrieval
2 2.1
5
COMBINING REPRESENTATIONS MANUAL AND AUTOMATIC INDEXING
The use of multiple representations of document content in a single search appears to have started with intermediaries searching bibliographic databases. These databases typically contain the titles, abstracts, and authors of scientific and technical documents, along with other bibliographic information and manually assigned index terms. Index terms are selected from a controlled vocabulary of terms by indexers based on their reading of the abstract. The query languages supported by typical bibliographic search systems allow the searcher to specify a Boolean combination of words, possibly restricted by location, and index terms as the retrieval criterion. An example of this would be (DRUGS in TI) and (DRUG near4 MEMORY) and (SIDE_EFFECTS_DRUG in DE)
This query is designed to find documents about the side effects of drugs related to memory. It specifies that the word “drugs” should be in the title of the document, and that the text of the document should contain the word “drug” within 4 words of the word “memory”, and that the controlled vocabulary term “side_effects_drug” has been used to index the document (i.e. it is present in the descriptor or DE field). Early studies showed the potential effectiveness of this search strategy. For example, Fisher and Elchesen, 1972, showed that searching title words in combination with index terms was better than searching either representation alone. Svenonius, 1986, in a review of research related to controlled vocabulary and text indexing, makes it clear that the two representations have long been thought of, and used, as complementary. A number of major studies, such as the Cranfield tests (Cleverdon, 1967), the SMART experiments (Salton, 1971), and the Cambridge experiments (Sparck Jones, 1974), also used multiple representations of documents but focused on establishing the relative effectiveness of each representation, rather than on the effectiveness of combinations of representations. The Cranfield tests considered 33 different representations and a number of these were combinations of simpler representations. The major classes of representations, however, were considered separately. Specifically, there were representations based on single words from the text of the documents (“free text”), representations based on controlled index terms, and representations based on “concepts”. Some single word representations were combinations of other single word representations, and similarly for index term representations, but there were no representations that were combinations of single words and index terms. The conclusion of the Cranfield study was that single word representations appeared to perform somewhat better than index term and concept representations, but no mention was made of combining them.
6
ADVANCES IN INFORMATION RETRIEVAL
The first large-scale evaluation of combining representations was reported in Katzer et al., 1982. This study was based on a previous study (McGill et al., 1979) that found low overlap in the documents retrieved by different representations for the same queries. Katzer et al considered representations based on free text and controlled vocabularies. They found that different representations retrieve quite different sets of documents in a Boolean search system. There was low overlap between the relevant document sets retrieved by the representations (about 28% on average) and even lower overlap between all the documents in the retrieved sets (about 13% on average). Despite this, there was little difference in retrieval effectiveness for the representations. In addition, documents with a high probability of relevance had the highest overlap in the retrieved sets and each representation retrieved some unique relevant documents. Using the same data, Rajashekar and Croft, 1995, showed that significant effectiveness improvements could be obtained by combining free text and controlled vocabulary indexing in a probabilistic retrieval system. Turtle, 1990, obtained a similar result with a different set of data. Fox, 1983, carried out combination experiments with controlled vocabulary using a retrieval system based on the vector space model and was also able to improve effectiveness. In each of the experiments using search systems based on ranking, the best results were obtained when the controlled vocabulary representation was treated as weaker evidence than the free text representations. The methods of weighting evidence and choosing weights are important aspects of the overall framework for combining evidence. These frameworks are discussed further in section 6. Controlled vocabulary terms are only one of the alternative representations of documents that have been studied in combination experiments. Citations, passages, phrases, names and multimedia objects have all been considered as sources of evidence about relevance. We will describe each of these here. In order to use some of these representations in a retrieval system, extended representations of the information need (i.e. the query) will also be needed. For example, controlled vocabulary terms could be manually included in a query formulation, as shown in the example query above. Other techniques for extending the query with alternate representations are possible, such as relevance feedback (e.g., Salton et al., 1983), and these techniques will be discussed further in section 3. Query extension is not, however, required for all representations. Controlled vocabulary terms can be matched directly with free text queries, for example, and retrieval results can be altered by relationships, such as citations, between documents.
2.2
CITATIONS
Citations have long been recognized as an alternative document representation (Salton, 1968; Small, 1973). A number of studies have established that there
Combining Approaches to Information Retrieval
7
is low overlap in the documents found using citation representations compared to those found using word or index term representations (e.g., Pao and Worthen, 1989). Retrieval experiments that combine citations with other representations have established that significant effectiveness benefits can be obtained (Salton, 1974; Fox et al., 1988; Croft et al., 1989; Turtle, 1990). The best results in these studies, which used relatively small data sets, improved average precision by 510%. It was also consistently found that the evidence for relevance provided by citations was weaker than that provided by the word-based representation. The citation approach was extended to include hypertext links (Frisse and Cousins, 1989; Croft and Turtle, 1989) and is now being used as the basis for some Web search engines (e.g., Google2). Detailed evaluations of representations based on Web “citations” are not available, but qualitatively the techniques appear to scale well to these very large databases.
2.3
PASSAGES
The basic premise behind passage retrieval is that some parts (or passages) of a document may be more relevant to a query than other parts. By representing a document as a collection of passages rather than a monolithic block of text, more accurate retrieval may be possible (O’Connor, 1975; O’Connor, 1980). A number of definitions of document passages are possible (Callan, 1994). Discourse passages are based on textual discourse units such as sentences, paragraphs and sections (e.g., Salton et al., 1993; Wilkinson, 1994). Semantic passages are based on similarities of the subject or content of the text (e.g., Hearst and Plaunt, 1993; Mittendorf and Schauble, 1994). Window passage s are based upon a fixed number of words (Callan, 1994). Almost all of the research related to passages involves retrieval experiments where passage-level representations are combined with global (whole document) representations. For example, Salton et al., 1993, refine rankings based on global similarity using sentence, paragraph and section similarities. Mittendorf and Schauble, 1994, combine a probabilistic model of relevant text passages with a model of text in general. Callan, 1994, combines global document evidence and window-based passage evidence in a probabilistic framework. Recent research by Kaszkiel and Zobel, 1997, confirms Callan’s earlier result that fixed-length, overlapping passages produce the best effectiveness, and that the best window size is between 150-250 words. More than 20% improvements in average precision were obtained in some experiments.
2
http://www.google.com
8
2.4
ADVANCES IN INFORMATION RETRIEVAL
PHRASES AND PROPER NOUNS
Simple noun phrases are an extremely important part of the searcher’s vocabulary. Many of the queries submitted to current Web search engines consist of 2-3 words (about 50% in Jansen et al., 1998, and many of those queries are phrases. Phrase representations of documents were used in the earliest bibliographic systems and were evaluated in early studies (Cleverdon, 1967). Salton and Lesk, 1968, reported retrieval experiments that incorporated statistical phrases based on word co-occurrence into the document representation as additional index terms. Fagan, 1989, also studied statistical phrases but ranked documents by a weighted average of the scores from a word representation and a phrase representation. In both of these studies, the effectiveness improvements obtained were mostly small but varied considerably depending on the document collections being used (from -1.8% to 20% improvement in average precision). Fagan’s experiments with syntactic methods of recognizing phrases were less successful (Fagan, 1987). Croft et al., 1991, and Callan et al., 1995a, introduced a phrase model that explicitly represents phrasal or proximity representations as additional evidence in a probabilistic framework. This approach yielded results that were somewhat more effective than simple statistical phrases, but not consistently. Bartell et al., 1994, also demonstrated improvements from combining word-based and phrase-based searches. Another representation that has been treated as additional index terms are the so-called named entities found using information extraction (MUC-6, 1995). These are special classes of proper nouns mentioned in document text such as people, companies, organizations, and locations. Callan and Croft, 1993, described how these entities could be incorporated into the retrieval process. For the queries used in this study, the impact on effectiveness was not significant.
2.5
MULTIMEDIA
Documents, in general, can be complex, multimedia objects. We have described some of the representations that can be derived from the text and links associated with the document. Other media such as speech, images, and video may also be used in queries and documents, and should be considered alternative representations. A number of people have described how multimedia objects can be retrieved using associated text, such as captions, surrounding text, or linked text (e.g., Croft and Turtle, 1992; Harmandas et al., 1997). This is the primary representation used by image searching engines on the Web. Increasingly, however, image and video search systems are making use of image processing techniques that help to categorize pictures (e.g., Frankel et al., 1996) or compare images directly (Flickner et al., 1995; Ravela and Manmatha, 1997). These image-based techniques typically use very different data structures and algorithms compared to text-based techniques. As a result, combining the ev-
Combining Approaches to Information Retrieval
9
idence about relevance from text and image representations (and potentially other representations) involves combining the rankings from multiple subsystems. This has been a concern of both the IR community (e.g., Croft et al., 1990; Fuhr, 1990; Callan et al., 1995b) and the multimedia database community (e.g., Fagin, 1996), and will be discussed further in sections 4 through 6.
3
COMBINING QUERIES
We have described various representations of documents that could be used as evidence for relevance. Experiments with combinations of these representations show that, in general, using more than one representation improves retrieval effectiveness. They also show that when one source of evidence is weaker (less predictive of relevance) than the others, this must be reflected in the process of accumulating evidence or effectiveness will suffer. These observations are consistent with the simple probabilistic framework mentioned in the first section. Estimating relevance, however, involves more than document representations. Queries, which are representations of the searcher’s information need, are an important part of the process of calculating P(R|D, Q). Each additional piece of evidence that the query contains about the true information need can make a substantial difference to the retrieval effectiveness. This has long been recognized and is the basis of techniques such as relevance feedback, where user judgments of relevance from an initial ranked list are used to modify the initial query (Salton and McGill, 1983), and query expansion, which involves the automatic addition of new terms to the query (Xu and Croft, 1996; Mitra et al., 1998). Relevance feedback and query expansion can also be viewed as techniques for creating alternative representations of the information need. Traditional query formulation tools, such as the thesaurus, can be viewed the same way. Even in the earliest retrieval experiments with a thesaurus (Salton and Lesk, 1968) and automatic query expansion using term clustering (Sparck Jones, 1971), the thesaurus classes or term clusters were treated as alternative representations that were combined with word-based queries. As mentioned in the last section, in order to make use of some alternative document representations, the query must at least be partially described using the same representations. Salton et al., 1983, used relevance feedback to add citations to the initial query, and consequently was able to use citations in the document representations to improve the ranking. Crouch et al., 1990, also used feedback to add controlled vocabulary terms to the query. Xu and Croft, 1996, used an automatic query expansion technique to construct a phrasebased representation of the query that was combined with the initial wordbased representation using weighted averaging. Callan et al., 1995a, describe
10
ADVANCES IN INFORMATION RETRIEVAL
a number of other strategies for automatic construction of alternative query representations. The idea that there are alternative queries only makes sense with the assumption that there is an underlying information need associated with the searcher. A given query is a noisy and incomplete representation of that information need. By constructing multiple queries, we are able to capture more pieces of evidence about relevance. The best source of information about the information need is, of course, the searcher. A number of studies have looked at or observed the effect of capturing multiple queries from a single searcher or from multiple searchers given the same specification of an information need. McGill et al., 1979, carried out a study of factors affecting ranking algorithms, and noticed that there was surprisingly little overlap between the documents retrieved by different search intermediaries (people who are experts in the use of a particular search system) when they were assigned the same information need as a starting point. Saracevic and Kantor, 1988, also found that when different intermediaries constructed Boolean search formulations based on the same descriptions of the information need, there was little overlap in the retrieved sets. In addition, they observed that the odds of a document being judged relevant was proportional to the number of times it was in a retrieved set. Based on these studies, Turtle and Croft, 1991, proposed a retrieval model that explicitly incorporated the notion of multiple representations of the information need. They report the results of experiments that combined wordbased and Boolean queries to improve retrieval effectiveness. Rajashekar and Croft, 1995, extended this work by combining word-based queries with two other queries based on different types of manual indexing. Combining pairs of query representations produced consistent performance improvements, and a weighted combination of all three achieved the best retrieval effectiveness. Belkin et al., 1993, carried out a more systematic study of the effect of query combination in the same probabilistic framework. They verified that retrieval effectiveness could be substantially improved by query combination, but that the effectiveness of the combination depends on the effectiveness of the individual queries. In other words, queries that provided less evidence about relevance had to have lower weights in the combination to improve performance. Bad query representations could, in fact, reduce effectiveness when combined with better representations. In a subsequent, larger study, Belkin et al., 1995, obtained essentially the same results and compared query combination to the strategy of combining the output of different systems (which they called data fusion). The latter strategy is discussed in the next two sections.
Combining Approaches to Information Retrieval
4
11
COMBINING RANKING ALGORITHMS
The next two sections discuss techniques for combining the output of several ranking algorithms. The ranking algorithms can be implemented within the same general framework, such as probabilistic retrieval or the vector space approach, or they can be implemented in different frameworks. The ranking algorithms can also be operating on the same databases, overlapping databases, or totally disjoint databases. In this section, we focus on the combination of the output of ranking algorithms implemented in the same framework and operating on the same data. Croft and Harper, 1979, noted that a cluster-based ranking algorithm retrieved different relevant documents than a word-based probabilistic algorithm, even though their average performance was very similar. They proposed using clustering as an alternate search strategy when word-based ranking failed. Similar observations were made about the performance of other ranking algorithms, particularly in the TREC evaluations (Harman, 1995). The approach of providing alternative ranking algorithms was incorporated into the design of some experimental retrieval systems, most notably I 3R (Croft and Thompson, 1987) and CODER (Fox and France, 1987). Attempts were made to select the best algorithm for a given query using an adaptive network (Croft and Thompson, 1984), and combine the results of multiple ranking algorithms using a plausible inference network (Croft et al., 1989). Turtle and Croft, 1991, showed how a nearest neighbor cluster search could be described and combined with a word-based search in a probabilistic framework. As mentioned in section 1, combining the output of ranking algorithms can be modeled as combining the output of multiple classifiers (Tumer and Ghosh, 1999). A ranking algorithm defines a classifier for each query, where the classes are associated with relevance and non-relevance (Van Rijsbergen, 1979). These classifiers can be trained using relevance feedback, but typically the only information that is available about the relevant class comes from the query. In fact, the approach of combining multiple representations of the information need, discussed in the last section, is more properly viewed as constructing multiple classifiers (one for each query representation) and combining their output. From this point of view, experiments such as Rajashekar and Croft, 1995, and Belkin et al., 1995, can be viewed as validation of the effectiveness of combining multiple classifiers for IR. Combining classifiers has been extensively studied in neural network research and the machine learning area in general. Tumer and Ghosh, 1999, provide a good overview of the literature in this area. They point out that given limited training data and a large, noisy “pattern space” (possible document descriptions), variations in weighting, initialization conditions, and the internal structure of the classifier produce different outputs. This is exactly what IR re-
12
ADVANCES IN INFORMATION RETRIEVAL
searchers have observed, even to the extent that variations in the “tf.idf’ weighting functions3 can retrieve substantially different documents (Lee, 1995). Tumer and Ghosh observe that simply averaging the output of the classifiers is the most common combining strategy, although more complex strategies such as learning appropriate weighted averages have been evaluated. They analyze the averaging strategy and show that for unbiased, independent classifiers, the “added error” above the Bayes error will be reduced by a factor of N for N classifiers. They model the output of a classifier for a given input (a document) as a combination of the probability distribution for each class and a noise distribution (the error). Reducing the error corresponds to reducing the variance of the noise. Since classification errors correspond to not retrieving relevant documents or retrieving non-relevant documents, reducing this error will improve retrieval effectiveness (Van Rijsbergen, 1979). Tumer and Ghosh also mention that the simple combining strategies are best suited for situations where the classifiers all perform the same task (which is the case for IR), and have comparable success. Simple combination strategies can fail when even one of the classifiers being combined has very poor performance or very uneven performance. There is some evidence that simple combination strategies such as summing, averaging or weighted averaging may be adequate for IR. For example, most of the experiments described in sections 2 and 3 used these strategies. Weighted averaging was required in cases where one of the classifiers were based on a poor document description (controlled vocabulary terms). Bartell et al., 1994, describe an approach to learning a weighted, linear combination of classifiers for IR. Fox and Shaw, 1994, conducted an evaluation of combination strategies using different retrieval algorithms in a vector space model. They found the best combination strategy consisted of summing the outputs of the retrieval algorithms, which is equivalent to averaging in terms of the final ranking. Hull et al., 1996, compared simple and complex combinations of classifiers for the document filtering problem, which has substantially more training data than is the case for IR. They found that the best improvement in performance came from the simple averaging strategy. IR classifiers will often not be independent, since they typically use the same document and query representations. Some of the best results in terms of improving retrieval effectiveness have come from combining classifiers based on very different representations, such as the citation experiments described in section 2. Combining classifiers that are very similar, such as those based on minor differences in the “tf.idf” weights, usually does not improve perfor-
3The “tf.idf” weight is a combination of weights derived from within-document term frequency (tf) and the inverse of the number of documents in the database that contain the term (idf). There are many variations of this weight discussed in the literature (Salton and Buckley, 1988).
Combining Approaches to Information Retrieval
13
mance. Lee, 1995, conducted an extensive study of combining retrieval output based on different weighting schemes in the vector space model. He combined these outputs by averaging the normalized scores. His experiments showed that combining classifiers (“retrieval runs” in his paper) based on similar weighting schemes had little impact on performance. Combining classifiers based on substantially different weighting schemes, however, produced significant improvements. Specifically, he found that combining rankings based on cosine normalization of the tf.idf weight with rankings based on other normalization schemes was effective (about 15% improvement in average precision). Hull et al., 1996, found that the gains in performance from combination were limited by the correlation between the classifiers they used. The correlation was caused primarily by using the same training data and was strongest for classifiers that used the same document representations. The discussion of classifier combination in Tumer and Ghosh assumes that the classifiers have comparable output in that they are trying to make the same decision within the same framework. For probabilistic systems, this means they are all attempting to estimate P(R|D, Q) and we can combine these estimates using simple strategies. The lack of knowledge of prior probabilities and the lack of training data, however, make the accurate estimation of these probabilities difficult and can make the outputs of the classifiers less compatible. Combining a cluster-based retrieval algorithm with a word-based algorithm, for example, can be quite difficult because the numbers produced by these algorithms for ranking may have little relationship to the probabilities of relevance. With sufficient training data, the relationship between these numbers and the probabilities can be learned but this data is usually not available. Incompatibility of classifier output also occurs in the vector space model, as discussed in Lee, 1995. In that paper, the scores produced by different retrieval runs were normalized by the maximum scores for each run in order to improve compatibility. This problem is particularly acute for an approach that combines the output of completely different search systems, with little idea of how the numbers output by those systems are calculated. This situation is discussed in the next section.
5
COMBINING SEARCH SYSTEMS
The idea of combining the output of different search systems was introduced during the DARPA TIPSTER project (Harman, 1992) and the associated TREC evaluations (Harman, 1995). These evaluations involve many search systems running the same queries on the same, large databases. The results of these searches are made available for research and a number of studies have been done of combination strategies. Belkin et al., 1995, combined the results of searches from a probabilistic system and a vector space system and showed performance improvements. Lee, 1997, combined the results from six selected retrieval
14
ADVANCES IN INFORMATION RETRIEVAL
systems from a TREC evaluation. He investigated the combination strategies used in Fox and Shaw, 1994, and Lee, 1995. Scores from the different retrieval systems were normalized using the maximum and minimum scores according to the formula: unnormalized_score – min-score normalized_score = max_score – min_score Lee’s results showed that the most effective combinations (up to approximately 30% improvement relative to a single search) were between systems that retrieve similar sets of relevant documents, but different sets of nonrelevant documents. This is related to the observation in Lee, 1995, that retrieval algorithms with low overlap in the retrieved sets will, given similar overall performance, produce the best results in combination. Vogt and Cottrell, 1998, in a study of the factors that predict good combination performance, looked at pairwise combinations of all systems (61 of them) from another TREC evaluation. They were able to verify Lee’s observation that the best combinations were between systems that retrieve similar sets of relevant documents and dissimilar sets of nonrelevant documents. These results can be simply explained in terms of uncorrelated classifiers. The sets of documents that are being compared in these studies are the top 1000 documents retrieved by each system for each query. The number of relevant documents for a given query is typically not large (100-200). We would expect, therefore, that many of the relevant documents are retrieved by most systems. This is in fact shown by Lee’s analysis of the correlation between the relevant retrieved document sets. Since there are large numbers of nonrelevant documents, we would expect that uncorrelated classifiers (search systems) would retrieve different sets of nonrelevant documents and this is what was observed. We would also expect that uncorrelated systems would produce different rankings of the relevant documents, even when the overlap in the sets of retrieved relevant documents is high. Vogt and Cottrell observed this difference in rankings for good combinations. They also observed, as did Lee, that the best combinations occur when both systems being combined have good performance, although it is possible to get improvement when only one of the systems has good performance. All of these observations are consistent with the statement that the combination with the lowest error occurs when the classifiers are independent and accurate. Lee, 1997, presented two other results related to combination strategies for different search systems. The first of these results was that combining the outputs of search systems using the ranks rather than the scores of the documents was, in general, not as effective. The exception to that was when the search systems had very different characteristics in terms of the shape of the score-rank curve. This can be interpreted as evidence that the normalized score is usually a better estimator for the probability of relevance than the rank. Using the ranks
Combining Approaches to Information Retrieval
15
is a more drastic form of smoothing that appears to increase error except when the systems being combined have very different scoring characteristics. The second of Lee’s results was that the best combination strategy was to sum the normalized scores and then multiply by the number of nonzero scores in the combination. This was better than simply summing the scores by a small but consistent margin. This form of combination heavily favors the documents retrieved by more than one system. A zero score for a document means that it was not retrieved in the top 1000 for that system rather than being a true estimate of the probability of relevance. Given that, the combined estimate of the probability for such a document is likely to have a much higher error than the estimates for documents which have only non-zero scores. A combination strategy that favors these documents could be interpreted, then, as favoring estimates with lower error. The experiments mentioned previously combined the outputs of multiple search systems using the same database. The outputs of search systems using overlapping or disjoint databases could also be combined. This type of combination has been called collection fusion, distributed IR, or meta-search. Voorhees et al., 1995, report experiments on techniques for learning weights to associate with each of the systems in the combination. The weights are used with the ranked document sets to determine how the documents will be mixed for the final ranking. This approach has some similarity to the rank combination strategies used by Lee, 1997. Callan et al., 1995b, show that using scores weighted by an estimate of the value of the database for the query is substantially better than interleaving ranks. Both Voorhees and Callan used disjoint databases for their experiments. In many practical environments, such as metasearch on the web, the databases used by the search systems will be overlapping. This will result in a situation similar to that described by Lee, 1997, where documents would have a varying number of scores associated with them. Although there are no thorough evaluations of the combination of web search results, it appears that Lee’s results may apply in this situation. This means that the best combination strategy may be to normalize the scores from each search engine, sum the normalized scores for each document, and multiply the sum by the number of search engines that returned that document (at a given cutoff). If the overlap between the databases used by the search engines is low (i.e. there are substantial differences in the amount of the web indexed), the last step would be less effective. The situation of combining the outputs of multiple search systems also applies to multimedia retrieval (Croft et al., 1990; Fagin, 1996; Fagin, 1998). In this case, we are typically combining the output of a text search and the output of one or more ranking algorithms that compare image features such as color distributions or texture. The experimental results discussed above suggest that the scores from these image and text retrieval algorithms should be
16
ADVANCES IN INFORMATION RETRIEVAL
combined by normalizing and then summing, potentially taking into account the number of non-zero scores. There is, unfortunately, no current evidence that this is a good choice other than the theoretical argument about classifier combination. Fagin, 1996, develops an algorithm for combining scores in a multimedia database using the standard operators of fuzzy logic, namely min and max. Lee’s experiments do provide evidence that these combination operators perform significantly worse than summing. Ciaccia et al., 1998, also discuss ranking in a multimedia database environment. They are concerned primarily with the efficiency of combination, as is Fagin, and present performance results for a range of combination operators.
6
COMBINING BELIEF
The previous sections have described the results of many different experiments with combining evidence to improve retrieval effectiveness. We have described these experiments in terms of combining evidence about relevance in a single classifier and then combining the outputs of multiple classifiers. Figure 1.1 (derived from Tumer and Ghosh, 1999) shows this overall conceptual view of combination. In this view, multiple representations (called feature sets in the classification literature) are constructed from the raw data in the documents. Both retrieval algorithms and search systems are regarded as classifiers that make use of these representations to calculate the probability of relevance of the documents. Some algorithms and systems combine representations in order to reduce the error of that calculation. The output of the retrieval algorithms or search systems can then be combined to further reduce the error and improve retrieval effectiveness. This simple framework can be used to explain some of the basic results obtained with combination experiments, such as the increased probability of relevance for documents retrieved multiple times by alternative representations and search systems, and the low overlap between the outputs of the best combined systems. According to this view of combination, there are only two requirements for minimizing the classification error and obtaining the best retrieval performance. The first of these is that each individual classifier (retrieval algorithm or system) should be as accurate as possible. This means that each classifier should produce probabilities of relevance with low error. The second requirement is that the classifiers that are combined should be uncorrelated. This means that we do not want to combine classifiers that repeatedly produce the same or similar rankings for documents, regardless of whether those rankings are accurate or inaccurate. Classifiers that use different representations and retrieval algorithms are more likely to be independent. A number of other frameworks have been proposed in the IR literature for providing a formal basis for the combination processes described in Figure 1.1.
Combining Approaches to Information Retrieval
Figure1.1
17
Combining strategies for retrieval
Some frameworks address the combination of representations for retrieval algorithms, some the combination of retrieval algorithms, and others the combination of search system output. We will discuss the frameworks using these categories.
6.1
FRAMEWORKS FOR COMBINING REPRESENTATIONS
The vector space model has been used as the basis for a number of combination experiments. In this model, documents and queries are characterized by vectors of weighted terms. Fox et al., 1988, proposed using subvectors to describe different “concept types” or representations derived from documents. An overall similarity between a document and a query, which is used to rank the documents, is computed as a linear combination of the similarities for each subvector. For example, if documents were represented using words, authors, and citations, the similarity function would be : sim(Q, D)
=cword . sim(Qword, Dword) + cauthor . sim(Qauthor, Dauthor) + ccite . sim(Qcite, Dcite)
18
ADVANCES IN INFORMATION RETRIEVAL
where the ci values are coefficients. This type of weighted linear combination is identical to that used in the experiments described in previous sections for combining the output of classifiers or search systems. This shows that there is little difference in this framework between combining representations and combining search output. Fox et al., 1988, used the similarity function to predict relevance and then performed a regression analysis with test collection data to determine the values of the coefficients. Fuhr and Buckley, 1991, also used regression to learn effective combinations of document representations. This work was based on a probabilistic model that estimates P(R|x(t, d)), which is the probability of relevance given a “relevance description” x of the term and document characteristics, instead of P(R|t, d). By using representations based on these characteristics rather than directly on words, more training data is available to estimate the probabilities in the model. They used a least-squared error criterion to compute coefficients of polynomial combinations of the term and document characteristics. Their results showed that a linear combination produced the best overall performance. The characteristics that were used in the relevance description included withindocument frequency of terms, the maximum frequency ofa term in a document, the number of documents in which a term occurs, the number of documents in the collection, the number of terms in a document, and whether a term occurs in the title or the body of a document. Gey, 1994, developed a logistic inference model that used logistic regression, which is generally considered more appropriate for estimating probabilities, to compute the coefficients of a formula for the log of the odds of relevance given the presence of a term. The formula was a linear combination of term characteristics in the documents and the queries. Greiff, 1998, described a probabilistic model developed using exploratory data analysis, which involves looking at large amounts of data about terms, documents and relevance to discover relationships. This approach has some similarity to the regression models described above, but does not make assumptions about the underlying distributions. In Greiff, 1999, he extends his approach to incorporate multiple sources of evidence (representations) based on the Maximum Entropy Principle, which is a way of determining an appropriate probabilistic model given known constraints. The model that results from this approach scores documents using a linear combination of a within-document frequency (tf) component and an inverse document frequency component (idf) for each term that matches the query. Although this is a relatively simple model in terms of the number of representations being combined, the framework he develops is sufficiently general to incorporate any of the representations mentioned previously. Incorporating a new representation involves studying retrieval data involving that representation, and using regression to determine coefficients of a formula that predicts relevance given the evidence provided by the new
Combining Approaches to Information Retrieval
19
representation, conditioned on the evidence provided by the existing representations. This new formula can then be simply added to the linear combination of formulas for other representations. The common characteristic for all frameworks that use training data and regression, of which Greiff’s can be viewed as the most general, is that a retrieval algorithm or search system based on them will produce estimates of probabilities of relevance instead of just normalized similarity values. Other probabilistic systems, such as INQUERY (Callan et al., 1995a), assume that because the goal of the system is to rank documents, the parts of a probabilistic formula that are constant for a given query (such as prior probabilities) can be ignored. In addition, ad-hoc (but effective) formulas are used to calculate parts of the document scores. This means that the numbers produced by these systems are not probabilities. This is a significant disadvantage when it comes to combining the output of a system with the output of other systems. Systems with compatible outputs, and accurate probability estimates, will produce the best combinations assuming they are independent.
6.2
FRAMEWORKS FOR COMBINING RETRIEVAL ALGORITHMS
The inference network framework, developed by Turtle and Croft (Turtle and Croft, 1991; Turtle and Croft, 1992) and implemented as the INQUERY system (Callan et al., 1995a), was explicitly designed for combining multiple representations and retrieval algorithms into an overall estimate of the probability of relevance. This framework uses a Bayesian network (Pearl, 1988) to represent the propositions and dependencies in the probabilistic model (Figure 1.2). The network is divided into two parts: the document network and the query network. The nodes in the document network represent propositions about the observation of documents (D nodes), the contents of documents (T nodes), and representations of the contents (K nodes). Nodes in the query network represent propositions about the representations of queries (K nodes and Q nodes) and satisfaction of the information need (I node). This network model corresponds closely to the overview of combining classifiers in Figure 1.1. The parts of the network that model the raw data in documents, the features extracted from that data, the classifiers that use the features to predict relevance, and the overall combiner for the classifier outputs are labeled in Figure 1.2. In this model, all nodes represent propositions that are binary variables with the values true or false, and the probability of these states for a node is determined by the states of the parent nodes. For node A, the probability that A is true is given by:
20
ADVANCES IN INFORMATION RETRIEVAL
Figure 1.2
Bayesian net model of information retrieval
where Ds is a coefficient associated with a particular subset S of the n parent nodes having the state true, and pi is the probability of parent i having the state true. Some coefficient settings result in very simple but effective combinations of the evidence from parent nodes. For example, if αS = 0 unless all parents have the state true, this corresponds to a Boolean and. In this case, p(A) = 3 i=1n pi. The most commonly used combination formulas in this framework are the average and the weighted average of the parent probabilities. These formulas are the same as those shown in other research to be the best combination strategies for classifiers and discussed earlier in the paper. The combination formula based on the average of the parent probabilities comes from a coefficient setting where the probability of A being true depends only on the number of parent nodes having the state true. The weighted average comes from a setting where the
Combining Approaches to Information Retrieval
21
probability of A depends on the specific parents that are true. Parents with higher weight have more influence on the state of A . The INQUERY search system provides a number of these “canonical” combination formulas as query operators. The three described above are #and, #sum, and #wsum. In the INQUERY system, different document representations are combined by constructing nodes corresponding to propositions about each representation (i.e. is this document represented by a particular term from a representation vocabulary) and constructing queries using those representation nodes. The queries for each representation are combined using operators such as #wsum (Rajashekar and Croft, 1995). For example, there may be nodes corresponding to word-based terms and nodes corresponding to controlled vocabulary terms. These nodes are connected to documents by the probabilities that those terms represent the contents of the documents. Query operators are then used to construct a query based on words (qword) and a query based on controlled vocabulary (qcontrol). These queries may be complex combinations of the evidence in those terms. Each of the queries is, in fact, a classifier that could produce an individual ranking of the documents. The two representations are combined by combining the query nodes. If the searcher wanted to weight the word-based representation twice as much as the controlled vocabulary representation, the final INQUERY query would be #wsum(2.0 qword 1.0 qcontrol). This example shows that the inference net framework can be used for most of the combination processes in Figure 1 . 1. Individual classifiers are built using representation nodes and combination operators, and the output of those classifiers is combined using other operators. It is possible to represent different retrieval algorithms in the inference net framework by using different combinations of representation nodes (i.e. new operators), new types of representation nodes, and different techniques for computing initial probabilities for representation nodes. The probabilities associated with the query node propositions are computed from the probabilities associated with representation nodes. The probabilities associated with representation nodes, however, can be computed from evidence in the raw data of the documents. For example, a tf.idf formula is used in INQUERY to compute the probability of a word-based representation node for a particular document. Turtle and Croft, 1991, describe how retrieval based on document clustering and hypertext links can be incorporated into the inference net framework by changing the probability estimates for nodes representing linked documents. The advantage of the inference net is that it provides a probabilistic framework for rapidly constructing new classifiers based on different representations and retrieval algorithms and combining their output. Not every retrieval algorithm can be represented in this framework, however. Greiff et al., 1997, describes the class of combination operators that can be computed easily and how these can be used to model a well-known vector space ranking algorithm.
22
ADVANCES IN INFORMATION RETRIEVAL
Although this effort was successful in achieving comparable or better effectiveness, it shows the difficulty of modeling even relatively simple retrieval algorithms that are not based on a probabilistic approach. A complex retrieval algorithm based on, for example, a neural net architecture, could not be modeled in an inference net. Another problem with the inference net is that the output of the classifiers do not correspond well to real probabilities. This is due to the heuristic estimation formulas used (such as tf.idf), the lack of knowledge of prior probabilities, and the lack of training data. Haines and Croft, 1993, describe how relevance feedback can be used to modify the query network to produce more effective rankings, although their approach does not improve the correspondence between document scores and probabilities. Haines, 1996, discusses how the structure of the inference net could be changed to better accommodate learning, but his approach was difficult to implement in the INQUERY system. A similar comment applies to the general techniques for learning with Bayesian networks (Heckerman et al., 1994).
6.3
FRAMEWORKS FOR COMBINING SEARCH SYSTEM OUTPUT
The strategies for combining the output of search systems described in Lee, 1997, are implemented in a simple, heuristic framework. The output of the systems are treated as similarity values that are normalized and combined using one of a variety of possible strategies. The specific normalization and combination strategies are selected based solely on empirical evidence, although some of the combination strategies, such as min and max, can be justified with formal arguments. Hull et al., 1996, also experimented with various combination strategies, but in the context of a framework of combining classifier output. They derive a formula for the combination of the output of n classifiers C1, . . . , Cn that are conditionally independent given relevance (R) and non relevance:
This formula is then used as justification for the strategy of summing the logodds numbers derived from the classifier output. There are two problems with this framework. The first is that the conditional independence assumption is not warranted. The second is that P(R|Ci) is not the same as the output of the classifier. The explanation for this is best done in a Bayesian network framework where the Ci are nodes representing the decisions of the document classifiers and R is a node representing the combined decision. The structure of this network is shown in Figure 1.3. Each classifier can be viewed as voting
Combining Approaches to Information Retrieval
23
on the overall decision of the combined network. The output of classifier Ci for a given document is the probability associated with the state Ci = true. For this network,
where C1, . . . , Cn represents a particular configuration of nodes with values true and false, and the summation is over all possible configurations. In the Bayesian network framework, the evidence supporting each classifier’s decision is assumed to be independent and
This is simply a reformulation of the function for node probability given in section 6.2. As mentioned previously, there are a number of possibilities for calculating P(R|C1,. . . , Cn). If the probability of voting for relevance overall depends on the number of classifiers that vote for relevance, the resulting combining function is the average of the classifier probabilities. We could base the overall vote for relevance on the vote of the classifier with the maximum (or minimum) probability of voting for relevance. This would result in the combining function being max (or min). A number of other combining functions are possible, but they have the common characteristic that determining the probability of relevance involves looking at the collective vote of the classifiers. The overall vote and the probability of that vote cannot be determined by looking at each classifier’s vote independently of the others. Thus the assumption of conditional independence is not appropriate. Hull et al., 1996, also report the interesting result that the probability estimates obtained from the combined classifier were not as accurate as the best individual classifier, even though the combined rankings were significantly better. This is related to the problem of combining similarity scores mentioned by Lee, 1997. Referring to Figure 1.3, each of the classifiers is, in effect, voting on relevance. The probabilities associated with a particular classifier may be inaccurate relative to the true probability of relevance, but still be consistent with respect to ranking the “votes” for that classifier. This would produce an effective ranking overall, but inaccurate probabilities. Indeed, if one of the classifiers were producing very accurate probability estimates, the combined estimates would be worse. The best situation is, of course, when all classifiers are producing reasonably accurate estimates. In that case, the combined estimate, as well as the combined vote, should be more accurate (Tumer and Ghosh, 1999). Fagin (Fagin, 1996; Fagin, 1998) proposed a framework based on fuzzy logic for combining the output of multiple search systems in a multimedia
24
ADVANCES IN INFORMATION RETRIEVAL
Figure 1.3
Combining the output of classifiers in a Bayesian net
database system. In this framework, a query is a Boolean combination of “atomic queries”. Each atomic query is of the form object attribute = value and the result of an atomic query is a “graded set” or a list of objects with their scores. Fagin combines the results of atomic queries using the min and max combining functions because they are unique in preserving logical equivalence of queries involving conjunction and disjunction. This property is important for the query optimization strategies he develops, but as we have discussed, min and max do not produce effective retrieval compared to averaging the output of the search systems. Query efficiency will continue to increase in importance, however, as multimedia systems are scaled up to accommodate the enormous volume of data being generated. The assumptions that have been used to build large distributed text retrieval systems may also be different in a multimedia environment, requiring changes in the query processing strategies. For these reasons, it is important to consider efficiency when implementing a combining strategy. The inference network model has also been proposed as a framework for multimedia retrieval (Croft and Turtle, 1992). One of the main problems with incorporating the results of an image retrieval algorithm into a probabilistic combination framework is that the image techniques typically rank images based on distance or similarity scores that are not probabilities. As mentioned in section 5, the scores can be normalized, but this is acompletely ad-hoc procedure that does not produce accurate probability estimates. There has recently been work done on object recognition using probabilistic models (Schneiderman and
Combining Approaches to Information Retrieval
25
Kanade, 1998). They develop a formula for P(object|image), the probability of an object (such as a face) being present in an image. In an image retrieval setting, a query typically includes an image or part of an image and possibly some text. The task of the image retrieval component of the system is to find images that are “similar” to the query image. This task can be based on a probabilistic model, as in Schneiderman and Kanade, 1998. One such probabilistic model would be to compute P(target image|database image). This probability could be calculated using a representation based on visual index terms, similar to the text models described above. Alternatively, a probability of generating the target image could be calculated. This approach corresponds to the use of language models for retrieval, as described in the next section.
7
LANGUAGE MODELS
Most probabilistic retrieval models attempt to describe the relationship between documents and index terms by estimating the probability that an index term is “correct” for that document (Fuhr, 1992). This is a difficult probability to explain, and as a result, heuristic tf.idf weights are used in the retrieval algorithms based on these models. In order to avoid these weights and the awkwardness of modeling the correctness of indexing, Ponte and Croft, 1998, proposed a language modeling approach to information retrieval. The phrase “language model” is used by the speech recognition community to refer to a probability distribution that captures the statistical regularities of the generation of language (Jelinek, 1997). Generally speaking, language models for speech attempt to predict the probability of the next word in an ordered sequence. For the purposes of document retrieval, Ponte and Croft modeled occurrences at the document level without regard to sequential effects, although Ponte, 1998, showed that it is possible to model local predictive effects for features such as phrases. Mittendorf and Schauble, 1994, used a similar approach to construct a generative model for retrieval based on document passages. The approach to retrieval described in Ponte and Croft, 1998, is to infer a language model for each document and to estimate the probability of generating the query according to each of these models. Documents are then ranked according to these probabilities. In this approach, collection statistics such as term frequency, document length and document frequency are integral parts of the language model and do not have to be included in an ad hoc manner. The score for a document in the simple unigram model used in Ponte and Croft is given by:
26
ADVANCES IN INFORMATION RETRIEVAL
where P(Q|D) is the estimate of the probability that a query can be generated for a particular document, and P(w|D) is the probability of generating a word given a particular document (the language model). Much of the power of this simple model comes from the estimation techniques used for these probabilities, which combine both maximum likelihood estimates and background models. This part of the model benefits directly from the extensive research done on estimation of language models in fields such as speech recognition and machine translation (Manning and Schutze, 1999). More sophisticated models that make use of bigram and even trigram probabilities are described in Ponte, 1998, and are currently being investigated (Miller et al., 1999; Song and Croft, 1999). The Ponte and Croft model uses a relatively simple definition of relevance that is based on the probability of generating a query text. This definition does not easily describe some of the more complex phenomena involved with information retrieval. The language model approach can, however, be extended to incorporate more general notions of relevance. Berger and Lafferty, 1999, show how a language modeling approach based on machine translation provides a basis for handling synonymy and polysemy. Hofmann, 1999, describes how mixture models based on latent classes can represent documents and queries. The latent classes can be thought of as language models for important topics in a domain. The language model approach can also be integrated with the inference net model, as described later. For this paper, the important issue is how the language model frameworks for retrieval deal with combination of evidence. In fact, it is ideally suited to the combination approach. The language model framework can readily incorporate new representations, it produces accurate probability estimates, and it can be incorporated into the general Bayesian net framework. Miller et al., 1999, point out that estimating the probability of query generation involves a mixture model that combines a variety of word generation mechanisms. They describe this combination using a Hidden Markov Model with states that represent a unigram language model (P(w|D)), a bigram language model (P(wn|wn–1, D)), and a model of general English (P(w|English)), and mentions other generation processes such as a synonym model and a topic model. Hofmann, 1999, and Berger and Lafferty, 1999 also describe the generation process using mixture models, but with different approaches to representation. Put simply, incorporating a new representation into the language model approach to retrieval involves estimating the language model (probability distribution) for the features of that representation and incorporating that new model into the overall mixture model. The standard technique for calculating the parameters of the mixture model is the EM (Expectation-Maximization) algorithm (McLachlan and Krishnan, 1997). This algorithm, like the regression techniques mentioned earlier, can be applied to training data that is pooled across queries and this, together with techniques for smoothing the maximum likelihood estimates, re-
Combining Approaches to Information Retrieval
27
sults in more accurate probability estimates than a system using tf.idf weights without training, such as INQUERY. Combining representations to produce accurate classifier output is only part of the overall combination process. The need to combine the outputs of multiple classifiers still exists. The other classifiers might be based on alternate language modeling approaches or completely different retrieval models, as we have described in the last section. To accomplish this level of combination, the language modeling approach can be incorporated into the inference network framework described in section 6.2. Figure 1.4 shows the unigram language model approach represented using a simplified part of the network from Figure 1.2. The W nodes that represent the generation of words by the document language model replace the K nodes representing index terms describing the content of a document. The Q node represents the satisfaction of a particular query. In other words, the inference net computes the value of P(Q is true). In the Ponte and Croft model, the query is simply a list of words. In that model, Q is true when the parent nodes representing words present in the query are true and the words not in the query are false. The document language model gives the probabilities of the true and false states for the W nodes. More generally, however, we can regard the query as having an underlying language model, similar to documents. This language model is associated with the information need of the searcher and can be described by P(W1, . . . , Wn|Q). This probability is directly related (by Bayes rule) to the probability P(Q|W1,. . . , Wn) that is computed by the inference network. More complex query formulations, such as those used in the INQUERY system, and relevance feedback provide more information about the searcher’s underlying language model. This information can be directly incorporated into the inference network version of the language model approach by adding more links between the Q node and the W nodes, and changing how the evidence from the W nodes is combined at the Q node. For example, if we learn from relevance feedback that W2 is an important word for describing the user’s language model, we can assign more weight to this word in calculating P(Q|W1, . . . , Wn). The inference network, therefore, provides a mechanism for comparing the document language model to the searcher’s language model. The two language models could also be compared using the Kullback-Leibler divergence or a similar measure (Manning and Schutze, 1999). The advantages of using the inference net mechanism are that it provides a simple method of using the relatively limited information that is known about the searcher’s language model in a typical retrieval environment and it allows the language model approach to be directly combined with other classifiers described in this framework. In sections 6.2 and 6.3, we described how the inference net model could be used to represent the combination of different retrieval algorithms and search systems. This framework can now be extended to include the language model
28
ADVANCES IN INFORMATION RETRIEVAL
Figure 1.4
The language model approach represented in a Bayesian net
approach. The inference net incorporating language models will provide more accurate probability estimates than the inference net based on tf.idf weights. Language models also provide another view of the query formulation process that supports a more direct use of learning than was done in the INQUERY system.
8
CONCLUSION
It is clear from this survey of the experimental results published over the last twenty years that combination is a strategy that works for IR. Combining representations, retrieval algorithms, queries, and search systems produces, most of the time, better effectiveness than a single system. Sometimes the performance improvement is substantial. This approach to IR can be modeled as combining the output of classifiers. Given some assumptions, this model specifies that the best results will be achieved when the classifiers produce good probability estimates and are independent. Even when they are not independent, some improvement can still be expected from combination (Tumer and Ghosh, 1999). This simple prescription for good performance explains many of the results obtained from previous research, including those involving ad-hoc normalization and combination strategies for rankings based on similarity values. The inference net framework was an attempt to provide a general mechanism for combination in a single search system. This framework is, in fact, an instantiation of the combination of classifiers model. Although it is a probabilistic
Combining Approaches to Information Retrieval
29
model, the difficulty of estimating indexing probabilities led to ad-hoc tf.idf weights being used in the INQUERY implementation of the model. This, and the lack of training data, meant that the output of the system was not probabilities. The language model approach to retrieval can more easily be trained to produce accurate probabilities and can be integrated into the inference net framework. Other probabilistic approaches such as logistic regression or maximum entropy models could also be integrated into this framework. Do we need to combine multiple approaches to retrieval and multiple search systems? Combination is, after all, expensive in terms of time, space and implementation effort. The combination of classifiers model implies that, given the extremely high-dimensional, noisy data contained in documents and the general lack of training data, many different classifiers for the retrieval problem could be built that are to some degree independent and would produce different, but equally effective, rankings. This means that combination is both inevitable and beneficial. Given that combination will need to be done, we should try to build search systems that use multiple representations to produce accurate output and we should provide a framework for those systems to be combined effectively. The inference net incorporating language models is a candidate for this framework. Providing such a framework will also be an important part of providing scalability for the immense information stores of the future.
Acknowledgments This material is based on work supported in part by the National Science Foundation under cooperative agreement EEC-9209623. It is also supported in part by the United States Patent and Trademark Office and the Defense Advanced Research Projects Agency/ITO under ARPA order D468, issued by ESC/AXS contract number F19628-95-C-0235, and by SPAWARSYSCEN contract N66001-99- 1-8912. Any opinions, findings and conclusions or recommendations expressed in this material are the authors and do not necessarily reflect those of the sponsors.
References Bartell, B., Cottrell, G., and Belew, R. (1994). Automatic combination of multiple ranked retrieval systems. In Proceedings of the17th ACM SIGIR Conferenceon Research and Development in Information Retrieval, pages 173–181. Belkin, N., Cool, C., Croft, W., and Callan, J. (1993). The effect of multiple query representations on information retrieval system performance. In Proceedings of the 16th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 339–346. Belkin, N., Kantor, P., Fox, E., and Shaw, J. (1995). Combining the evidence ofmultiple query representations forinformation retrieval. Information Processing and Management, 31(3):431–448.
30
ADVANCES IN INFORMATION RETRIEVAL
Berger, A. and Lafferty, J. (1999). Information retrieval as statistical translation. In Proceedings of the 22nd ACM SIGIR Conference on Research and Development in Information Retrieval, pages 222–229. Callan, J. (1994). Passage-level evidence in document retrieval. In Proceedings of the 17th ACM SIGlR Conference on Research and Development in Information Retrieval, pages 302–3 10. Callan, J. and Croft, W. (1993). An evaluation of query processing strategies using the TIPSTER collection. In Proceedings of the 16th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 347– 355. Callan, J., Croft, W., and Broglio, J. (1995a). TREC and TIPSTER experiments with INQUERY. Information Processing and Management, 31(3):327–343. Callan, J., Lu, Z., and Croft, W. (1995b). Searching distributed collections with inference networks. In Proceedings of the 18th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2 1–28. Ciaccia, P., Patella, M., and Zezula, P. (1998). Processing complex similarity queries with distance-based access methods. In Proceedings of the 6th International Conference on Extending Database Technology (EDBT) , pages 9–23. Springer-Verlag. Cleverdon, C. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19:173–192. Croft, W. and Harper, D. (1979). Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 35:285– 295. Croft, W., Krovetz, R., and Turtle, H. (1990). Interactive retrieval of complex documents. Information Processing and Management, 26(5):593–613. Croft, W., Lucia, T.J.and Cringean, J., and Willett, P. (1989). Retrieving documents by plausible inference: An experimental study. Information Processing and Management, 25(6):599–614. Croft, W. and Thompson, R. (1984). The use of adaptive mechanisms for selection of search strategies in document retrieval systems. In Proceedings of the 7th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 95–110. Cambridge University Press. 3 Croft, W. and Thompson, R. (1987). I R: A new approach to the design of document retrieval systems. Journal of the American Society for Information Science, 38(6):389–404. Croft, W. and Turtle, H. (1989): A retrieval model incorporating hypertext links. In Proceedings ofACM Hypertext Conference, pages 213–224. Croft, W. and Turtle, H. (1992). Retrieval of complex objects. In Proceedings of the 3rd International Conference on Extending Database Technology (EDBT), pages 217–229. Springer-Verlag.
Combining Approaches to Information Retrieval
31
Croft, W., Turtle, H., and Lewis, D. (1991). The use of phrases and structured queries in information retrieval. In Proceedings of the 14th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 32–45. Crouch, C., Crouch, D., and Nareddy, K. (1990). The automatic generation of extended queries. In Proceedings of the 13th ACM SlGlR Conference on Research and Development in Information Retrieval, pages 369–383. Deerwester, S., Dumais, S., Furnas, G., Landauer, T., and Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41:391–407. Fagan, J. (1987). Experiments in automatic phrase indexing for document retrieval: A comparison of syntactic and non-syntactic methods. PhD thesis, Computer Science Department, Cornell University. Fagan, J. (1989). The effectiveness of a nonsyntactic approach to automatic phrase indexing for document retrieval. Journal of the American Society for Information Science, 40(2):115–132. Fagin, R. (1996). Combining fuzzy information from multiple systems. In Proceedings of the 15th ACM Conference on Principles of Database Systems (PODS), pages 216–226. Fagin, R. (1998). Fuzzy queries in multimediadatabase systems. In Proceedings of the 17th ACM Conference on Principles of Database Systems (PODS), pages 1–10. Fisher, H. and Elchesen, D. (1972). Effectiveness of combining title words and index terms in machine retrieval searches. Nature, 238: 109–110. Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q.,Dom, B., Gorkani, M., Lee, D., Petkovix, D., Steele, D., and Yanker, P. (1995). Query by image and video content: The QBIC system. IEEE Computer Magazine, 28(9):23– 30. Fox, E. (1983). Extending the Boolean and vector space models of information retrieval with p-norm queries and multiple concept types. PhD thesis, Computer Science Department, Cornell University. Fox, E. and France, R. (1987). Architecture of an expert system for composite document analysis, representation, and retrieval. Journal of Approximate Reasoning, 1:151–175. Fox, E., Nunn, G., and Lee, W. (1988). Coefficients for combining concept classes in a collection. In Proceedings of the 11th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 291–308. Fox, E. and Shaw, J. (1994). Combination of multiple searches. In Proceedings of the 2nd Text Retrieval Conference (TREC-2), pages 243–252. National Institute of Standards and Technology Special Publication 500–215.
32
ADVANCES IN INFORMATION RETRIEVAL
Frankel, C., Swain, M., and Athitsos, V. (1996). WebSeer: An image search engine for the World Wide Web. Technical Report TR-96-14, University of Chicago Computer Science Department. Frisse, M. and Cousins, S. (1989). Information retrieval from hypertext: Update on the dynamic medical handbook project. In Proceedings of ACM Hypertext Conference, pages 199–212. Fuhr, N. (1990). A probabilistic framework for vague queries and imprecise information in databases. In Proceedings of the Very Large Database Conference (VLDB), pages 696–707. Fuhr, N. (1992). Probabilistic models in information retrieval. Computer Journal, 35(3):243–255. Fuhr, N. and Buckley, C. (1991). A probabilistic learning approach for document indexing. ACM Transactions on Information Systems, 9(3):223–248. Gey, F. (1994). Inferring probability of relevance using the method of logistic regression. In Proceedings of the 17th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 222–23 1. Greiff, W. (1998). A theory of term weighting based on exploratory data analysis. In Proceedings of the 21st ACM SIGIR Conference on Research and Development in Information Retrieval, pages 11–19. Greiff, W. (1999). Maximum entropy, weight of evidence, and information retrieval. PhD thesis, Computer Science Department, University of Massachusetts. Greiff, W., Croft, W., and Turtle, H. (1997). Computationally tractable probabilistic modeling of Boolean operators. In Proceedings of the 20th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 119–128. Haines, D. (1996). Adaptive query modification in a probabilistic information retrieval model. PhD thesis, Computer Science Department, University of Massachusetts. Haines, D. and Croft, W. (1993). Relevance feedback and inference networks. In Proceedings of the 16th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2–11 . Harman, D. (1992). The DARPA TIPSTER project. ACM SIGIR Forum, 26(2):26– 28. Harman, D. (1995). Overview of the second text retrieval conference (TREC-2). Information Processing and Management, 3 1(3):27 1–289. Harmandas, V., Sanderson, M., and Dunlop, M. (1997). Image retrieval by hypertext links. In Proceedings of the 20th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 296–303. Hearst, M. and Plaunt, C. (1993). Subtopic structuring for full-length document access. In Proceedings of the 16th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 59–68.
Combining Approaches to Information Retrieval
33
Heckerman, D., Geiger, D., and Chickering, D. (1994). Learning Bayesian networks: The combination of knowledge and statistical data. In Proceedings of the 10th Conference on Uncertainty in Artificial Intelligence, pages 293–301. Morgan Kaufmann. Hofmann, T. (1999). Probabilistic latent semantic indexing. In Proceedings of the 22nd ACM SIGIR Conference on Research and Development in Information Retrieval, pages 50–57. Hull, D., Pedersen, J., and Schutze, H. (1996). Method combination for document filtering. In Proceedings of the 19th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 279–287. Jansen, B., Spink, A., and Saracevic, T. (1998). Real life information retrieval: A study of user queries on the Web. SIGIR Forum, 32( 1):5–17 Jelinek, F (1997). Statistical methods for speech recognition. MIT Press, Cambridge. Kaszkiel, M. and Zobel, J. (1997). Passage retrieval revisited. In Proceedings of the 20th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 178–185. Katzer, J., McGill, M., Tessier, J., Frakes, W., and DasGupta, P. (1982). A study of the overlap among document representations. Information Technology: Research and Development, 1(4):261–274. Larkey, L. and Croft, W. (1996). Combining classifiers in text categorization. In Proceedings of the 19th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 289–297. Lee, J. (1995). Combining multiple evidence from different properties of weighting schemes. In Proceedings of the 18th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 180–188. Lee, J. (1997). Analyses of multiple evidence combination. In Proceedings of the 20th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 267–276. Lewis, D. and Hayes, P. (1994). Special issue on text categorization. ACM Transactions on Information Systems, 12(3). Manning, C. and Schutze, H. (1999). Foundations of statistical natural language processing. MIT Press, Cambridge. McGill, M., Koll, M., and Noreault, T. (1979). An evaluation of factors affecting document ranking by information retrieval systems. Final report for grant NSF-IST-78-10454 to the National Science Foundation, Syracuse University. McLachlan, G. and Krishnan, T. (1997). The EM algorithm and extensions. Wiley, New York. Miller, D., Leek, T., and Schwartz, R. (1999). A Hidden Markov Model information retrieval system. In Proceedings of the 22nd ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2 14–221. Mitchell, T. (1997). Machine Learning. McGraw-Hill, New York. a
34
ADVANCES IN INFORMATION RETRIEVAL
Mitra, M., Singhal, A., and Buckley, C. (1998). Improving automatic query expansion. In Proceedings of the 21st ACM SIGIR Conference on Research and Development in Information Retrieval, pages 206–2 14. Mittendorf, E. and Schauble, P. (1994). Document and passage retrieval based on Hidden Markov Models. In Proceedings of the 17th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 318-327. MUC-6 ( 1995). Proceedings of the Sixth Message Understanding Conference (MUC-6). Morgan Kaufmann, San Mateo. O’Connor, J. (1975). Retrieval of answer-sentences and answer figures from papers by text searching. Information Processing and Management, l1(5/7):155– 164. O’Connor, J. (1980). Answer-passage retrieval by text searching. Journal of the American Society for Information Science, 3 1(4):227–239. Pao, M. and Worthen, D. (1989). Retrieval effectiveness by semantic and citation searching. Journal of the American Society for Information Science, 40(4):226–235. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann, San Mateo. Ponte, J. (1998). A Language Modeling Approach to Information Retrieval. PhD thesis, Computer Science Department, University of Massachusetts. Ponte, J. and Croft, W. (1998). A language modeling approach to information retrieval. In Proceedings of the 21st ACM SIGIR Conference on Research and Development in Information Retrieval, pages 275–28 1. Rajashekar, T. and Croft, W. (1995). Combining automatic and manual index representations in probabilistic retrieval. Journal of the American Society for Information Science, 46(4):272–283. Ravela, C. and Manmatha, R. (1997). Image retrieval by appearance. In Proceedings of the 20th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 278–285. Robertson, S. (1977). The probability ranking principle in information retrieval. Journal of Documentation, 33:294–304. Robertson, S. and Sparck Jones, K. (1976). Relevance weighting of search terms. Journal of the American Society for Information Science, 27:129– 146. Salton, G. (1968). Automatic information organization and retrieval. McGrawHill, New York. Salton, G. (1971). The SMART retrieval system - Experiments in automatic document processing. Prentice-Hall, Englewood Cliffs. Salton, G. (1974). Automatic indexing using bibliographic citations. Journal of Documentation,27:98–100. Salton, G., Allan, J., and Buckley, C. (1993). Approaches to passage retrieval in full text information systems. In Proceedings of the 17th ACM SIGIR
Combining Approaches to Information Retrieval
35
Conference on Research and Development in Information Retrieval, pages 49–56. Salton, G. and Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing and Management, 245 13–523. Salton, G., Fox, E., and Voorhees, E. (1983). Advanced feedback methods in information retrieval. Journal of the American Society for Information Science,36(3):200–210. Salton, G. and Lesk, M. (1968). Computer evaluation of indexing and text processing. Journal of the ACM, 15:8–36. Salton, G. and McGill, M. (1 983). Introduction to modern information retrieval. McGraw-Hill, New York. Salton, G., Wong, A., and Yang, C. (1975). A vector space model for automatic indexing. Communications of the ACM, 18:613–620. Saracevic, T. and Kantor, P. (1988). A study of information seeking and retrieving. Part 111. Searchers, searches, overlap. Journal of the American Society for Information Science, 39(3): 197–216. Schneiderman, H. and Kanade, T. (1998). Probabilistic modeling of local appearance and spatial relationships for object recognition. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 45–51. Small, H. (1973). Co-citation in scientific literature: A new measure of the relationship between two documents. Journal of the American Society for Information Science, 24:265–269. Song, F. and Croft, W. (1999). A general language model for information retrieval. In Proceedings of the Conference on Information and Knowledge Management (CIKM), pages 316–321. Sparck Jones, K. (197 1). Automatic keyword classification for information retrieval. Butterworths, London. Sparck Jones, K. (1974). Automatic indexing. Journal of Documentation, 30(4):393– 432. Svenonius, E. (1986). Unanswered questions in the design of controlled vocabularies. Journal of the American Society for Information Science, 37(5):331– 340. Tumer, K. and Ghosh, J. (1999). Linear and order statistics combiners for pattern classification. In Sharkey, A., editor, Combining Artificial Neural Networks, pages 127–162. Springer-Verlag. Turtle, H. (1990). Inference networks for document retrieval. PhD thesis, Computer Science Department, University of Massachusetts. Turtle, H. and Croft, W. (1991). Evaluation of an inference network-based retrieval model. ACM Transactions on Information Systems, 9(3): 187–222. Turtle, H. and Croft, W. (1992). A comparison of text retrieval models. Computer Journal, 35(3):279–290.
36
ADVANCES IN INFORMATION RETRIEVAL
Van Rijsbergen, C. (1979). Information Retrieval. Butterworths, London. Van Rijsbergen, C. (1986). A non-classical logic for information retrieval. Computer Journal, 29:481–485. Vogt, C. and Cottrell, G. (1998). Predicting the performance of linearly combined IR systems. In Proceedings of the 21st ACM SIGIR Conference on Research and Development in Information Retrieval, pages 190–1 96. Voorhees, E., Gupta, N., and Johnson-Laird, B. (1995). Learning collection fusion strategies. In Proceedings of the 18th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 172–179. Wilkinson, R. (1994). Effective retrieval of structured documents. In Proceedings of the 17th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 311–3 17. Xu, J. and Croft, W. (1996). Query expansion using local and global document analysis. In Proceedings of the 19th ACM SIGIR Conference on Research and Development in Information Retrieval, pages 4–11.
Chapter 2 THE USE OF EXPLORATORY DATA ANALYSIS IN INFORMATION RETRIEVAL RESEARCH Warren R. Greiff The MITRE Corporation Bedford, Massachusetts greiff
@mitre.org
Abstract
1
We report on a line of work in which techniques of Exploratory Data Analysis (EDA) have been used as a vehicle for better understanding of the issues confronting the researcher in information retrieval (IR). EDA is used for visualizing and studying data for the purpose of uncovering statistical regularities that might not be apparent otherwise. The analysis is carried out in terms of the formal notion of Weight of Evidence (WOE). As a result of this analysis, a novel theory in support of the use of inverse document frequency (idf) for document ranking is presented, and experimental evidence is given in favor of a modification of the classical idf formula motivated by the analysis. This approach is then extended to other sources of evidence commonly used for ranking in information retrieval systems.
INTRODUCTION
Information retrieval (IR) systems work because, for most information needs, they are able to order documents of a collection in such a way that documents relevant to the user’s query appear, on average, much higher in the ranking than could be expected by chance. They work because they exploit statistical regularities of document collections and user queries. In response to a query, the earliest systems returned the documents that contained the largest number of query terms. This was effective because a document that contains a query term is more likely to be relevant to the user. A document that contains many of the query terms is more likely to be relevant than a document that contains fewer of them. More modem systems also take into consideration the rarity of a query term and its frequency of occurrence in a document. All other things being equal, a document that contains rare query
38
ADVANCES IN INFORMATION RETRIEVAL
terms is more likely to be relevant than a document containing more common terms. In the same way, a document containing more occurrences of a query term is more likely to be relevant than a document with fewer occurrences of the same term. Out of four decades of research have evolved IR systems that take advantage of these statistical characteristics. While modem retrieval systems have developed sophisticated means for exploiting statistical regularities, IR research has not, as a rule, tended to focus directly on the study of these regularities. Although research in IR is varied and a diversity of philosophies, strategies and techniques have been employed, two major trends may be discerned. For the purposes of this discussion, we may refer to these as the engineering and the a priori modeling approaches to IR research and system development. In the engineering approach, intuitions concerning the nature of document collections and the behavior of users posing queries to an IR system are encoded in a (typically parameterized) retrieval algorithm. Experiments are run comparing a system incorporating this algorithm to a benchmark system; the result of varying parameter settings is studied; alternate versions of the algorithm are tried. If the research is successful, robust improvement to previous retrieval practice is realized and a better understanding of what is needed for effective retrieval performance results. Even in the absence of improved retrieval performance, new insight is often gained into the nature of the IR problem. Much of the progress of information retrieval is due to research of this nature. The a priori modeling approach adopts a more theoretical, formal line of attack. In this case, the researcher attempts to formalize her intuitions in the form of a priori assumptions, and a theory, typically a probabilistic theory, of information retrieval is developed. From this theory an information retrieval strategy is usually derived. This strategy can then be implemented and tested. The proponents of this approach believe that the field of information retrieval is well served by the development of formal theoretical foundations. Formal theories promote precision in discourse and permit the application of deductive reasoning to the analysis of information retrieval questions. In Cooper’s words, probabilistic theories “bring to bear . . . a high degree of theoretical coherence and deductive power” (Cooper, 1994, p. 242). Cooper also emphasizes that a formal approach assists investigators to articulate their important underlying assumptions. “When the underlying theoretical postulates are known and clearly stated,” he tell us, “their plausibility can be subjected to analysis” [p. 2441. In contrast to these, the research described in this paper may be considered a data driven approach. The goal of this research is the development of a model of IR document ranking based on observed statistical regularities of retrieval situations. It is similar to much work in probabilistic information retrieval in that the objective is to formally model the probability of a document being judged relevant to a query conditioned on the available evidence. It is significantly
The Use of Exploratory Data Analysis in Information Retrieval Research
39
different from both the engineering and a priori modeling approaches in the emphasis that is placed on the study of existing retrieval data. The methodology involves the adaptation of techniques of exploratory data analysis to the specific conditions encountered with regard to document ranking in information retrieval. The data driven methodology centers on the analysis of the weights of evidence in favor of relevance given by various sources of evidence. The end product is a model for weight of evidence from which a ranking formula can be derived directly. We will discuss how the data driven methodology has been applied to the analysis of existing retrieval data and a probabilistic model has been produced. The result of this analysis is a model of the weight of evidence in favor of relevance for the totality of evidence associated with a query/document pair being evaluated. The remainder of this paper is organized as follows. Section 2 and Section 3 describe the exploratory data analysis approach, and the concept of weight of evidence, both of which are central to this research. Section 4 presents an initial stage of the research in which attention was focused on the relationship between the document frequency for a term and the weight of evidence associated with its occurrence in a document. The result of the analysis is an hypothesis regarding how the classical inverse document frequency formula may be modified to improve retrieval performance. Substantial experimental evidence is given to support the hypothesis. In Section 5, the methodology used for the study of inverse document frequency in Section 4 is extended to deal with multiple sources of evidence. Retrieval data is analyzed and a stochastic model is developed. From the model, a ranking formula is derived. Experimental results are given for tests of the ranking formula which indicate the viability of the overall approach to IR research and system development.
2
EXPLORATORY DATA ANALYSIS
Hartwig and Dearing define EDA as “a state of mind, a way of thinking about data analysis – and also a way of doing it” (Hartwig and Dearing, 1979, p. 9). They advance adherence to two principles. First, that one should be skeptical of data summaries which may disguise the most enlightening characteristics of the phenomenon being investigated. Second, that one must be open to unanticipated patterns in the data, because uncovering such patterns can be, and often is, the most eventful outcome of the analysis. The article on EDA in the International Encyclopedia of Statistics says that it is the “manipulation, summarization, and display of data to make them more comprehensible to human minds, thus uncovering underlying structure in the data and detecting important departures from that structure” (Andrews, 1978). It goes on to point out that “these goals have always been central to statistics
40
ADVANCES IN INFORMATION RETRIEVAL
and indeed all scientific inquiry”, but that there has been a renaissance in the latter part of this century. This is due in no small part to the accessibility of powerful electronic computers for the accumulation of voluminous quantities of data, high-speed calculation and efficient graphical display. EDA embodies a set of useful methods and strategies, fomented primarily by John W. Tukey (Tukey, 1977). Four distinguishing aspects of this practice, each of which plays an important role in the probability modeling discussed in this paper, are graphical displays, smoothing, the study of residuals, and the transformation of variables. The emphasis in exploratory data analysis is on making the most of graphical displays of the data, a historical review of which is given in Beniger and Brown, 1978. The human mind is far better at uncovering patterns in visual input than in lists or tables of numbers. Depending solely on the reduction of large quantities of data to a few summary statistics erases most of the message the data have for us. Smoothing and non-parametric regression techniques are used with the objective of identifying the component of the data considered to be the signal from that which, for the purposes of the analysis at hand, are to be treated as noise. A residual is the difference between the observed value of a response variable and the value predicted by a given model. By studying graphs of residuals against potential predictor variables, possibilities for extending the model can be explored. Finally, supported by the production of graphical displays, features of the data can be transformed in a variety of ways in order to make underlying regularities in the data more evident.
3
WEIGHT OF EVIDENCE
Formal definition of Weight of Evidence I. J. Good formally defines the weight in favor of a hypothesis, h, provided by evidence, e, as (Good, 1983b; Good, 1950): (2.1) where (2.2) is the prior odds of the hypothesis, h being true, and (2.3) is the posterior odds of the hypothesis h being true conditioned on the evidence e having been observed. He believes this is a concept “almost as important as that of probability itself’ (Good, 1950, p. 249).
The Use of Exploratory Data Analysis in Information Retrieval Research
41
In (Good, 1983a, chap. 4), Good points out that, in various guises, the notion of weight of evidence had appeared in the work of others. As early as 1878, the quantity given in eq. 2.1 appears in the work of the philosopher Charles Sanders Pierce. Good credits Pierce with the original use of the term weight of evidence. More recently, Minsky and Selfridge also refer to this quantity and call it weight of evidence as well (Minsky and Selfridge, 1961). Turing had O(h|e) ____ , as the factor in favor of the hypothesis h provided labeled the quantity, O(h) by the evidence e, and Harold Jeffreys made much use of the concept, referring to it as support (Jeffreys, 1961). Weight of evidence is related to Keynes’s concept of the amount of information, which he defined as the log of p(e h)/p(e)p(h) (Good, 1983a, chap. 11). This is more commonly referred to today as mutual information and is discussed in Section 4.3 in connection with the relationship between weight of evidence and inverse document frequency. Keynes also used the term weight of evidence, but in a different sense from that used by Pierce, Minsky & Selfridge, and Good (Good, 1983a, chap. 15). A generalization of the concept of weight of evidence as it is defined in eq. 2.1, will also play an important role in what follows. Weight of evidence can be conditional. That is, attention may be restricted to some sub-space of the full event space. The notation that will be used for conditional weight of evidence will be:
^
(2.4) Desiderata for a Concept of Weight of Evidence Good elucidates three simple, natural desiderata for the formalization of the notion of weight of evidence (Good, 1989; Good, 1983b). 1. Weight of evidence is some function of the likelihoods: woe (h : e) = f[p(e | h),p(e | h)]
(2.5)
For example, let us suppose that a document is evaluated with respect to the query, Clinton impeachment proceedings and we discover the term Hyde in the document: h e
= =
document is relevant to the query Hyde occurs in the document
This first criterion states that the weight of evidence provided by finding Hyde in the document should be some function of the likelihood of finding Hyde in a document that is relevant to the query and the (for this example,
42
ADVANCES IN INFORMATION RETRIEVAL
presumably lower) likelihood of finding Hyde in a document that is not relevant to the query. 2. The final (posterior) probability is a function of the initial (prior) probability and the weight of evidence: (2.6)
p(h | e) = g[p(h),woe(h : e)]
In the context of the same example, this states that the probability associated with the document being about the impeachment proceedings after having observed that the document contains the term, Hyde, should be a function of 1) the probability associated with the document being relevant to the query prior to obtaining knowledge as to the terms it contains, and 2) the weight of evidence associated with finding Hyde. 3. Weight of evidence is additive: woe(h : e1
^ e ) = woe(h : e ) + woe(h : e |e ) 2
1
2
1
(2.7)
This property states that the weight in favor of a hypothesis provided by two sources of evidence taken together is equal to the weight provided by the first piece of evidence, plus the weight provided by the second piece of evidence, conditioned on our having previously observed the first. The weight of the second piece of evidence is conditioned on the first in the sense that, woe(h : e2) is calculated on the subspace corresponding to the event, e1. If, for example, the two pieces of evidence are: e1 = Hyde occurs in the document e2 = Henry occurs in the document then the weight of evidence provided by finding both of the terms, Hyde and Henry, in the document is the sum of the evidence given by finding Hyde and the evidence given by finding Henry. The weight of evidence provided by occurrence of the term, Henry, is conditioned on having previously taken into consideration the evidence provided by encountering Hyde in the document. Starting from these desiderata, Good is able to show that, up to a constant factor, weight of evidence must take the form given in eq. 2.1: O(h|e) woe(h : e) = log _______ O(h)
(2.8)
The constant factor may be absorbed in the base of the logarithm. For the purposes of this discussion all logarithms will be understood to be in base 10, which will make the scales shown on graphs easier to interpret.
The Use of Exploratory Data Analysis in Information Retrieval Research
43
Weight of Evidence and Information Retrieval There is nothing new about using either log-odds or weight of evidence in information retrieval. The term weights that result from the Robertson/Sparck Jones Binary Independence Model (Robertson and Sparck Jones, 1977) is motivated by the desire to determine the log-odds of relevance conditioned on the term occurrence pattern of a document. The weight can be viewed as the difference between the weights of evidence in favor of relevance provided by the occurrence and non-occurrence of the term. Also, the focus of statistical inference based on logistic regression is the probability of the event of interest transformed by the logit function; that is, the log-odds.
4
ANALYSIS OF THE RELATIONSHIP BETWEEN DOCUMENT FREQUENCY AND THE WEIGHT OF EVIDENCE OF TERM OCCURRENCE
This section reports on a study of the relationship between the weight of evidence in favor of relevance provided by the occurrence of a query term in a document and the frequency with which the term is found to occur in the collection as a whole. From the empirical study of retrieval data, statistical regularities are observed. Based on the patterns uncovered, a theory of term weighting is developed. The theory proposes mutual information between term occurrence and relevance as a natural and useful measure of query term quality. We conclude that this measure is correlated with document frequency and use this to derive a theoretical explanation in support of idf weighting which is different from theories that have previously been proposed. The theory developed, in conjunction with the empirical evidence, predicts that a modification of the idf formula should produce improved performance. In Section 4.4, experiments are presented that corroborate this prediction.
4.1
DATA PREPARATION
The study presented here involved data from queries 051-100 from the first Text REtrieval Conference (TREC) and the Associated Press (AP) documents from TREC volume 1 (Harman, 1993). Each data point corresponds to one query term. The query terms were taken from the concepts field of the TREC 1 topics. For the purposes of uncovering underlying statistical regularities, a set of quality query terms was desired that would keep to a minimum the noise in the data to be analyzed. For this reason the concepts field was used. Initially, the plan was for all query terms to be plotted. Two problems immediately presented themselves. First, rare terms are likely to have zero counts and this is problematic. For variables that are functions of log odds, a zero count translates to a (positive or negative) infinite value. One way around the problem is to add a small value to each of the counts of interest (relevant
44
ADVANCES IN INFORMATION RETRIEVAL
documents in which term occurs, relevant documents in which term does not occur, non-relevant documents in which term occurs, non-relevant documents in which term does not occur). This is a common approach, taken for instance in Robertson and Sparck Jones, 1977, where, for the purpose of estimating wrsj, 0.5 is added to each count. For the purposes of data analysis, however, there is a problem with this approximation. The choice of constant is to a large degree arbitrary. For many of the plots of interest, the shape of the plot at the low frequency end will vary considerably with the value chosen for the constant. Two slightly different choices for the constant value can give a very different overall picture of the data when they are plotted, particularly at the low frequency end. Since our objective is precisely to infer the true shape of the data, this approach is inadequate to our needs. A second problem, is that the variance of the variables we are interested in is large, relative to the effects we hope to uncover. This can be seen clearly, for example, at the left of Figure 2.4 where p(occ | rel) is plotted against log O(occ) for all terms for which it has a finite value. That this variable increases with increasing df is somewhat evident, but subtler details of the relationship are obscured. In order to confront both of these problems, data points were binned. Query terms were sorted in order of document frequency. Then for some bin size, k, sequences of k query terms1 were grouped together in bins. Each bin was then converted into a single pseudo-term by averaging all counts (number of relevant documents in which term occurs, number of relevant documents in which term does not occur, number of non-relevant documents in which term occurs, number of non-relevant documents in which term does not occur). Calculations of probabilities, weights, etc. were done on the pseudo-terms and these results were then plotted. A bin size of k = 20 was found to be best for our purposes. The plot of binned pseudo-terms corresponding to the left of Figure 2.4 is shown at the right of the same figure, Although we will focus on the binned plots, each of these plots will be displayed alongside its unbinned version, in order that the reader may get a feel for the raw data. It should be kept in mind, however, that points with zero counts are not represented in the unbinned versions. Kernel regression methods (Devroye, 1987; Härdle, 1990; Silverman, 1986) were also explored as an alternative to binning (Greiff, 1999).
4.2
PLOTTING THE DATA
The data analysis begins by focusing on the components, p(occ | rel) and p(occ | rel), and how these components correlate with document frequency.
More precisely, k or k + 1 query terms, so that no bin was much smaller than the rest.
1
The Use of Exploratory Data Analysis in Information Retrieval Research
Figure 2.2
45
p(occ|rel) as function of p(occ)
Because the goal is to compare various document sets of differing sizes, we prefer not to plot the data in terms of absolute document frequencies. Instead, we plot against p(occ) = the probability that the term will occur in a document chosen randomly from the collection. __ Occurrence in Non-relevant Documents With respect to p(occ|rel), we see in Figure 2.1 that it is __ well approximated by p(occ). We will return to analyze the variable, p(occ|rel), in more detail.
46
ADVANCES IN INFORMATION RETRIEVAL
Figure 2.3
Histograms for p(occ) and log O(occ)
Occurrence in Relevant Documents More interesting is Figure 2.2, which shows a plot of p(occ | rel) as a function of p(occ). We see from this scatter plot that as the document frequency (equivalently, probability of term occurrence) gets small (left end of graph), the probability of the term occurring in a relevant document tends to get small (lower on the graph) as well. A glance at this graph suggests that a re-expression of variables may be indicated. The histogram shown at the left in Figure 2.3 confirms that, as both intuition and Figure 2.2 suggest, the distribution of document frequencies is highly skewed. With this type of skew, a logarithmic2 transformation is often found to be beneficial (Tukey, 1977). Here, we go one step further and reexpress the variable as: log O(occ) = log
p(occ) 1 – p(occ)
(2.9)
For practical purposes, given typical document frequencies for query terms, the difference between log p(occ) and log O(occ) is negligible. For the development of a general theory, log O(occ) tends to be a preferable scale on which to As mentioned earlier, in order to aid intuitive comprehension, all logarithms in this document are logarithms to the base 10.
2
The Use of Exploratory Data Analysis in Information Retrieval Research
Figure 2.6 log
p(occ | rel) _________ p(occ)
47
as function of log O(occ)
work, due to the symmetric treatment it gives to probabilities below and above 0.5, as discussed in Section 3. The histogram at the right in Figure 2.3 shows the distribution of the variable after it has been re-expressed as log O(occ). Of course, our interest in log p(occ) or log O(occ) is further motivated by the knowledge that this statistic is, in fact, known to be a useful indicator of term value. The variable, p(occ | rel), is re-plotted as a function of log O(occ) in Figure 2.4. The plot against the log-odds shows that the decrease in p(occ | rel) continues as document frequency get smaller and smaller, a fact that was obscured by the bunching together of points below p(occ) ≈ 0.01 in the original plot (Figure 2.2). p(occ|rel) Relative to p(occ) Despite the transformation of the independent variable, looking at p(occ | rel) directly makes it hard to appreciate the phenomenon of interest. The conditional probability of occurrence is higher for high frequency terms. But, high frequency terms are more likely to appear in documents, in general. It comes as no great surprise, then, that they are more likely to occur in relevant documents. This is particularly obvious for very high frequency terms as compared to very low frequency terms. Take for example, two terms: t1 with probability of occurrence, p(t1) = 0.2, and t2 with probability of occurrence, p(t2) = 0.0001. We would expect that p(t1 | rel) is at least 0.2. In contrast, we could hardly expect a term which only appears in one of every ten thousand documents in the collection to appear in as many as two out of ten of the relevant documents. In fact, if the probability of relevance for the query is, say, one in a thousand, simple algebra shows that
48
ADVANCES IN INFORMATION RETRIEVAL
Figure 2.5
p(occ | rel) ___________ p(occ)
as function of log O(occ)
it will not be possible for p(t2 | rel) to be greater than 0.1:
, What may be of more interest to us, then, is how much more likely it is for a term to occur in the relevant documents compared to its being found in an arbitrary document of the collection as a whole. Figure 2.5 shows a plot of the ratio
as a function of log O(occ). We observe here a clear non-linear increase in this ratio as document frequency decreases. From this plot it is evident that, in general: 1) query terms are more likely to appear in relevant documents than in the collection as a whole, and 2) how much more likely their appearance in relevant documents is correlates inversely with document frequency. The apparent exponential nature of this correlation calls out for the logarithm of p(occ | rel) to _______ be investigated. p(occ) Log of the Ratio of p(occ | rel) to p(occ) In Figure 2.6 the log of the p(occ | rel) ratio _______ is plotted against the logarithm of the odds of occurrence in the p(occ) collection. In the plot, we observe:
•
a roughly linear overall increase in p(occ | rel) log _________ with decreasing log O(occ); p(occ)
The Use of Exploratory Data Analysis in Information Retrieval Research
49
p(occ | rel) _ as function of log O(occ) Figure 2.6 log ________ p(occ)
•
a stronger linear relationship apparent in the midrange of document frequencies;
•
an apparent flattening of this growth at both high and low frequencies.
A number of comments are in order. First, a clear pattern has emerged that is difficult to attribute to chance. Furthermore, the “reality” of this regularity is corroborated by our inspection of data from other collections included in TREC volumes 1 and 2. Second, the apparent flattening of the curve at the two extremes is supported by theoretical considerations. At the low-frequency end, we note that: (2.10) If we assume that, for a given query, the probability of relevance across the p(occ | rel) must entire collection is approximately one in a thousand3, then log _______ p(occ) be below 3.0. We can conclude that, on average, the growth of the log ratio observed between log O(occ) = -1.0and log O(occ) = –3.0 cannot be sustained for very small document frequencies. It is reasonable to assume that this p(occ | rel) growth should begin to taper off as log _______ approaches – log p(rel). p(occ) The argument is similar at the high frequency end. We can safely assume that, on average, a query term, even a very high frequency query term, is more likely to appear in a relevant document than it is to appear in an arbitrary document of For the AP collection, the average probability of relevance over the 50 queries is 0.00085.
3
50
ADVANCES IN INFORMATION RETRIEVAL
the collection. Hence, the ratio,
p(occ | rel) _______
is greater than 1, and its logarithm p(occ | rel) _______ greater than 0. Since we conclude that log can be expected to be p(occ) positive at all document frequencies, its rate of descent must taper off at same point before reaching 0. Presumably it approaches 0 asymptotically as the log odds of occurrence goes to ∞ (i.e. term occurs in all documents). It is reasonable to entertain the hypothesis that this leveling off is what we are observing with the rightmost points in Figure 2.6 (and have observed in plots for other collections as well). We must be cautious, however. The leveling off may, in truth, occur at higher frequencies; the flattening suggested by the few points in question attributable to chance happening. Finally, we note that the quantity: p(occ)
has connections to information theory. Often referred to as mutual infomation, it has frequently been used as a measure of variable dependence in information retrieval. It has been used in attempts to include co-occurrence data in models with weaker independence assumptions (van Rijsbergen, 1979); for the purposes of corpus-specific stemming (Croft and Xu, 1995); and for term selection in query expansion based on relevance feedback (Haines and Croft, 1993). It is also often used as a measure of variable dependence in computational linguistics (Church et al., 1991). In a very important sense, it can be taken as a measure of the information about one event provided by the occurrence of another (Fano, 1961). In our context, it can be taken as a measure of the information about relevance provided by the occurrence of a query term. In what follows, we shall adopt the notation MI(occ,rel) for this quantity, which we believe to be an object worthy of attention as a measure of term value in IR research. It will be the main focus in the analysis that follows.
4.3
MUTUAL INFORMATION AND IDF
Our interest is in modeling the weight of evidence in favor of relevance provided by the occurrence or non-occurrence of a query term. Presumably, the occurrence of a query term provides positive evidence and its absence is negative evidence. If we will assign a non-zero score only to those terms that appear in a document, this score should be, woe(rel : occ) – woe(rel : occ) This quantity, which we shall denote by 'woe, measures how much more evidence we have in favor of relevance when the term occurs in a document than we do when it is absent. Based on the formal definition of weight of evidence (eq. 2.1), together with that for mutual information, 'woe can be
The Use of Exploratory Data Analysis in Information Retrieval Research
51
expressed as:
(2.11)
By accepting some empirically motivated assumptions concerning query terms, it can be shown that: ___ 1) log p(occ | rel) ≈ log p(occ) ___ 2) – log p(occ | rel) is not too big; ___ ___ 3) log p(occ |rel) ≈ 0 and from this, that 'woe ≈ MI(occ, rel). For details the reader is referred to Greiff, 1999. There is little question about our ability to infer from the available data that MI(occ, rel) increases with decreasing document frequency. To a first order approximation, we can say that this increase is roughly linear with respect to log O(occ). ' woe ≈ MI(occ, rel) ≈ k2 – k1 log O(occ)
(2.12)
But, k2 can be ignored. By casual inspection of Figure 2.6, we see that any p(occ | rel) as reasonable linear approximation of the plot of MI (occ, rel) = log _______ p(occ) a function of log O(occ) will have an intercept value relatively close to 0. We are now left with: ' woe
≈
–k1 log O(occ) = k 1(–log O(occ))
(2.13)
Once the constant k2 has been eliminated, the remaining constant, k1, becomes irrelevant for the purposes of ranking. It will affect only the scale of the scores obtained, having no affect on the ranking order itself. And so we conclude that the idf formulation, idf = – log O(occ) = log should produce good retrieval performance.
(2.14)
52
4.4
ADVANCES IN INFORMATION RETRIEVAL
IMPROVING ON IDF
We argued in Section 4.1 that both theoretical and empirical considerations give reason to assume a flattening of MI(occ, rel) at both ends of the practical spectrum of document frequencies. If we can assume that the “true” form of the function that maps log O(occ) to MI(occ, rel) involves flattening at the extremes, the map to 'woe will exhibit similar shape. If we accept the hypothesis that the plot of Figure 2.6 is representative of the general behavior of query terms for the types of queries and collections we study, we should expect improved retrieval performance from a term weighting formula that accounts for the observed flattening. To test this prediction, we compared retrieval performance of two versions of the INQUERY IR system (Callan et al., 1992) on each of the ad-hoc tasks for TREC 1 through TREC 6 (Harman, 1997). Queries were formed by taking all words from both the title and description. The baseline system used pure idf term weighting with idf = – log O(occ)4. The test system used a flattened version of idf. For this version, weights were kept at 0 for all values of – log O(occ) below 1.0; increased at the same rate as – log O(occ) from – log O(occ) = 1.0 to – log O(occ) = 3.0; and maintained at a constant value for all terms for which – log O(occ) exceeded 3.0. Table 2.1 3-piece Piecewise-linear vs. Linear Versions of Idf avg. prec. baseline test
% diff
sign
wilcoxon
TREC1 TREC2
0.1216 0.1312 0.0693 0.1021
7.88 18/32 0.0325 47.36 10/40 0.0000
0.0201 0.0000
TREC3 TREC 4 TREC5 TREC6
0.0676 0.0680 0.0466 0.1185
0.1257 86.03 4/46 0.0000 0.1002 47.42 15/34 0.0047 0.0688 47.63 17/32 0.0222 0.1422 20.01 12/37 0.0002
0.0000 0.0006 0.0006 0.0000
-/ +
The results of these tests are summarized in Table 2.1. The test version outperforms the baseline system in terms of average precision, on all six query sets, by 20% or more on five of the six. The test system also outperforms the baseline system on a majority of queries on each of the six query sets. The “-/+” column gives the number of queries for which the test system performed Tests with idf = – log p(occ) were also run. For all test sets, performance differences were small, with – log O(occ) outperforming – log p(occ) on all 6 of the test sets.
4
The Use of Exploratory Data Analysis in Information Retrieval Research
53
below/above baseline. The column labeled {sign” gives the results of the sign test for each query set. Each value indicates the probability of the test version outperforming the baseline on as many of the queries as it did were each system equally likely to outperform the other. The column labeled “wilcoxon” gives the analogous probability according to the wilcoxon test, taking into account the size of the differences in average precision for each of the queries. The test results showed statistically significant improvement at the 5% level on all test sets according to both statistical measures. Improvement at the 0.1 % level was observed in three of the six runs according to the sign test and five of the six according to the wilcoxon. Improvement was found at all (11) levels of recall on TREC’S 2 through 5; all but the 50% recall level on TREC 1 and all but the 80% recall level on TREC 6.
5
PROBABILISTIC MODELING OF MULTIPLE SOURCES OF EVIDENCE
This section presents an analysis of the weight of evidence in favor of relevance offered by query/document features traditionally used for ranking in information retrieval. The predominate objective is to obtain a more precise and rigorous understanding of the relationship these retrieval characteristics have to the probability that a document will be judged relevant. The ultimate goal of this analysis is the development of a retrieval formula, the components of which can be understood in terms of statistical regularities observed in the class of retrieval situations of interest. A methodology is presented for the analysis of the relationship between query-document characteristics and the probability that a document will be judged relevant to the query. The characteristics that have been studied are coordination level, inverse document frequency, term frequency, and document length. Application of the methodology to a homogeneous collection of documents – 1988 news articles from the Associated Press (AP88), taken from volume 2 of the TREC data, evaluated for queries 151-200 from TREC 3 (Harman, 1995) – will serve as the vehicle for exposition of the principal techniques involved. In what follows we will show how query/document features can be studied, how a model in terms of this evidence can be formulated, and how parameters for it can be determined. The resulting model can be used directly as a scoring mechanism for which the retrieval status values (RSVs)5 that are produced have a precise probabilistic interpretation. Results will be presented suggesting that the modeling framework, and more important the general approach to the analysis of evidence, developed in this
5RSV is a term often used to refer to the score given to a document by a retrieval algorithm. It originated in a paper by Bookstein and Cooper (1976).
54
ADVANCES IN INFORMATION RETRIEVAL
study may lead to a ranking formula that performs as well as state-of-the-art retrieval formulas that have evolved over the years.
5.1
OVERVIEW OF THE MODELING STRATEGY
The objective of the analysis is to develop a model for the weight of evidence in favor of relevance given by the query/document features under consideration: (2.15) where the weight of evidence is conditioned on the query being evaluated. To be more precise, the weight of evidence that will be modeled is restricted as well to the subspace corresponding to those query/document pairs for which at least one of the query terms appears in the document. This is indicated by the * in the condition of the weight of evidence given in eq. 2.15. In general, IR systems do not evaluate documents that do not include at least one of the query terms. For that reason, all probabilities and weights of evidence considered in this section will be conditioned on the occurrence of at least one term. We will explicitly include the * in the formulas appearing in the upcoming presentation of the four models, but for the sake of reducing clutter in the notation, it will be left implicit in the sections that follow. The data analysis will result in the development of four models, which will be denoted by: M0, M1, M2, and M3. The focus of the modeling effort for each of these models will be: M0 : logO(rel|Qry = q,*) M1 : woe(rel : Coord = co | Qry = q, *) M2 : woe(rel : Idf = idf |Coord = co,Qry = q,*) M3 : woe(rel : Tf = tf | Idf = idf, Coord = co, Qry = q, *)
(2.16) (2.17) (2.18) (2.19)
with each model extending the previous one, in terms of the constraints imposed on the probability distribution. Although, the direct objective at each of the last three steps is to model weight of evidence conditioned on the query, the logodds in favor of relevance, and hence the probability of relevance, for the query is modeled as well. For example, log O(rel|tf, idf, co, q, *)
=
log O(rel|idf, co, q, *) +woe(reL : tf | idf, co, q, *)
In each case, the conditional probability can be derived from the conditional log-odds by: where D = log O(rel| . . .) To begin the process, model M0 is developed by simply estimating, for each query, the probability that an arbitrary document will be found to be relevant
The Use of Exploratory Data Analysis in Information Retrieval Research
55
to that query. This is described in more detail in Section 5.2. We proceed, in Section 5.3, to analyze evidence corresponding to coordination level. This results in the M1 model of relevance conditioned on the query being evaluated and the number of query terms occurring in the document. In Section 5.4, we see that inverse document frequency is correlated with residual log-odds of relevance, relative to the M1 model. Extension of the model to include idfi for each of the query terms, i = 1,2, . . ., produces the M2 model. Finally, analysis of the role of term frequency, discussed in Section 5.5, results in the M3 model. It is this model on which a ranking formula will be based.
5.2
BASE MODEL
A modeling assumption, derived from the Maximum Entropy Principle ( Greiff and Ponte, 1999; Greiff, 1999), is that the weight of the query/document → evidence, e, shall be considered independent of the query, q. The retrieval status value to be used will then be this weight of evidence, which can be assigned without knowledge of the prior probability of relevance for the query, p(rel|q). However, because of possible confounding, modeling is best accomplished by including the probability of relevance conditioned only on the query, q, in order to eliminate potential problems due to confounding. The basis for the analysis is established by first developing a model of the probability of relevance conditioned only on the query being evaluated, p(rel | q). We shall loosely refer to the probability as the prior probability of relevance for query q in the sense that it is the probability that a randomly selected document will be found relevant to the query before any evidence is observed, that is before the contents of the document are known. The estimation of this probability is straightforward. From the TREC data, the probability of relevance, p(rel|q), can be estimated using counts as, p(rel | q) = #rels / #docs and this can be converted to log-odds. A graph of these data is shown in Figure 2.7. It can be seen in this graph that the prior probability of relevance ranges over almost three orders of magnitude, from a little less than one in ten (log-odds ≈ -1) for query #189, to slightly above one in ten thousand (log-odds ≈ -4), for query #181.
5.3
MODELING COORDINATION LEVEL
Using M0 as a base, we model the weight of evidence offered by coordination level. Calculation of Residual Log-odds In order to model the weight of evidence offered by coordination level, the data were first grouped by the value of the Coord variable into subsets, C1, C2, . . .
56
ADVANCES IN INFORMATION RETRIEVAL
Figure 2.7
(Prior) log-odds of relevance for TREC-3 queries
Ci = {(q,d) ∈ Q x D | Coord(q,d) = i} where Q is the set of queries and is the set of documents. For each of these subsets of query/document pairs, an expected number of relevant documents was ^ computed. based on the estimated probabilitiss of relevance, p(rel|Qry = q), for each query: (2.20) where ni,q is the number of documents that contain i terms from query q. The ^ probability, p(rel|Qry = q), is the conditional probability ofrelevance given by model, M0, that was estimated by counting the fraction of documents relevant ^ = q), is an estimate of the number to the query. The product, ni,q . p(rel|Qry of these ni,q documents that can be expected to be relevant. The sum of these over all queries is then an estimate of the number of relevant documents in the set Ci. Although somewhat less intuitive, the summation given in eq. 2.20 can also be expressed as: (2.21) This formulation will be seen to be more useful as this technique is extended to the analysis of idf and tf as sources of evidence. Accompanying the calculation of ri, the actual number of documents in Ci that are relevant, ri, can be counted. Both of these can be transformed into log-odds: log and log where ni = |Ci| is the number of query/document pairs for which the cbordination level is i. The difference between the two:
The Use of Exploratory Data Analysis in Information Retrieval Research
Figure 2.8
57
Residual log-odds as function of coordination level: unsmoothed
can be viewed as the residual log-odds of relevance; the difference between the observed log-odds of relevance and the log-odds that would be predicted by a model that only uses information about which query is being evaluated. After the residuals were calculated for each subset of query/document pairs, Ci, a residual plot was produced. Figure 2.8 shows the scatterplot of residuals against coordination level. The lightly shaded bars in the background give a cumulative histogram. The height of a bar at Coord = i indicates the fraction of query/document pairs under consideration for which Coord(q, d) d i. The small circle along the bottom of the graph at Coord = 7 indicates that there were 0 relative documents in ∧ ∧ subset C7. Since ri is undefined for 0 relevant documents (i.e. ri = –∞ and hence resi is undefined, the circle serves to remind us that a point is missing from the plot at this value for the predictor variable. Fitting a Regression Line Motivated by the linearity suggested by Figure 2.8, a weighted linear regression was performed. The point at Coord = 7 represented by the small circle in Figure 2.8 has been ignored for the purposes of the regression. Since the point corresponds to very few documents, its effect on the overall regression would be negligible. Producing the M1 Model This regression line can be used to form a model, M1, that takes into account both the query being evaluated and the coordination level. The points marked by small x’s at each value, co = 1,2, . . ., that lie about the line, res = 0, show the difference between the log-odds predicted by the model, M1, and the log-odds actually observed — their proximity to the line, res = 0, demonstrating that the model provides a close fit to the data. The log-odds difference given by the model is equivalent to the weight of evidence in favor of relevance provided by the coordination level, conditioned
58
ADVANCES IN INFORMATION RETRIEVAL
on the query: woe(rel : co | q) (2.22) This M1 model advances the development of the complete model. The next step will be to use it as a basis for the analysis of the evidence provided by the rarity of a term.
5.4
MODELING INVERSE DOCUMENT FREQUENCY
In the previous section, we reported on a study of idf In that phase of the research, the behavior of idf was analyzed in isolation, independent of other explanatory variables. For the model being developed here, the behavior of idf will be studying in association with other variables of interest. The analysis for idf, and later for tf, as sources of evidence, will also depend on the study of residual log-odds of relevance. Whereas Coord is a feature of query/document pairs, for both idf and tf, data points involve individual query terms as well. For the analysis of coordination level, each pair corresponded to only one data point. For the analysis of idf and tf, each query/document pair corresponds to multiple data points — one for each term appearing in the document. In order that each document, and hence each relevance judgment, be treated equally, each point will be considered weighted by: w(q, d, t) = 1/Coord(q, d). In this way, the 5 points corresponding to a relevant query/document pair with a coordination level of 5 will each receive a weight of 1/5 – i.e. each will be considered as 1/5 of a relevant document; 2 points corresponding to a nonrelevant document with a coordination level of 2 will be considered as 1/2 of a non-relevant document; in total, 1 relevant and 1 non-relevant document. In order to study the value of a term’s inverse document frequency as a source of evidence, the evidence associated with learning that a specific query term, t, was one of the terms that occurred in the document, was studied. The weight of evidence tied to this event was estimated for each of the query terms over all of the queries. In order to carry out this estimation process, the data were first grouped into subsets, Ii = {(q, d, t) ∈ Q× D × T | Qry(t)=q, t = i} (T, the set of query terms), with one subset for each query term. (Occurrences of the same word used in two or more different queries are, for this purpose, considered different terms.) Here again, the actual number of relevant documents can be counted and the fraction of documents that are relevant can be compared
The Use of Exploratoy Data Analysis in Information Retrieval Research
Figure 2.9
59
Residual log-odds as function of idf unsmoothed
against the expected fraction for each subset. It must be kept in mind that both the observed and expected values are based on counts of entries weighted by the inverse of the coordination level. More precisely,
The calculation of the expected number of relevant documents for each subset is analogous to that given in eq. 2.21:
. where the estimated probability is calculated from the estimated log-odds of relevance: log O(rel|co, q)
=
log O(rel|q) + woe(rel : co | q)
with the second term being the weight of evidence predicted by the M1 model. Figure 2.9 shows a scatterplot of these residuals against idf value. Again, small circles are shown at the bottom of the graph for each residual that is undefined because the corresponding term did not appear in any relevant documents. Also visible is a small circle in the upper right hand comer. This corresponds to a term that only appeared in relevant documents. For this term the estimated probability of relevance is 1, giving infinite odds, and hence infinite log-odds, of relevance. The vertical bars in the background give a cumulative histogram for the (weighted) data points. As discussed in Section 4, the study of the weight of evidence provided by term idf values, suggests that the weight of evidence provided by idf is well-modeled by a 3-piece linear function. Review of the general form of residual plots, generated at various level of smoothing, tended to corroborate
60
ADVANCES IN INFORMATION RETRIEVAL
Figure 2.10
Residual log-odds as function of idf smoothed with regression
these earlier findings. Together, these two factors motivated the attempt to model woe(rel : idf | co, q) as a 3-piece linear function. In order to realize this, a linear regression was performed to determine parameters for the following linear model: (2.23) (2.24) and E 1IDF , yield the model, M2, The resulting estimates forthe parameters, EIDF 0 that minimizes the mean square error of all those models for which the expected value, E[ri], of the residual is a 3-piece linear function of idf with flat segments at the two extremes, and elbows at idf = 1.0 and idf = 2.0. Regressions were also run with a 4-parameter function, allowing for a general 3-piece linear model (one without the flat-segments restriction). These regressions showed no statistical evidence of non-zero slope in either of the tails, an indication that might have justified consideration of a more general model. Regressions were also run for other settings for the elbows; with values close to 1.0 and 2.0 resulting in the best fit. The 3-piece linear curve shown in Figure 2.10 shows the resulting model imposed on the scatterplot of smoothed residual values.
5.5
MODELING TERM FREQUENCY
Analysis of the evidence provided by term frequency proceeds along the same lines as that of inverse document frequency. Data points are grouped into subsets TF1, TF2,..., according to the number of occurrences of the query term in the document, Tf (q, d, t). For each subset, TFi, the observed number of relevant documents is determined, and the expected number of relevant documents is
The Use of Exploratory Data Analysis in Information Retrieval Research
Figure 2.11
Figure 2.12
61
Residual log-odds as function of tf unsmoothed
Residual log-odds as function of tf: smoothed with curve fit
calculated as:
where p(rel|idf, co, q) is calculated from the log-odds of relevance according to model, M2: ^ log Ô(rel|idf, co, q) = log Ô (rel|q) + woe(rel : co | q) + woe(rel : idf | co,q)
A scatterplot of the resulting residuals is shown in Figure 2.11. A number of transformations of the variables involved were tried. Figure 2.12 shows a plot of the residuals with log(tf) as predictor variable, smoothed to 50 bins. (Some of the original points, in particular the point for tf = 1, i.e. log(tf) = 0, are spread over a number of bins, accounting for several points with the same
62
ADVANCES IN INFORMATION RETRIEVAL
Figure 2.13
Figure 2.14
Residual log-odds as function of log(tf): smoothed with regression
Residual log-odds as function of tf smoothed with regression on original scale
The Use of Exploratory Data Analysis in Information Retrieval Research
Figure 2.15
63
Residual log-odds as function of document length
coordinates, overlapping one another, on the smoothed version of the graph.) The apparent linearity motivated the application of a simple linear regression. The regression results in a line given by the equation: (2.25) This line is overlaid on the smoothed scatterplot of Figure 2.13. The fit of the curve to the smoothed data on the more natural, unlogged tf scale can be seen in Figure 2.14.
5.6
MODELING DOCUMENT LENGTH
Many modem retrieval systems normalize term frequencies by document length in some way. The intuition is that a term is more likely to occur a larger number of times, the longer a document is. In the vector space model, term frequency is typically normalized by use of the cosine rule (Witten et al., 1994). Here the score is normalized by the Euclidean length of the document vector, which will tend to be greater for longer documents. In INQUERY, the term formula component of the ranking formula is: (2.26) where dl is document length and avg-dl is the average document length over the entire collection, and also incorporates a form of normalization based on document length. In order to consider the role of document length in this work, two types of analysis were performed. First, a residual plot was produced with document length as predictor, as it was for the other variables. The data were grouped by document length, and for each group: an expected number of relevant documents was computed, based on M3; the actual number of relevants were
64
ADVANCES IN INFORMATION RETRIEVAL
Figure 2.16
Residual log-odds as function of coordination level for individual queries
counted; and from these a residual log-odds was calculated. A plot for 50 bins is shown in Figure 2.15. From the plot it does not appear that there is any predictive value associated with document length. A number of other analyses were performed on different data sets with similar results. Based on these results, it was decided not to include document length as a source of evidence in the final model.
5.7
DISCUSSION
In this section, we review some of the important issues involved in the analysis presented in the previous sections. Coordination Level as Evidence The study of coordination level gives convincing evidence that the weight of evidence provided by the number of query terms appearing in the document, Coord = co, is well modeled as a linear function of co. This conclusion is supported by analysis of individual queries, over a number of different data sets. Figure 2.16 shows plots of wôe(rel : co | q) as a function of coordination level, co, for all of the TREC 3 queries for which at least one document of AP88 was judged relevant. Remarkable regularity is evidenced by this plot. This is especially true when we take into account that the number of relevant documents corresponding to many of the queries is small, which would cause us to expect substantial variability in the data. Similar regularity was observed for other data sets that were examined. Inverse Document Frequency As Evidence The modeling of idf is more problematic in two respects. First, the variance of residual log-odds across terms with similar idf values is large. The trend of increasing weight of evidence for increasing idf in Figure 2.9 is clear, although the number of points that are undefined (circles along the bottom of the graph) must not be forgotten. The
The Use of Exploratory Data Analysis in Information Retrieval Research
65
trend is also evident in smoothed versions of the plot, where all query terms, including those that do not appear in relevant documents, contribute to the visual effect. Even in smoothed versions, large variance is in display. This large variance makes it difficult to have confidence in the modeling decisions. Second, although the general trend is quite robust, the magnitude of the effect and the exact form of the increase, seem to vary considerably across varying collections and query sets. More study will be required to arrive at a more thorough understanding of the nature of the evidence provided by the value of the idf feature. Term Frequency As Evidence There are two reasons that speak in favor of a log transformation of the term frequency variable. The shape of the curve shown in Figure 2.11 strongly suggests a sharp decrease in the weight of evidence provided by one additional occurrence of a query term as term frequency increases. All data sets studied exhibited this same behavior. Intuitively, this is what one would expect, and this intuition has been the inspiration for a number of ranking formulas that are used in IR research. The difference between T f = 1 and T f = 2 should not be treated as equal to the difference between T f = 21 and Tf = 22. This is an indication that a log transformation is likely to provide a more appropriate scale on which to analyze the data. Document Length As Evidence We have concluded that there is no convincing indication that document length should be included in our model. This goes against much evidence to the contrary in the experimentation literature. Two points can be made. First, there may not be any effect due to document length when attention is restricted to a relatively homogeneous collection of documents such as a collection of news articles. The benefits of document length normalization in an environment such as TREC may be due to differences in behavior across the variety of sub-collections. Second, the distribution of document lengths over a sub-collection is much more uniform than it is over an entire TREC collection. It is reasonable to conjecture that even if there is an effect due to document length, it may be too small to be detected, and perhaps too small to make a difference in retrieval effectiveness, when retrieval is limited to a sub-collection. The above comments are corroborated to a degree by the graphs shown in Figure 2.17. The two graphs were produced from data corresponding to a combination of the Associated Press articles for 1988 (Ap88) and Federal Register documents for the same year (FR88). The graph at the top of this figure shows weight of evidence as a function of term frequency for documents of 600 words or less, whereas the bottom graph shows the same curve for documents of 600 words or more. A comparison of the two graphs indicates
66
ADVANCES IN INFORMATION RETRIEVAL
Figure 2.17 Residual plot for term-frequency for two ranges of document length over a combination of AP88 and FR88 documents
that the rise in weight of evidence is more gradual for the longer documents than it is for the shorter ones.
5.8
DEVELOPMENT OF A RANKING FORMULA
In this section we take the M3 model and from it derive a probabilistic ranking formula in terms of weight of evidence. The Probability Ranking Principle (Robertson, 1977) counsels us to rank documents by the probability of relevance. Equivalently, we may rank by the log-odds of relevance. The evidence we have considered to this point is: the coordination value; and, for each of the query terms appearing in the document, the idf and tf values. In terms of this evidence, the log-odds of relevance is given by:
There are two steps that need to be taken to convert this to a ranking formula. First we note that we do not expect to have knowledge of the prior odds of relevance, O(rel|q), for the query. However, this value is not needed. Since O(rel|q) is constant for a given query, we can ignore it, and, for the purposes of ranking, simply use the weight of evidence as an RSV, in place of log-odds:
The Use of Exploratory Data Analysis in Information Retrieval Research
61
Greiff, 1999, shows that this can be written as,
Using the regression equations for coordination level, inverse document frequency, and term frequency, produced to model the weights of evidence in the above expression, yields a retrieval status value in the form of
(2.27)
In the above formula E 0co can be ignored for the purposes of ranking. The coefficient values corresponding to the regression which have been fit in the previous sections are given by: = = =
(-0.66, +0.42) (-0.49, +1.27) (-0.55, +1.25)
giving the ranking formula: RSV
=
It is instructive to compare the ranking formula given in eq. 2.27 with formulas commonly used in IR systems. First, the general form of the M3 formula is different from traditional tf-idf formulas. In classic versions of the formula, some function of the document frequency is multiplied by some function of the term frequency and these products are added over all terms appearing in the document. In the M3 version, the idf and tf components are added.
68
ADVANCES IN INFORMATION RETRIEVAL
Figure 2.18
Recall-precision graph for TREC 3 queries on AP88
A second difference is the introduction of a 3-piece linear function of – log as part of the ranking formula. This form of the idf function is a direct result of the data analysis performed. Performance Evaluation To test the M3 model, the INQUERY information retrieval system (Callan et al., 1992) was modified to apply a formula based on eq. 2.27 for the RSV calculation. Performance was compared to an unmodified version of INQUERY as a baseline. A series of tests were run with various parameter settings. Figure 2.18 shows an 11-point recall-precision graph for the M3 model using the best parameter settings found. It is compared against the unmodified INQUERY system. The test system is represented by the solid line, with a broken line used for the baseline. Performance is almost identical at all levels of recall. This is encouraging, giving reason to believe that the traditional multiplicative tf-idf formulation may ultimately yield to a probabilistic ranking formula that is founded on observable statistical regularities. The test results shown in Figure 2.18 correspond to testing the M3 model on the same data set for which it was developed. However, test results on other data sets are promising. Figures 2.19 and 2.20 show results for two other document collections: ZIFF2 and WSJ89. In both cases, performance for the two systems were again quite comparable. The collections involved are similar to the AP88 collection whose analysis is discussed in this paper. The ZIFF2 test was run with the same TREC 3 queries that were used for analysis with the AP88 data. Here, the test system performs slightly better than the baseline system. Interestingly, the test system performs well also on the WSJ89 test, even though both the collection and the query set, queries from TREC 1, were different from those used to develop the model.
The Use of Exploratory Data Analysis in Information Retrieval Research
Figure 2.19
Recall-precision graphs for TREC 3 queries on ZIFF2
Figure 2.20
Recall-precision graphs for TREC 1 queries on WSJ89
69
70
ADVANCES IN INFORMATION RETRIEVAL
6
CONCLUSIONS
In this paper, we have reported on the development of a methodology for the study of evidence used for the ranking of documents in response to the expression of a user's need for information. EDA has been applied to all sources of evidence that participate in the retrieval strategy. It can fairly be categorized as the first example of what we have termed the data driven approach. The methodology based on weight of evidence was applied to an available set of retrieval data. This resulted in the development of a model for weight of evidence in favor of relevance given by the query/document features that were studied. A ranking formula has been derived directly from the weightof-evidence model. The derived formula has two important characteristics. The first is that it has a strict probabilistic interpretation. It is the weight of evidence in favor of relevance given by the features of the query/document pair being evaluated. The second is that the components of the formula can be decomposed. Both the form and the parameter values correspond to observable regularities of the retrieval data. A central motivation for this research has been the development of a theoretical framework that allows for various forms of evidence to be incorporated in a general retrieval system in a systematic way. It should be possible to apply the techniques developed as a result of this research to other sources of evidence. Alternatively, the same sources of evidence can be analyzed in different retrieval settings. The approach taken here opens the door to a research paradigm that can be brought to bear on the study of all aspects of the information retrieval problem. Directions that are ripe for immediate exploration include the study of other sources of evidence such as phrases, thesaurus terms, expanded queries, and studying the nature of retrieval in other languages.
Acknowledgments This material is based on work supported in part by the National Science Foundation, Library of Congress and Department of Commerce under cooperative agreement number EEC-9209623, and also supported in part by United States Patent and Trademark Office and Defense Advanced Research Projects Agency/ITO under ARPA order number D468, issued by ESC/AXS contract number F19628-95-(2-0235. Any opinions, findings and conclusions or recommendations expressed in this material are the author's and do not necessarily reflect those of the sponsors.
References Andrews, D. E (1978). Data analysis, exploratory. In Kruskal, W. H. and Tanur, J. M., editors, International Encyclopedia of Statistics, volume 7, pages 210– 218. Free Press, New York.
The Use of Exploratory Data Analysis in Information Retrieval Research
71
Beniger, J. R. and Brown, D. L. (1978). Quantitative graphics in statistics: A brief history. The American Statistician, 32(1): 1–9. Bookstein, A. and Cooper, W. (1976). A general mathematical model for information retrieval systems. Library Quarterly, 46(2): 153–167. Callan, J. P., Croft, W. B., and Broglio, J. (1995). TREC and TIPSTER experiments with INQUERY. Information Processing & Management, 3 1(3):327– 343. Callan, J. P., Croft, W. B., and Harding, S. M. (1992). The INQUERY retrieval system. In Proceedings of the 3rd International Conference on Database and Expert Systems Applications, pages 78–83. Church, K., Gale, W., Hanks, P., and Hindle, D. (1991). Using statistics in lexical analysis. In Zernik, U., editor, Lexical Acquisition: Exploiting OnLine Resources to Build a Lexicon, pages 115–164, Hillsdale, NJ. Lawrence Erlbaum Associates. Cooper, W. S. (1994). The formalism of probability theory in IR: A foundation or an encumbrance. In Croft, W. B. and van Rijsbergen, C. J., editors, Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, pages 242–248, Dublin, Ireland. Croft, W. B. and Xu, J. (1995). Corpus-specific stemming using word form cooccurence. In Proceedings for the Fourth Annual Symposium on Document Analysis and Information Retrieval, pages 147–1 59, Las Vegas, Nevada. Devroye, L. (1987). A Course in Density Estimation. Birkhauser, Boston. Fano, R. M. (1961). Transmission of Information; a Statistical Theory of Communications. MIT Press, Cambridge, MA. Good, I. J. (1950). Probability and the Weighing of Evidence. Charles Griffin, London. Good, I. J. (1983a). Good Thinking: The Foundations of Probability and its Applications. University of Minnesota Press, Minneapolis. Good, I. J. (1983b). Weight of evidence: A brief survey. In Bernardo, J. M., DeGroot, M. H., Lindley, D. V., and Smith, A. F. M., editors, Bayesian Statistics 2, pages 249–269. North-Holland, Amsterdam. Good, I. J. (1989). Statistical evidence. In Kotz, S. and Johnson, N. L., editors, Encyclopedia of Statistical Sciences, pages 65 1–656. Wiley. Greiff, W. R. (1999). Maximum Entropy, Weight of Evidence and Information Retrieval. PhD thesis, University of Massachusetts, Amherst, Massachusetts. Greiff, W. R. and Ponte, J. (1999). The maximum entropy approach and probabilistic IR models. To appear in ACM Transactions on Information Systems. Haines, D. and Croft, W. B. (1993). Relevance feedback and inference networks. In Korfhage, R., Rasmussen, E., and Willett, P., editors, Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 191–203, Pittsburgh, Pa. USA.
72
ADVANCES IN INFORMATION RETRIEVAL
Harman, D. (1993). Overview of the first Text REtrieval Conference (TREC1). In Harman, D. K., editor, The First Text REtrieval Conference (TRECI), pages 1–20, Gaithersburg, Md. NIST Special Publication 500-207. Harman, D. (1995). Overview of the third Text REtrieval Conference (TREC3). In Harman, D. K., editor, The Third Text REtreival Conference (TREC-3), pages 1–20, Gaithersburg, Md. NIST Special Publication 500-225. Harman, D. (1997). Overview of the fifth Text REtrieval Conference (TREC5). In Voorhees, E. M. and Harman, D. K., editors, The Fifth Text REtreival Conference (TREC-5), pages 1–28, Gaithersburg, Md. 500-238. NIST Special Publication 500-238. Hartwig, F. and Dearing, B. E. (1979). Exploratory Data Analysis. Number 07016 in Sage university paper series: Quantitative applications in the social sciences. Sage Publications, Beverly Hills. Hardle, W. (1990). Applied Nonparametric Regression. Cambridge University Press, Cambridge. Jeffreys, H. (1961). Theory of Probability. Oxford University Press, Oxford, 3 edition. Minsky, M. and Selfridge, O. G. (1961). Learning in random nets. In Cherry, C., editor, Information Theory: Fourth London Symposium, pages 335–347, London. Butterworths. Neter, J., Wasserman, W., and Kutner, M. H. (1985). Applied Linear Statistical Models: Regression, Analysis of Variance, and Experimental Designs. R. D. Irwin, Homewood, Ill., 2 edition. Robertson, S. E. (1977). The probability ranking principle in IR. Journal of Documentation,33:294–304. Robertson, S. E. and Sparck Jones, K. (1977). Relevance weighting of search terms. Journal of the American Society for Information Science, 27:129–146. Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and Hall, London. Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11–21. Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley Publishing Company, Reading, MA. van Rijsbergen, C. J. (1979). Information Retrieval. Butterworths, London, 2 edition. Witten, I. H., Moffat, A., and Bell, T. C. (1994). Managing Gigabytes: Compressing and Indexing Documents and Images. van Nostrand Reinhold, New York.
Chapter 3 LANGUAGE MODELS FOR RELEVANCE FEEDBACK Jay M. Ponte GTE Laboratories 40 Sylvan Rd Waltham MA 02451 USA ponte @acm.org
Abstract The language modeling approach to Information Retrieval (IR) is a conceptually simple model of IR originally developed by Ponte and Croft (1998). In this approach, the query is treated as a random event and documents are ranked according to the likelihood that the query would be generated via a language model estimated for each document. The intuition behind this approach is that users have a prototypical document in mind and will choose query terms accordingly. The intuitive appeal of this method is that inferences about the semantic content of documents do not need to be made resulting in a conceptually simple model. In this paper, techniques for relevance feedback and routing are derived from the language modeling approach in a straightforward manner and their effectiveness is demonstrated empirically. These experiments demonstrate further proof of concept for the language modeling approach to retrieval.
1
INTRODUCTION
The language modeling approach to Information Retrieval (IR), developed by Ponte and Croft (1998), treats the query as a random event generated according to a probability distribution. Intuitively, it is assumed that the end user of an IR system will have an idea of a prototypical document in which he or she is interested and will choose query terms likely to occur in documents similar to that prototype. Viewed this way, one can then think estimating a model of the term generation probabilities for the query terms for each document and ranking the documents according to the probability of generating the query.
74
ADVANCES IN INFORMATION RETRIEVAL
This approach has the intuitive appeal that one ranks documents by much the same process in which the user is presumed to have asked for them. However, as intuitiveness is in the eye of the beholder, perhaps it will be useful to consider some examples where viewing the retrieval problem from the standpoint of language modeling can be beneficial. The main advantage to the language modeling approach is that it does not require inferences as to the semantic content of documents. This makes the model conceptually simple and easy to generalize. For example, consider a generalization of the information retrieval problem where document boundaries are not predefined. This could occur when, for example, one has a large repository of online audio or video and wishes to retrieve relevant segments. Information retrieval systems are usually implemented to retrieve documents and, as such, use the document level statistics of term frequency (tf) and inverse document frequency (idf). This approach can be very effective for document retrieval but does not have a natural generalization to the case of audio/video segment retrieval (unless one pre-segments the data, which presupposes that there there is a single correct segmentation irrespective of the information need of the user). When document boundaries are ill-defined, document level statistics do not have obvious counterparts. However, the language modeling approach does generalize to this case since one can estimate local term occurrence probabilities from the data even in the absence of document boundaries. How would one accomplish this? If efficiency were of no object, one could compute local probabilities of query generation at each position of each query term using a kernel style estimator (Silverman, 1986). Since the cost of this operation is likely to be prohibitive, one would need to optimize the process, perhaps by precomputing statistics for fixed length blocks and computing the full estimate only for the most likely candidates. So, while the language modeling approach-does not offer the solution “out of the box” it does suggest a clear path to the solution of the problem. As a second example, consider the retrieval of noisy data such as OCR text or automatically recognized speech transcripts. When one views these problems from the standpoint of traditional vector space or probabilistic IR models, it is not obvious how to proceed. The reason for the difficulty is that the presence or absence of an indexing feature is itself an uncertain event. On the other hand, when viewed as a problem of query term probability estimation, one can utilize probability distributions, e.g., from a speech recognition system, in a fairly straightforward manner. The probability of generating an indexing feature would now need to be estimated taking the feature presence probability into account. Again, the language modeling approach does not offer an immediate solution to this problem and, certainly, much work remains to solve them, but the path to the solution is clear since both sources of uncertainty are modeled by probabilities.
Language Models for Relevance Feedback
75
As a final example, we get to the main topic of this paper, relevance feedback or document routing from the perspective of language modeling. Some existing approaches to relevance feedback are quite complex and others are somewhat ad hoc as will be discussed in section 3. On the other hand, the language modeling approach offers an intuitive and even obvious method of performing relevance feedback. That method will be the topic of the remainder of this paper. The organization of the paper is as follows. First, the language modeling approach to ad hoc retrieval is described. Following that, previous approaches to relevance feedback are presented. Finally, the language modeling approach to relevance feedback is described along with empirical results for interactive relevance feedback as well as document routing. The paper closes with conclusions and future directions.
2
THE LANGUAGE MODELING APPROACH TO IR
In the language modeling approach to IR, a language model is inferred for each document and the documents are ranked according to the estimate of producing the query according to that model. The query generation probability p(Q|Md), is the probability of producing the query given the language model of document d. To estimate this probability, the first step will be to estimate the probability for each term in the query. This probability will be estimated starting with the maximum likelihood estimate of the probability of term t in document d:
where tf(t,d) is the raw term frequency of term t in document d and dld is the total number of tokens in document d. A simplifying assumption will be made. Assume that given a particular language model, the query terms occur independently. The maximum likelihood estimator gives rise to the ranking formula 3 t ∈Q p∧ ml(t|Md) for each document.
2.1
INSUFFICIENT DATA
There are two problems with the maximum likelihood estimator, or perhaps a better way to state this is that there are two symptoms of one underlying problem: insufficient data for the reliable estimation of maximum likelihood. These two problems (or symptoms of the same problem) will now be examined a little more closely. The obvious practical problem with this estimator is that we do not wish to assign a probability of zero for a document that is missing one or more of the query terms. Doing so would mean that if a user included several synonyms in the query, a document missing even one of them would not be retrieved
76
ADVANCES IN INFORMATION RETRIEVAL
In addition to this practical consideration, from a probabilistic perspective, it is a somewhat radical assumption to infer that p(t|Md) = 0, i.e., the fact that we have not seen it does not make it impossible. Why is that? Recall that we are trying to estimate the term generation probabilities from a hypothesized language model for the document. We don’t know the ‘real’ model and so we do not want to make the assumption that our hypothesized model could not generate a term just because it did not. Instead, we make the assumption that a non-occurring term is possible, but no more likely than what would be expected by chance in the collection, i.e., cft ____ cs where cft is the raw count of term t in the collection and cs is the raw collection size or the total number of tokens in the collection. This provides us with a more reasonable distribution and circumvents the practical problem. It should be noted that in homogeneous databases, one may need to use a more careful smoothing estimate of the collection probability since, in some cases, the absence of a very frequently occurring word in a document (i.e., a word with the characteristics of a stopword) could conceivably contribute more to the score of that document in which it does not occur than it would for a document in which it does occur. This is not a problem in the collections studied, as they are heterogeneous in nature, and stopwords have been removed. However, this issue should be addressed in the future to insure that this approach will be immune to these pathological cases. The other problem with this estimator was pointed out earlier. If we could get an arbitrarily large sample of data from Md we could be reasonably confident in the maximum likelihood estimator. However we only have a document sized sample from that distribution and so the variation in the raw counts may partially be accounted for by randomness. In other words, if one were given two documents of equal length generated from the same language model, one would expect these two documents to have different numbers of occurrences of most of the terms due entirely to random variation.
2.2
AVERAGING
Since it is not possible to draw additional data from the same random process that generated each document, one can view the problem with the maximum likelihood estimator as one of insufficient data. To circumvent the problem of insufficient data, we are going to need an estimate from a larger amount of data. That estimate is the mean probability estimate of t in documents containing it:
where dft is the document frequency of t. This is a more robust statistic in the sense that we have a lot more data from which to estimate it, but it too has
Language Models for Relevance Feedback
77
a problem. It cannot be assumed that every document containing t is drawn from the same language model, and so there is some risk in using the mean to estimate p(t|Md). Furthermore, if the mean were used by itself, there would be no distinction between documents with different term frequencies. In order to benefit from the robustness of this estimator, and to minimize the risk, the mean will be used to moderate the maximum likelihood estimator by combining the two estimates using the geometric distribution as follows:
_ where f t is the the mean term frequency of term t in documents where t occurs normalized by document length. The intuition behind this formula is that as the t f gets further away from the normalized mean, the mean probability becomes riskier to use as an estimate. For a somewhat related use of the geometric distribution see Ghosh et al., 1983. There are several reasons why the geometric function is a good choice in addition to the shape. In the first place, the mean of the distribution is equal _ to f t , the expected rate of occurrence according to the mean probability. Secondly, the variance of this distribution is larger than the mean. For this reason, the case where a large number of relatively low rates of occurrence along with relatively few cases where there is a large rate of occurrence are captured by this function. Finally, this function is defined in terms of only the mean and the t f so it can be computed without adding to the space overhead of the index and in minimal time. At the present time, IR systems need to be able to index gigabytes of text at rates of hundreds of megabytes per hour and to process queries in seconds. Complex estimation techniques that require significant additional indexing time, even if effective, would not be acceptable for real systems. In addition, techniques that require significant additional space, such as storing an additional number per word occurrence, would not be feasible. This makes the geometric distribution an excellent choice from a systems engineering perspec tive.
2.3
COMBINING THE TWO ESTIMATES ∧
The geometric distribution will be used in the calculation of p(Q|Md), the estimate of the probability of producing the query for a given document model as follows:
78
ADVANCES IN INFORMATION RETRIEVAL
In this formula, the first term is the probability of producing the terms in the query and the second term is the probability of not producing other terms. This can be regarded as a “background” model for each document that captures the likelihood of other terms for the document. This quantity essentially accounts for other terms that would be better discriminators of the document than the query terms, i.e., terms that would be unlikely to be left out of the query by users interested in the document. This functions in lieu of a more expensive estimation from missing data solution and allows for the tractable ∧calculation of the background probabilities. Also notice the risk function, Rt,d and the cft background probability ____ cs mentioned earlier. This function is computed for each candidate document and the documents are ranked accordingly. This approach will be applied to the problems of relevance feedback and routing in section 4. Prior to that, some other approaches to these problems will be discussed.
3 3.1
RELATED WORK THE HARPER AND VAN RIJSBERGEN MODEL
In 1978, Harper and van Rijsbergen developed a method for using relevance information, obtained by relevance feedback, to obtain better estimates for the probability of relevance of a document given the query. This work attempted to correct for the assumption of independence which the authors did not think was realistic (Harper and van Rijsbergen, 1978). Given complete relevance information, an approximation of the dependence of query terms was defined by the authors by means of a maximal spanning tree. Each node of the tree represented a single query term and the edges between nodes were weighted by a measure of term dependence. Rather than computing the fully connected graph, the authors computed a tree that spanned all of the nodes and that maximized the expected mutual information computed as follows:
where i and j range over the query terms, P(xi, xj) is the probability of term xi and term xj occurring in a relevant document, P(xi) is the probability of term xi occurring in a relevant document and, likewise, P(xj) is the probability of term xj occurring in a relevant document. Note that these probabilities refer to the probability of occurrence of the term or term pair one or more times
Language Models for Relevance Feedback
79
in a document. The within document frequency is not accounted for by this measure. Harper and van Rijsbergen did two sets of experiments using this approximation. The first was to determine the upper bound performance of the term dependence model vs. a term independence model using the complete relevance judgments. These experiments showed that under these conditions, the dependency graph did, in fact, yield useful information and resulted in more effective retrieval than the model that did not utilize this information. The second set of experiments used the dependency graph in the presence of a limited number of relevance judgments; both ten and twenty judged documents were tested. In order to compensate for the relatively sparse data, the authors used a Bayesian estimator with Jeffrey’s prior as recommended in van Rijsbergen, 1977, to estimate the probabilities. The authors found that with a limited number of judgments, it was still the case that the dependency graph model yielded better retrieval. It should be noted that the collections used for these experiments were quite small, as larger collections were not available at the time, and the authors cautioned against drawing firm conclusions from these results (Harper and van Rijsbergen, 1978).
3.2
THE ROCCHIO METHOD
A commonly used method of relevance feedback is due to Rocchio (Rocchio, 1971). The Rocchio method provides a mechanism for the selection and weighting of expansion terms and is defined as follows:
(3.1) where D is the weight assigned for occurring in the initial query, E is the weight assigned for occurring in relevant documents, J is the weight assigned for occurring in non-relevant documents, w(t) is a weighting function, generally based on term frequency _ and/or document frequency, R is the set of documents judged relevant and R is the set of documents judged non-relevant. This formula can be used to rank the terms in the judged documents. The top N can then be added to the query and weighted according to the Rocchio formula. This is a reasonable solution to the problem of relevance feedback that works very well in practice. However, there is no principled way to determine the optimal values of D E and J and so these parameters are generally set empirically. Also, the weighting function, w(t) is based on heuristic use of the collection statistics. The method described in Section 4 will be derived from the language modeling approach and does not require heuristics or empirically set parameters.
80
3.3
ADVANCES IN INFORMATION RETRIEVAL
THE INQUERY MODEL
The INQUERY inference network model was developed by Turtle (1991). See Figure 3.1 for an example network. The document portion of the network is computed in advance and the query portion is computed at retrieval time. The document side consists of document nodes d1 . . . di, text nodes t1 . . . tj and concept representation nodes r1 . . . rk. The document nodes represent abstract documents. A document may consist of text, figures, images, formatting information, etc. The text nodes represent the textual component of the documents. A given text node corresponds to the observation of the text of a document. For our purposes, we can assume that there is a one-to-one and onto relationship between text nodes and document nodes but, of course, in general that is not necessarily true. Several documents could share a textual component and, conversely, non-textual information can be considered as a another source of information within the same model, but these two cases are not relevant to the present discussion.
Figure3.1
Example inference network.
The concept representation nodes r1 . . . rk are features that can be possessed by the document texts. A link to an r node means that a document is “about” that particular concept. There is some uncertainty to be resolved due, for example,
Language Models for Relevance Feedback
81
to differences in word sense e.g., the word “train” may mean that a document is about trains as a mode of transportation or it may mean that document is about training employees. This distinction between words and concepts is the key distinction to be made here. The uncertainty of indexing in information retrieval is based on not having a direct representation of concepts. Instead, the probabilities of concepts are estimated from word occurrence statistics. Query Network. The query side of the network consists of the query concepts, c1 . . . cm, some number of queries, q1 and q2 in this diagram, and I the information need. The query concepts are the primitive concepts used to represent the query. The concepts will be represented by the words of the query, but again there is a fair amount of uncertainty due to differences in word sense. The query nodes represent individual queries. In this model, the information need of the user can be represented by more than one query, using more than one type of information. The information need itself is known only to the user and needs to be inferred from the queries. There can be an additional layer between the query nodes and the concept nodes representing intermediate operators such as phrases or Boolean operators. The task then is to calculate the belief in the information need of the user given each document. The documents will then be ranked by their belief scores (Turtle, 1991).
Figure 3.2
Close-up view of query network.
82
ADVANCES IN INFORMATION RETRIEVAL
Relevance Feedback. Turtle included discussion of relevance feedback, however, the theoretical work of actually adding relevance feedback to the inference network model and the implementation of the theoretical ideas was done by Haines (1996). A new type of node was added to the inference network to reflect the relevance judgments provided by the user. These annotation nodes were added to the query side of the inference network and the evidence from the annotations was propagated through the network using message passing. Figure 3.2 shows acloseup view of a portion of the query network from Figure 3.1. Once again, c1 and c2 represent query concept nodes, ql represents a query and I represents the information need of the user. The relevance judgments will be represented by means of a set of annotation nodes.
Figure 3.3 Annotated query network.
In order to incorporate annotations, each query concept node is annotated using three additional nodes as shown in figure 3.3. The nodes k1 and k2 represent the proposition that nodes c1 and c2 imply that the information need I has been satisfied. The nodes jl and j2 represent the observed relevance judgments. The and nodes are used to require that the query concept occur in the document in question in order for an annotation to have an effect on the score. Haines was able to show that an inference network annotated in this way can be computed efficiently (Haines, 1996), thereby showing that relevance
Language Models for Relevance Feedback
83
feedback can be incorporated into the inference network model in practice as well as in theory. The only drawback of this technique is that it requires inferences of considerable complexity. In order to make use of relevance judgments, two additional layers of inference and several new propositions are required. So, while this method has been shown to work, it is more complex than is desirable. Due to the complexity, the implications of this technique for improvement of relevance feedback are not obvious. In contrast, the method of relevance feedback that will be described in section 4 follows more directly from the language modeling approach to information retrieval. This technique is very straightforward and will be shown to work as well as existing methods of relevance feedback.
3.4
EXPONENTIAL MODELS
Beeferman et al., 1997, describe an approach to predicting topic shifts in text using exponential models. The model utilized ratios of long range language models and short range language models as follows:
where pl (x) is the probability of seeing word x given the context of the last 500 words and ps(x), is the probability of seeing word x given the two previous words. The intuition behind this approach is that when a long range language model is not able to predict the next word better than a short range language model, that indicates a shift in topic. A similar ratio based method will be used in this paper for the identification of useful terms for relevance feedback. This method will now be described in detail.
4
QUERY EXPANSION IN THE LANGUAGE MODELING APPROACH
Query expansion techniques have a natural interpretation in the language modeling approach. Recall that the underlying assumption of this approach is that users can choose query terms that are likely to occur in documents in which they would be interested, and that separate these documents from the ones in which they are not interested. Also note that this notion has been developed into a ranking formula by means of probabilistic language models. Since probabilities can be estimated as shown in Section 2.3 to, in some sense, evaluate terms chosen by the user, the same probabilities can be used by the system to ‘choose’ terms in much the same way.
84
4.1
ADVANCES IN INFORMATION RETRIEVAL
INTERACTIVE RETRIEVAL WITH RELEVANCE FEEDBACK
Consider the relevance feedback case, where a small number of documents are judged relevant by the user. The relevance of all of the remaining documents is unknown to the system. The system can estimate the probability of producing terms from each document’s probability distribution. From this, a random sample could be drawn from that distribution, but this would be undesirable since, of course, the most common terms would tend to be drawn. However, it is also possible to estimate the probability of each term in the collection as a whole using the term counts. The log ratio of the two can then be used to do as the user does: choose terms likely in documents of interest but unlikely in the collection as a whole.
4.2
DOCUMENT ROUTING
In the routing case, a training collection is available with a large number of relevance judgments, both positive and negative, for a particular query. The task is to use the training collection to construct a query which can then be run on a second collection for which relevance information is not available. Since both relevant and non-relevant documents are known, the ratio method can utilize this additional information by estimating probabilities for both sets. This process is described in more detail in section 4.6. Once again, the task is to choose terms associated with documents of interest and to avoid those associated with other documents. The ratio method provides a simple mechanism to do that.
4.3
THE RATIO METHOD
Incorporating relevance feedback into the language modeling approach turns out to be quite straightforward. Recall that in the language modeling approach, the prototypical user considers the documents of interest and chooses terms that will separate these documents from the rest of the collection. An analogous method will be used to choose terms for relevance feedback. The judged documents can be used to estimate a language model of the documents of interest using the method described in Section 2.3. The probability of a term t cft the collection in the collection as a whole can be estimated, as before, by ____ cs frequency of t normalized by the total number of tokens in the collection. These two probability models can then be used in manner similar to that used by Beeferman et al., 1997, for the purposes of topic boundary prediction. Recall from section 3.4 that the method they used was to take the log ratio of a long range language model vs. a short range language model in order to predict topic shifts.
Language Models for Relevance Feedback
85
In the relevance feedback case, the two estimates from above will be used in a similar manner to predict useful terms. Terms can then be ranked according to the probability of occurrence according to the relevant document models as compared to the collection as a whole, i.e. terms can be ranked according to the sum of the log ratio of each relevant model vs. the collection model.
where R is the set of relevant documents, P(t|Md) is the probability of term t given the document model for d as defined in Section 2.3, cf t is the raw count of term t in the collection and cs is the raw collection size. Terms are ranked according to this ratio and the top N are added to the initial query.
4.4
EVALUATION
Results are measured using the metrics of recall and precision. Table 3.1 shows the contingency table for retrieval of a set of documents where: r= Relevant –r = Non-Relevant R= Retrieved – R = Non-Retrieved
Table 3. I
Set Set Set Set
Contingency table for sets of documents.
If one were retrieving a set of documents, recall is the probability that a document has been retrieved given that it is a member of the set of relevant documents. Precision is the probability that a document is a member of the relevant set given that it has been retrieved. Defining these measures in terms of the contingency table yields: Recall
=
Precision
=
A third measure, fallout, is the probability that a document is not relevant given that is has been retrieved. Fallout, in terms of the contingency table looks like this:
86
ADVANCES IN INFORMATION RETRIEVAL
Fallout
=
Measuring performance in terms of recall and fallout would be more typical in a classification task. However, fallout is not a very interesting measure for retrieval. Suppose, for example, that a collection consisted of one million documents, and suppose one hundred of them were relevant, which would be a typical proportion. In that case one can do a reasonable optimization for fallout by not retrieving any documents at all. Due to the huge disparity between the numbers of relevant and non-relevant documents, recall and precision are the preferred metrics. For the evaluation of ranked retrieval, precision can be measured at several levels of recall to show the tradeoff. Other measures include the average precision over all relevant documents and precision after R documents have been retrieved for various values of R. Each of these measures will be reported for each of the experiments.
4.5
EXPERIMENTS
The first series of experiments compares the Rocchio method of relevance feedback (Rocchio, 1971) to the language modeling approach with varying numbers of relevance judgments. The experiments were performed on on TREC disks 2 and 3 using TREC topics 202-250 (Harman, 1996). For the language model approach, terms are ranked by the log ratio of the probability in the judged relevant set vs. the collection as a whole. For the Rocchio method, the weighting function was tf.idf and no negative feedback was used J = 0). The results of feeding back one relevant document and adding five terms to the initial query are shown Table 3.2. The top of the table lists the interpolated precision at 0 - 100% recall. The next row of the table shows the uninterpolated average precision. The bottom part of the table shows the precision at increasing numbers of documents. Finally, the row labeled ‘R-Precision’ is the precision after ‘R’ documents have been retrieved. For each query, ‘R’ is chosen to be the number of relevant documents for that query. The language modeling approach is slightly better at the top of the ranking but Rocchio is better at most of the points. The first result column shows the performance of the Rocchio method. The second column shows the corresponding result for the language modeling approach. The third column reports the percent change. Column four is of the form I/D where I is the count of queries for which performance improved using the new method and D is count of queries for which performance was different. Column five reports significance values according to the sign test and column six does likewise according to the Wilcoxon test. The entries in these
Language Models for Relevance Feedback
87
Table 3.2 Comparison of Rocchio to the Language modeling approach using 1 document and adding 5 terms on TREC queries 202-250 on TREC disks 2 and 3. Rocchio
LMRF
%chng.
Sign
Wilc.
16/39
0.1684
0.3004
+2.5 -9.8 -3.3 -11.5 -12.2 -18.5 - 17.2 -28.8 -45.9 -29.9 126.4 –10.91
2/2 21/44 22/44 19/43 15/40 12/36 9/29 7/22 4/12 216 2/3 22/49
0.2500 0.4402 0.5598 0.2712 0.0769 0.0326* 0.0307* 0.0669 0.1938 0.3438 0.5000 0.2841
undef 0.0707 0.2997 0.1413 0.1323 0.0434* 0.0355* 0.04 12* 0.0680 undef undef 0.0841
-5.6 -6.4 -9.1 -8.6 -5.8 -8.3 -2.6 -4.4 -2.9 -9.66
13/30 15/35 15/37 13/37 21/43 19/43 20/42 19/42 16/39 18/43
0.2923 0.2498 0.1620 0.0494* 0.5000 0.2712 0.4388 0.3220 0.1684 0.1802
0.3036 0.2041 0.0861 0.0873 0.2024 0.0961 0.3399 0.1976 0.3004 0.0626
Relevant: Rel. ret.: Precision
6501 3366
6501 3270
-2.85
at 0.00 at 0.10 at 0.20 at 0.30 at 0.40 at 0.50 at 0.60 at 0.70 at 0.80 at 0.90 at 1.00 Avg: Precision at:
0.9694 0.5346 0.4209 0.3484 0.2694 0.2156 0.1478 0.0890 0.0519 0.0135 0.0047 0.2410
0.9932 0.4822 0.4070 0.3082 0.2365 0.1757 0.1224 0.0633 0.028 0.0094 0.0060 0.2147
5 docs: 10 docs: 15docs: 20 docs: 30 docs: 100 docs: 200docs: 500 docs: 1000 docs: R-Precision:
0.5878 0.5102 0.4803 0.4490 0.3891 0.2537 0.1835 0.1119 0.0687 0.2930
0.5551 0.4776 0.4367 0.4 102 0.3667 0.2327 0.1788 0.1070 0.0667 0.2647
1/D
two columns marked with a star indicate a statistically significant difference at the 0.05 level. Note that these are one sided tests. Overall, the two techniques tend to track each other reasonably well, though the Rocchio method outperforms the language modeling approach at most levels of recall and on average. The results of feeding back two relevant documents and adding five terms to the initial query are shown in Table 3.3. Once again, the baseline technique is Rocchio. Notice that the improvements of the language modeling approach over Rocchio are not statistically significant. Again, as with a single document, the two techniques provide similar performance at most levels of recall. Tables 3.4 and 3.5 show results of feeding back ten relevant documents adding five and ten terms respectively. As in the previous experiments, the baseline technique is Rocchio. Note that for five terms, recall, average precision and precision at several levels of recall are significantly better using the language modeling approach.
88
ADVANCES IN INFORMATION RETRIEVAL
Table 3.3 Comparison of Rocchio to the Language modeling approach using 2 documents and adding 5 terms on TREC queries 202-250 on TREC disks 2 and 3. Rocchio
LMRF
%chng.
1/D
Relevant: Rel. ret.: Precision
6501 3650
6501 3590
Sign
Wilc.
–1.64
24/44
0.7743
0.4351
at0.00 at0.10 at 0.20 at0.30 at 0.40 at0.50 at 0.60 at 0.70 at0.80 at0.90 at1.00 Avg:
0.9864 0.5679 0.4703 0.3889 0.3115 0.2494 0.1778 0.1215 0.0551 0.0156 0.0093 0.2739
0.9908 0.5874 0.5072 0.4013 0.3186 0.2652 0.1888 0.1099 0.0608 0.0222 0.0184 0.2825
+0.4 +3.4 +7.8 +3.2 +2.3 +6.4 +6.2 -9.6 +10.4 +42.2 +98.6 +3.16
33 22/45 26/46 26/45 23/40 22/39 17/33 7/25 8/15 4/6 213 27/49
0.5000 0.6170 0.2307 0.1856 0.2148 0.2612 0.5000 0.0216* 0.5000 0.3438 0.5000 0.2841
undef 0.2882 0.1361 0.2473 0.2726 0.2173 0.3116 0.1264 0.4548 undef undef 0.2003
0.6735 0.5939 0.5388 0.5092 0.4456 0.2890 0.2067 0.1225 0.0745 0.3238
0.7061 0.6102 0.5537 0.4939 0.4367 0.2853 0.2120 0.1200 0.0733 0.3264
+4.8 +2.7 +2.8 -3.0 -2.0 -1.3 +2.6 -2.1 -1.6 +0.79
21/36 2443 21/40 18/41 19/44 25/46 27/43 24/42 2444 25/45
0.2025 0.2712 0.4373 0.2664 0.2257 0.7693 0.0631 0.8600 0.7743 0.2757
0.1853 0.2710 0.22 18 0.4053 0.4744 0.5737 0.1509 0.4502 0.435 1 0.3258
Precision at:
5 docs: 10 docs: 15docs: 20 docs: 30 docs: 100 docs: 200docs: 500 docs: 1000 docs: R-Precision:
However, it should be noted that ten relevant documents is a large number to expect from a typical user. The results for ten terms are similar. Also note that both techniques show improved results by the addition of more terms, as expected. The main point to take away from this series of experiments is that a relevance feedback technique that follows directly from the language modeling approach works well without any ad hoc additions. Note that in most IR systems, relevance feedback techniques make use of term weighting. No attempt was made to use term weighting in these experiments. This matter is discussed in more detail in section 5.1.
4.6
INFORMATION ROUTING
In the routing task, a training collection with relevance judgments, positive and negative, is provided. The task is to develop a query that will perform well on a test set of new documents. The language modeling approach to
Language Models for Relevance Feedback
89
Table 3.4 Comparison of Rocchio to the Language modeling approach using 10 documents and adding 5 terms on TREC queries 202-250 on TREC disks 2 and 3. Rocchio
LMRF
%chng.
1/D
Sign
Wilc.
Relevant: Rel. ret.: Precision
6501 3834
6501 4124
+7.56
28/44
0.0481*
0.0168*
at 0.00 at 0.10 at0.20 at 0.30 at 0.40 at0.50 at 0.60 at 0.70 at 0.80 at0.90 at1.00 Avg: Precision at:
0.9064 0.6134 0.5317 0.4151 0.3257 0.2706 0.2120 0.1374 0.0731 0.0173 0.0103 0.2943
0.9083 0.7117 0.5701 0.4875 0.4054 0.3198 0.2384 0.1364 0.0658 0.0175 0.0051 0.3279
+0.2 +16.0 +7.2 +17.4 +24.5 +18.2 +12.5 -0.7 - 10.0 +1.2 -50.0 $11.44
9/18 32/42 28/45 27/46 29/46 24/39 26/38 19/32 12/20 2/8 0/3 31/49
0.5927 0.0005* 0.0676 0.1510 0.0519 0.0998 0.0168* 0.8923 0.8684 0.9648 0.1250 0.0427*
0.6763 0.0005* 0.0431* 0.0200* 0.0037* 0.0194* 0.0355* 0.7728 0.8047 undef undef 0.0186*
5docs: 10 docs: 15docs: 20 docs: 30docs: 100 docs: 200docs: 500 docs: 1000docs: R-Precision:
0.7061 0.6306 0.5728 0.5337 0.4810 0.3169 0.2251 0.1290 0.0782 0.3474
0.7347 0.6878 0.6218 0.5796 0.5156 0.3408 0.2457 0.1367 0.0842 0.3773
+4.0 +9.1 +8.6 +8.6
14/28 23/36 25/33 29/40 26/39 26/4 I 25/42 27/46 28/44 29/43
0.5747 0.0662 0.0023* 0.0032* 0.0266* 0.0586 0.1400 0.1510 0.0481* 0.0158*
0.2545 0.0176* 0.0160* 0.0110* 0.0230* 0.0284* 0.0576 0.0452* 0.0168* 0.0181*
+7.2
+7.5 +9.2 +6.0 +7.6 +8.61
information routing will use similar techniques to those used for relevance feedback. The experiments described will show two different ratio methods. The first is identical to the relevance feedback method described in section 4 where documents were either relevant or unknown. The second method uses models of both the relevant and non-relevant sets of documents. It will be shown that the negative information is very useful in the context of the language modeling approach, just as it is in existing approaches to routing. Ratio Methods With More Data. In this section, the ratio method will be described in the context of the routing task. In addition, a second ratio method will be developed to make use of the additional information available in the routing task. Empirical results will be shown in section 4.6. As in the relevance feedback case, terms are ranked by the log ratio of the probability in the judged relevant set vs. the collection as a whole. These two
90
ADVANCES IN INFORMATION RETRIEVAL
Table 3.5 Comparison of Rocchio to the Language modeling approach using 10 documents and adding 10 terms on TREC queries 202-250 on TREC disks 2 and 3. Rocchio
LMRF
%chng.
Sign
Wilc.
35/45
0.0001*
0.0013*
-1.3 +5.2 +6.6 +0.5 +14.7 +12.9 +23.6 +25.5 +47.9 +29.4 -4.6 +8.11
6/10 24/42 27/45 25/46 30/46 3 0/45 28/39 22/35 13/21 5/9 0/2 33/49
0.8281 0.2204 0.1163 0.3294 0.0270* 0.0178* 0.0047* 0.0877 0.1917 0.5000 0.2500 0.0106*
0.4392 0.1482 0.I169 0.3819 0.0037* 0.0153* 0.0164* 0.0442* 0.0315* undef undef 0.0190*
+4.9 +3.4 +3.3 +3.2 +3.0 +2.9 +2.5 +4.9 +7.0 +7.43
13/23 20/37 22/37 27/46 23/40 31/44 26/43 29/45 35/45 30/45
0.3388 0.3714 0.1620 0.1510 0.2148 0.0048* 0.1110 0.0362* 0.0001* 0.0178*
0.242 I 0.1709 0.1066 0.0987 0.1519 0.0405* 0.1102 0.0346* 0.0013* 0.0431*
Relevant: Rel. ret.: Precision
6501 3933
6501 4210
+7.04
at 0.00 at 0.10 at 0.20 at0.30 at 0.40 at 0.50 at 0.60 at 0.70 at0.80 at0.90 at 1.00 Avg: Precision at:
0.9478 0.6882 0.5509 0.4766 0.3608 0.2829 0.1973 0.1242 0.0650 0.0203 0.0094 0.3138
0.9354 0.7242 0.5873 0.4789 0.4138 0.3193 0.2439 0.1559 0.0961 0.0263 0.0090 0.3393
5 docs: 10 docs: 15docs: 20docs: 30 docs: 100 docs: 200 docs: 500 docs: 1000 docs: R-Precision:
0.7469 0.6653 0.6150 0.5755 0.5150 0.3300 0.2356 0.1327 0.0803 0.3550
0.7837 0.6878 0.6354 0.5939 0.5306 0.3396 0.2415 0.1391 0.0859 0.3813
1/D
term sets will be used to construct queries as described below. In the results section, this method will be referred to as ratio 1. The second ratio method, referred to as ratio 2 in the results section, uses the log ratio of the average probability in judged relevant documents vs. the average probability in judged non-relevant documents as shown.
Language Models for Relevance Feedback
91
Recall that in the relevance feedback case, a small number number of documents were known to be relevant but the relevance of the rest of the collection was unknown. In the routing case, complete relevance judgments are available for the training collection. Ratio method 2 makes use of this additional information. Routing Results and Discussion. Experiments were performed on the TREC 2 routing task. The training data was TREC disks 1 and 2 and the test set was TREC disk 3. The queries were derived from TREC topics 51-100. The initial queries consisted of the terms from the concept fields. These queries were augmented by adding the top 20 terms as ranked by each of the ratio methods. The results for these two augmented query sets are shown in Figure 3.4. Notice that ratio 2 outperforms ratio 1, which is to be expected, since ratio 2 makes use of the non-relevance judgments in addition to the relevance judgments. For purposes of comparison, the official UMass TREC 2 results which used the Rocchio method are also included in the graph. Notice that ratio method 1 performs better than the UMass run at the high precision end of the curve, is approximately equal in the middle and drops below at the high recall end of the curve. Ratio method 2 is better for most of the curve dropping below the UMass run at the high recall end of the curve. Also see Table 3.6.
Figure 3.4
Comparison of ratio methods 1 and 2 on TREC 93 routing task.
92
ADVANCES IN INFORMATION RETRIEVAL Table 3.6
Comparison of ratio methods one and two on TREC 93 routing task. One
TWO
%chng.
I/D
Sign
Wilc.
0.0047*
0.0039*
Relevant: Rel. ret.: Precision
10485 7246
10485 7599
$4.87
28/39
at0.00 at0.10 at0.20 at0.30 at0.40 at 0.50 at 0.60 at 0.70 at0.80 at 0.90 at 1.00 Avg: Precision at:
0.8300 0.6712 0.5752 0.4953 0.4259 0.3613 0.2909 0.2159 0.1273 0.0594 0.0055 0.3543
0.8525 0.7026 0.6241 0.5350 0.4605 0.3983 0.3229 0.2411 0.1575 0.0639 0.0047 0.3839
+2.7 +4.7 +8.5 +8.0 +8.1 +10.2 +11.0 +11.7 +23.7 $7.7 -14.9 +8.36
8/14 28/42 32/48 30/48 27/50 29/48 28/44 23/37 18/28 9/16 113 34/50
0.3953 0.0218* 0.0147* 0.0557 0.3359 0.0967 0.0481* 0.0939 0.0925 0.4018 0.5000 0.0077*
0.1653 0.0047* 0.0020* 0.0060* 0.0535 0.0424* 0.0499* 0.0228* 0.0065* 0.2190 undef 0.0007*
5 docs: 10docs: 15docs: 20docs: 30docs: 100docs: 200 docs: 500 docs: 1000docs: R-Precision:
0.6880 0.6680 0.6440 0.6360 0.5987 0.4612 0.3660 0.2337 0.1449 0.3901
0.7040 0.7020 0.6840 0.6590 0.6260 0.4888 0.3865 0.2452 0.1520 0.4171
+2.3 +5.1 +6.2 +3.6 +4.6 +6.0 +5.6 +4.9 +4.9 +6.92
12/18 18/29 22/32 21/36 22/34 29/43 28/41 25/41 28/39 29/43
0.1189 0.1325 0.0251* 0.2025 0.0607 0.0158* 0.0138* 0.1055 0.0047* 0.0158*
0.2040 0.0536 0.0158* 0.0720 0.0246* 0.0036* 0.0063* 0.0150* 0.0039* 0.0071*
Also notice that the average precision for ratio 2 is competitive with the best systems in TREC to date. For purposes of comparison, the best routing average precision result in any of the TREC evaluations is approximately 41 % (Harman, 1996). This best ever result was obtained using a search based optimization of term weights. The results presented here did not require any term weighting at all, nor were other sources of evidence, such as proximity information, often used by systems in TREC. These techniques may improve results further, but this question will be left for future work. These results provide more evidence that the language modeling approach is a good model for retrieval since a technique that follows from this approach yields routing performance competitive with the best results in the field.
5
DISCUSSION AND FUTURE WORK
The experiments already performed indicate that the language modeling approach to relevance feedback and routing is a reasonable alternative to existing
Language Models for Relevance Feedback
93
methods. In the future, additional empirical study will need to be done to determine practical considerations such as the number of terms to add to a query. In addition, term weighting methods will be incorporated into the relevance feedback method and additional empirical studies will be performed to determine the utility of these techniques in practice. An interesting question that was not addressed in this study is the relative performance of the Rocchio method vs. the language modeling approach as the number of documents increases. Recall that the Rocchio method performed better with a single document fed back, approximately the same with two documents and not as well with ten documents. It can be conjectured that a Bayesian approach where one starts with a biased model that will be overcome by the document estimates as the number of documents increases would provide the best of both worlds. This matter is left for future work.
5.1
QUERY TERM WEIGHTING
Within-query term weighting is used by most modern retrieval systems. The language modeling approach, as currently defined by ?, does not include query term weighting. However, there are extensions that can be made to the model to incorporate query term weighting in a probabilistically justified manner. Two possibilities will be considered here. Risk Functions. The current method of probability estimation uses the maximum likelihood probability and the average probability, combined with a geometric risk function. A simple implementation of term weighting is to modify the risk function. This can be accomplished by means of a Bayesian prior over the term distribution which forces the less important terms to be weighted more heavily by the average probability. This makes the documents less distinguishable from each other based on occurrences of the less important terms. For more important terms, the maximum likelihood term will be allowed to dominate, making the more important terms better able to distinguish between documents. The implementation of this idea would change the ranking formula slightly. For a non-occurring term, the current ranking function estimates the probability as where cft is the raw count of term t in the collection and cs is the the total number of tokens in the collection. The change will be to mix the estimate for non-occurring terms with the mean in the same way the maximum likelihood estimator is currently for terms that do occur. The intuitive meaning of this idea is that the current risk function treats all terms equally, but with prior knowledge (or prior belief) about the relative importance of terms, one can vary the risk of relying on the mean according to the prior belief of the term importance. For example, suppose a term is deemed to be completely useless, the risk function would be modified so that all of the weight is assigned to the mean. The result is that this term is assigned an equal
94
ADVANCES IN INFORMATION RETRIEVAL
probability estimate for every document, causing it to have no effect on the ranking. A stopword such as ‘the’ would fall into this category. One can regard the differences in observed values as pure noise in this case. User Specified Language Models. Currently, queries are treated as a specific type of text produced by the user. One could also allow the user to specify a language model for the generation of query text. The term weights are equivalent to the generation probabilities of the query model. In other words, one would generate query text according to the probabilities and, conceptually, the query is the result of generating a large amount of query text which could then could be processed using the current method. This probably makes more sense in a routing environment where there is enough data from which to estimate the probabilities, but it could conceivably be used for ad hoc retrieval if users have any intuitions about the probabilities.
Acknowledgments This material is based on work supported in part by the National Science Foundation under cooperative agreement EEC-9209623. It is also supported in part by United States Patent and Trademark Office and by the Defense Advanced Research Projects Agency/ITO under ARPA order D468, issued by ESC/AXS contract number F19628-95-C-0235, Any opinions, findings and conclusions or recommendations expressed in this material are the authors and do not necessarily reflect those of the sponsors.
References Beeferman, D., Berger, A., and Lafferty, J. (1997). Text segmentation using exponential models. In Proceedings of Empirical Methods in Natural Language Processing. Ghosh, M. J., Hwang, T., and Tsui, K. W. (1983). Construction of improved estimators in multiparameter estimation for discrete exponential families. Annals of Statistics, 11:351–367. Haines, D. (1996). Adaptive query modification in a probabilistic information retrieval model. PhD thesis, Computer Science Department, University of Massachusetts. Harman, D. (1996). Routing results. In Proceedings of the 4th Text Retrieval Conference (TREC-4), pages A53–A81. Harper, D. J. and van Rijsbergen, C. J. (1978). An Evaluation of Feedback in Document Retrieval Using Co-occurrence Data. Journal of Documentation, 34(3):189–216. onte, J. and Croft, W. (1998). A language modeling approach to information retrieval. In Proceedings of the 21st ACM SIGIR Conference on Research and Development in Information Retrieval, pages 275–28 1.
Language Models for Relevance Feedback
95
Rocchio, J. J. (1971). Relevance Feedback in Information Retrieval, chapter 14, pages 313–323. Prentice-Hall Inc. Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and Hall. Turtle, H. R. (1991). Inference networks for document retrieval. Technical report, University of Massachusetts Ph.D. dissertation. van Rijsbergen, C. J. (1977). A theoretical basis for the use of co-occurrence data in information retrieval. Journal of Documentation, pages 106–119.
Chapter 4 TOPIC DETECTION AND TRACKING: EVENT CLUSTERING AS A BASIS FOR FIRST STORY DETECTION Ron Papka Dataware Technologies 100 Venture Way Hadley, MA 01035
[email protected]
James Allan Center for Intelligent lnformation Retrieval Computer Science Department University of Massachusetts Amherst, MA 01003 allan @cs.umass.edu
Abstract Topic Detection and Tracking (TDT) is a new research area that investigates the organization of information by event rather than by subject. In this paper, we provide an overview of the TDT research program from its inception to the third phrase that is now underway. We also discuss our approach to two of the TDT problems in detail. For event clustering (Detection), we show that classic Information Retrieval clustering techniques can be modified slightly to provide effective solutions. For first story detection, we show that similar methods provide satisfactory results, although substantial work remains. In both cases, we explore solutions that model the temporal relationship between news stories. We also investigate the use of phrase extraction to capture the who, what, when, and where contained in news.
Within Information Retrieval research, texts are usually indexed, retrieved, and organized on the basis of their subjects. Texts are assigned subject headings, a query searches for texts that are “about” a subject, or texts might be grouped
98
ADVANCES IN INFORMATION RETRIEVAL
by the subjects they cover. The research in this work takes a different view, focusing on the events that are described by the text rather than the broader subject it covers. This approach allows us to address questions such as “What is the major event discussed within this story?’ or “Do these texts discuss the same event?” Of course, not all texts can be reduced to a set of events—books on gardening, papers about mathematics, and tracts discussing bridge building do not necessarily provide a narrative of things that happened, but instead convey general information about a subject. This work will necessarily apply only to texts that have an event focus, such as announcements or news broadcasts. The CIIR has participated in a multi-site research effort called Topic Detection and Tracking that has been investigating event-based organization of news stories for the past several years. In this study we present the several-year history of that effort to show the range of problems that is being considered. We then present detailed experimental results showing how the traditional notion of “clusters” in Information Retrieval can be used to address two event-based organization problems: Event Clustering and First Story Detection.
1
TOPIC DETECTION AND TRACKING
The work in this study arose out of and is part of the Topic Detection and Tracking (TDT) initiative. The purpose of that effort is to organize broadcast news stories by the real world events that they discuss. News stories are gathered from several sources in parallel—television, radio, and newswire services are all gathered together to create a single stream of constantly arriving news. The first two types of sources are a particularly important part of the TDT project since they require some mechanism for converting the audio stream into text (the television shows might have closed captioning, but it cannot be guaranteed that they will). Automatic speech recognition (ASR) is therefore a critical component of the entire project. In the autumn of 1999, TDT is in its third phase, called TDT-3. TDT-2 ran throughout 1998 and TDT-1 was a pilot study that ran from mid-1996 through 1997. Each phase of TDT has had a different focus and has helped to develop an improved notion of event-based topics for information organization.
1.1
TDT-1, THE PILOT STUDY
The TDT pilot study was a proof-of-concept effort to explore the extent to which state-of-the-art technologies from Information Retrieval could address organization of information by event-based topics. TDT-1 included three research sites—Carnegie Melon University’s (CMU) Language Technology Institute (LTI), Dragon Systems, and the University of Massachusetts at Amherst’s (UMass) Center for Intelligent Information Retrieval (CIIR)—and participants
Topic Detection and Tracking
99
from the Department of Defense. The study started in mid-1996 and concluded with a final meeting in October of 1997 (Department of Defense, 1997). The first project undertaken by the TDT-1 team was a definition of the problem. An event was defined to be “some unique thing that happens at some point in time.” (Allan et al., 1998a) The properties of time and spatial locality are what distinguish an event from the more general subject. For example, “computer virus detected at British Telecom, March 3, 1993,” is considered an event, whereas “computer virus outbreaks” is the general subject comprising occurrences of this type of event. This definition was never viewed as entirely satisfactory, but was usable as a guideline for building a small pilot corpus to investigate the issue. That is, it could be used to identify most events, even if no one felt that it was a robust definition. The team also defined three research problems to be investigated during the course of the pilot study: 1. Segmentation. This is the problem of breaking a set of contiguous news into discrete stories. Recall that an important component of the TDT problem is dealing with broadcast news sources such as television and radio. Unlike newswire sources, the broadcast information does not have clear boundaries between individual stories—how can we detect a shift from one story to another just by looking at the transcript? 2. Detection. This is the problem of identifying when new topics have appeared in the news stream. There were two versions of the detection problem considered, the endpoints of a spectrum of possibilities. An underlying assumption of these tasks is that no news story discusses more than one topic—an assumption that is known to be false on occasion, but is believed to be usually true. (a) event clustering. The goal here is to consider a complete corpus of news stories and to partition it into clusters, each of which is about a single news topic, and such that no news story is in more than one cluster. (This task was actually referred to as “Retrospective Detection” within the pilot study.) (b) first story detection. The task here is to flag the onset of a previously unseen news topic by marking stories as “new” or “old” as the story arrives on the stream. That is, unlike the retrospective case, a story must be identified as novel before the next story can be considered. (This task was actually called On-Line Detection in the pilot study.) 3. Tracking. In this task, a system is given a small number (Nt = 1, 2, 4, 8, or 16) of sample stories that are known to be on the same news topic. The system’s job is to monitor the stream of stories that follow that Ntth story to find all subsequent stories on the same topic.
100
ADVANCES IN INFORMATION RETRIEVAL
In addition to defining those tasks, the TDT-1 team also cooperated to create a small evaluation corpus to explore the effectiveness of state-of-the-art techniques to address the problems. This corpus was created from 15,683 news stories, roughly balanced between CNN and Reuters news stories, covering July 1, 1994, through June 30, 1995. The stories were extracted and formatted for TDT-1 by the team. To evaluate the problems above using that corpus, the team also generated a set of 25 news topics by selecting “significant” events from that period, supplemented by a small amount of semi-random sampling. This resulted in topics spanning the Oklahoma City bombing, an asteroid’s collision with Jupiter, and the removal of a polyp from Vice President Dan Quayle’s nose. The team employed a two-prong method for assigning relevance judgments between topics and stories. One group of people (split between CMU and Dragon) read every story in the corpus, keeping all 25 topics in mind, and marked whether the story was on topic for each of them. A second group of people (at UMass) used a search engine to look for stories on each of the 25 topics. The two sets of judgments were adjudicated by a team at Dragon, resulting in the final corpus. Using this corpus, the three research groups individually attacked each of the three tasks outlined above. The approaches were varied, though because they were leveraging existing technologies, were similar in some core aspects. Evaluation was on the basis of miss and false alarm rates. Another standard performance measure in TDT is the Detection Error Tradeoff (DET) graph (Martin et al., 1997) that shows the relationship between miss and false alarm rates. It is similar in spirit to the precision and recall graphs that are common in Information Retrieval research. The graphs are similar in nature to Receiver Operating Characteristic (ROC) graphs (Swets, 1998), which have been used to evaluate machine classification (Provost and Fawcett, 1998) and medical diagnostic experiments. For simplicity, in this study we are using F1 and cost measures exclusively. The conclusion of the team was that segmentation was of reasonable quality, tracking effectiveness appeared to be quite high, and both detection problems needed substantial work. In short, the state-of-the-art technologies solved substantial amounts of the problems, but left equally substantial portions to be solved by later research. The TDT-1 pilot study included a small effort toward understanding the impact of speech recognition (ASR) errors on the tasks. Those very small experiments suggested that ASR impacted all of the problems, though tracking the least. The TDT pilot study showed the value of further research on the problem. Its final report appeared at a workshop on broadcast news transcription (Allan et al., 1998a). TDT-2 was initiated a few months later.
Topic Detection and Tracking
1.2
101
TDT-2, A FULL EVALUATION
The primary goal of TDT-2 was to create a full-scale evaluation of the TDT tasks began in the pilot study. TDT-2 ran throughout 1998, and was reported out at the DARPA Broadcast News Workshop (DARPA, 1999). It included almost a dozen participating organizations, including the three pilot sites. The TDT-2 research team opted to redefine the tasks somewhat. The first change was that the two detection tasks were “merged” to create an on-line version of the event clustering task. The TDT-2 Detection task was still to partition the corpus into clusters, with each story in exactly one cluster, and every cluster representing a discrete news topic. However, unlike the TDT-1 event clustering task, the stories arrived in small groups (corresponding roughly to half-hour news broadcasts) that had to be clustered before the next group could be processed. The segmentation and tracking tasks were maintained as defined earlier, though the introduction of stories in small batches was added in those tasks, too. The TDT-2 effort also developed revised evaluation methodologies. Detection Error Tradeoff graphs (Martin et al., 1997) remained a primary investigative tool, but miss and false alarm rates were combined to create a cost measure that was the “official” measure, allowing sites to tune algorithms to find the “best” parameters (for that cost function). One of the most significant changes in TDT-2 was the creation of a largescale corpus. By the standards of classic Information Retrieval research, the roughly 60,000 stories were miniscule. But the effort to gather and annotate the stories was substantial. Stories were gathered from two television sources, two radio sources, and two newswire sources, in roughly equal numbers of stories per source. The corpus covers January through June of 1998. For the audio sources, the information was gathered as an audio recording, as closed captioned transcription, and as recognized speech from a speech recognition system provided by Dragon. Recognizing speech on roughly 36,000 stories (630 hours) of broadcast news was an impressive feat given existing technology, and provided a major limiting factor on the collection size. The Linguistic Data Consortium (LDC) developed the TDT-2 corpora, defining true story segmentation, identifying news topics and their definition, and labeling news stories vis-a-vis topics. Approximately 100 news topics were identified by random selection, their scope was defined using carefully crafted “rules of interpretation,” and they were labeled by teams of trained relevance assessors who read every story for each of the topics. The exhaustive relevance assessments were clearly another limiting factor on the collection size. The TDT-2 research teams were given the first four months of TDT-2 news along with approximately 2/3 of the tagged news topics, to use as training. The
102
ADVANCES IN INFORMATION RETRIEVAL
final two months and corresponding news topics were held out to be used as blind evaluation material at the end of the project. The results of the TDT-2 effort were similar to those of the TDT-1 pilot study. Tracking was deemed to be essentially solved—there remains work to be done, but the accuracy is quite good. The same was said about the segmentation task. Detection, on the other hand, clearly needed more work. The ASR version of the corpus allowed the community to do contrastive studies on ASR and closedcaption versions of the corpus. Tracking and segmentation did not appear to be affected by ASR errors, but detection was. The full evaluation confirmed the trends that the pilot study suggested, and showed that substantial more work could be done. So began TDT-3.
1.3
TDT-3, MULTI-LINGUAL TDT
The most recent portion of the TDT initiative is the TDT-3 effort, running throughout 1999. TDT-3 is different from TDT-2 in the range of event-based organizational tasks considered, the introduction of multi-lingual sources, and a new evaluation corpus. More specifically: 1. TDT-3 includes a group of Chinese newswire and radio sources. The goal is to perform all TDT tasks across the languages, not just within each language. That is, the tracking task might be given Nt = 4 sample training stories in Chinese, and be expected to find all subsequent stories, whether they are in English or Chinese. 2. The entire six months of TDT-2 data is used as the training data for TDT-3 (augmented with some Chinese news stories and topic judgments). The evaluation set is an additional three months of news from the end of 1998, across numerous sources. 3. A new task called Story Link Detection was created, wherein a system is presented with two news stories and is asked to determine whether or not the two stories discuss the same news topic. The motivation for introducing this new task was to provide a base task out of which all the others could be created. 4. The first story detection task (called on-line detection in the pilot study) was reintroduced. The event clustering task (called “detection” in TDT-3) was maintained as before. Preliminary results from TDT-3 suggest that the cross-language component degrades effectiveness somewhat. Effectiveness on the other tasks is similar to the TDT-2 and TDT-1 versions. Results from TDT-3 will be reported in the spring of 2000.
Topic Detection and Tracking
1.4
103
TDT AT THE CIIR
The CIIR has participated in all three phases of TDT. We spent considerable effort on story segmentation in the pilot study (Ponte and Croft, 1997) but did not explore that problem further—although it was not solved, we did not feel that Information Retrieval techniques would be of greater help. Similarly, we have not expended substantial effort on the tracking task. We feel that its similarity to the Information Filtering and Routing problems make it equally likely that progress will come from the thriving activity at TREC (Voorhees and Harman, 1998). The CIIR has instead focused most of its energies on forms of the Detection tasks—specifically first story detection and event clustering. In the remainder of this study, we discuss our efforts to explore the limits of traditional Information Retrieval approaches to solve those problems. We will work within the TDT-1 and TDT-2 frameworks throughout, using the corpora and relevance judgments created for those efforts. Our approach to the problem of first story detection is strongly related to the problem of on-line event clustering. In particular, one approach to first story detection is to cluster the story stream, and to return to the user the story in each cluster with the lowest time-stamp. Assuming this approach is effective, the problem of first story detection becomes an issue of finding a clustering approach that works well in an on-line environment.
2
ON-LINE CLUSTERING ALGORITHMS
As mentioned above, our solution to first story detection is related to the problem of event clustering, or on-line story clustering. Previous clustering work has been done primarily in a retrospective environment where all the stories are available to the process before clustering begins. In this study we focus on an on-line solution to first story detection—i.e., where the stories arrive a few at a time. We also re-evaluate some of the common approaches to retrospective clustering and analyze their effectiveness in an on-line environment. In this context, the goal of clustering—i.e., grouping news stories—is to gather together those stories that discuss the same news topic. Clustering typically involves these steps: converting stories to a vector of weighted features that represents the content of the story, comparing stories and clusters to one another, and applying a threshold to determine if clusters are sufficiently similar that they should be merged. We refer to the combination of vector and threshold as a classifier. An overview of our use of clustering follows (it is explained in more detail below). For each story, we formulate a separate fixed-length classifier from the most frequent words in each story (excluding stopwords). A classifier’s initial threshold is its similarity value when compared to the story from which
104
ADVANCES IN INFORMATION RETRIEVAL
it was created. We assume no subsequent story will exceed this threshold, and so we use it as an initial estimate for the threshold. As new stories arrive on the stream, they are compared to previously formulated classifiers, and clusters are formed based on a comparison strategy. A separate threshold is estimated for each classifier, and stories on the stream that have similarity exceeding the threshold are classified as positive instances of an event, that is, the contents of the story are assumed to discuss the same topic as the story with which the classifier was formulated.
2.1
RELATED CLUSTERING WORK
Several issues pertaining to story clustering have been explored using the agglomerative hierarchical clustering approach (Willett, 1998). Probabilistic approaches to clustering have emerged (van Rijsbergen, 1979; Croft and Harper, 1979; Allan et al., 1998a; Walls et al., 1999), as well as Vector Space Model approaches (Voorhees, 1985; Willett, 1998; Salton, 1989). Most of those algorithms assume that all the stories are available before clustering begins. In an on-line environment, stories are processed sequentially, and each story is either placed into an existing cluster or initiates a new cluster; therefore, many aspects of retrospective approaches are not directly applicable to on-line story processing. In addition, several algorithms require specification of the number of clusters to generate before the clustering process begins (Can and Ozkarahan, 1990). However, these solutions are not applicable to event clustering since we do not know a priori the number of topics that will be encountered during processing. Some work has been done in single-pass clustering where the goal is to build a cluster by making a single pass through the data. In such a setting, the final set of clusters depends greatly upon the order in which the stories are processed, and that has typically been viewed as a disadvantage (van Rijsbergen, 1979). In the TDT setting, however, the order of stories is pre-determined and so its impact on the final clustering is anticipated.
2.2
CREATING STORY VECTORS
Recall that a first stop in clustering is to transform stories into a set ofweighted features. We represent each story by a vector of features weighted by the INQUERY (Broglio et al., 1994) tf.idf function where, and
(4.1)
where t is the number of times the lexical feature appears in the story, and dl is the story’s length in words, and avg_dl is the average number of terms in a story. (Equation 4.1 is variant of the function originally introduced by Robertson et al.,
Topic Detection and Tracking
105
1995.) Also, (4.2) where C is the number of stories in the auxiliary corpus, and df is the number of stories in which the term appears. Both C and df are derived from an auxiliary corpus since they cannot be known in advance in the on-line setting. If the term does not appear in the auxiliary corpus, a default value of 1 is used for df. During processing, a classifier is represented using just the tf information (Equation 4.1). An individual story (when it is compared to existing classifiers) is represented at time j as: dj,k = 0.4 + 0.6 . tfk . idfk
(4.3)
where k is the index of the word co-occurring in the classifier and story, and idf is calculated using Equation 4.2. Since future word occurrence statistics are unknown in real-time applications, the number of stories within which a word will appear is unknown. Several solutions to this problem have been tested by TDT participants—eg., estimating df from an auxiliary corpus (Papka et al., 1999), using the df from the current stream (Schultz and Liberman, 1999), or some combination. We continue to estimate df for Equation 4.2 from an auxiliary corpus in a similar domain. In previous experiments, we evaluated different sources for df and their resulting impact on classification accuracy, and found that idf calculated using story frequencies from TREC volumes 1, 2, and 3, combined with the TREC-4 routing volume (“TREC 123R4”), was a more effective source for df than newsonly stories from TREC 123R4 as well as an independent CNN broadcast news corpus. Also, TREC 123R4 was a more effective source for story frequency (the value of df) than using the entire corpus being processed. In general, the overall changes in effectiveness between using different sources for idf and not using idf at all were small at best. Our experience using incremental idf for tracking in the TDT pilot study (Allan et al., 1998b) suggests that constantly updating story frequency can be a useful statistic, though we are unable to use it consistently well.
2.3
COMPARING CLUSTERS
The methodology for comparing a story to a cluster or the contents of two clusters has the greatest effect on the resulting grouping of stories. For example, in agglomerative hierarchical clustering algorithms three common strategies for combining similarity values are known as single-link, complete-link, and group-average (average-link) clustering (van Rijsbergen, 1979; Salton, 1989). In general, the three strategies lead to different clusterings (Voorhees, 1985).
106
ADVANCES IN INFORMATION RETRIEVAL
Each of those strategies is associated with retrospective clustering algorithms, and in particular with agglomerative hierarchical clustering, where a similarity matrix between all stories is available before processing begins. In the experiments to follow, we test these strategies in the context of on-line clustering. Each strategy, when applied to the on-line environment, results in a different grouping of stories than when the same strategy is applied to a retrospective environment.
Figure 4.1
On-line Cluster Comparison Strategies.
In the on-line clustering environment, each story is analyzed as it arrives and is either placed into an existing cluster or initiates a new cluster—and thus becomes a cluster seed. Figure 4.1 illustrates the differences the three comparison strategies can have on the classification of a story. In the figure, the current story is C, and six stories have been processed and assigned to clusters R, B, and G. The similarities between C and each of the six stories are represented by the numbers in the clusters, ranging from 0.05 to 0.95 (the higher the number, the more similar). Each strategy assigns a different comparison value for C against the three existing clusters, as shown in the bottom of the figure. For single-link, acluster is compared to C using the maximum similarity. For complete link, it is compared using the minimum similarity, and for averagelink, the average of all similarities is used. The boxes at the bottom of the figure provide the comparison values for each strategy and indicate which cluster is chosen for that strategy. This process would then continue with the next arriving story.
Topic Detection and Tracking
107
A similarity value is calculated when comparing a classifier to a story. In the experiments that follow, the similarity between a classifier q formulated at time i when compared to a story arriving on the stream at time j is calculated using the weighted sum operator of INQUERY, which is defined as (4.4) A story is assumed to discuss the topic represented by the classifier if its similarity to the classifier exceeds a certain threshold.
2.4
THRESHOLDS FOR MERGING
In addition to the cluster comparison strategy, a thresholding methodology is needed that affects the decision for generating a new cluster. When we use clustering for first story detection, choosing this threshold is important. From the example in Figure 4.1, choosing a threshold of 0.5 would result in story C initiating a new cluster using the on-line compiete-link strategy. The other strategies result in cluster comparison values that are above this threshold, and thus C is assigned to an existing cluster; however, a high enough threshold (e.g., 0.97) would result in C creating a new cluster for all of the strategies. Time-based thresholds. A side-effect of the timely nature of broadcast news is that stories closer together on the news stream are more likely to discuss related topics than stories farther apart on the stream. When a significant new event occurs, there are usually several stories per day discussing it; over time, coverage of old events is displaced by more recent events. Figure 4.2, for example, depicts the number of stories per day that arrived on CNN broadcast news and AP newswire pertaining to two news topics: an earthquake in Japan and the downing of a US Airforce pilot over Bosnia. Most of the stories pertaining to each of these topics occurred within 10 days, and this appears to be the pattern for many of the topics in the TDT corpora. Therefore, one hypothesis about the domain of broadcast news is that exploiting the time between stories will lead to improved classification accuracy. The threshold model we use is a linear function that controls similarity based on the number of days between the formulation of the classifier and the arrival of the story. During processing, a classifier’s actual threshold is recomputed at each time increment—i.e., each time a new story arrives on the stream. For any classifier formulated at time i, its threshold for a story arriving at a later time j is threshold(qi , dj ) = 0.4 + T (sim(qi , di ) – 0.4) + E (datej - datei ) (4.5) where sim(qi , di ) is the similarity value between theclassifier and the story from which it was formulated (Equation 4.4), and the constant 0.4 is an INQUERY
108
ADVANCES IN INFORMATION RETRIEVAL
Figure 4.2
Daily story counts for two TDT-1 topics.
parameter. The value of (datej – datei) is the number of days between the arrival of story dj and the formulation of classifier qi, and we use the global system parameter E to control the effects of this value. The values for T and E control classification decisions, and our method for finding appropriate settings are discussed below. Decision Scores. The threshold model described above is what decides for each classifier whether a story is a positive instance. Our confidence in the decision that story dj is a positive instance of classifier qi is the extent to which the story exceeds the classifier’s threshold. In what follows, the score we used between a story and a classifier is decision(qi, dj) = sim(qi, dj) – threshold(qi, dj).
(4.6)
A decision scores greater than zero implies that dj is a positive instance of the topic represented by qi, and it also implies that stories di and dj are similar. In many clustering approaches, the similarity between di and dj is symmetric. Since the stories and classifiers have different representations—di is not the same as qi—using our approach to clustering, the similarity between stories is not necessarily symmetric.
3 3.1
EXPERIMENTAL SETTING DATA
One of the main contributions of the TDT research effort is the creation of event-based corpora comprising newswire and broadcast news sources (discussed earlier). Stories were judged relevant to event-based topics using a ternary scale: non-relevant, having content relevant to the event, or containing only a brief mention of the event in a generally non-relevant story. Consistent with TDT methodology, we remove brief mentions from processing and mea-
Topic Detection and Tracking
109
sure classification using the relevant and non-relevant stories only. The news collected for TDT-1 and TDT-2 is divided into four sets, depicted in Table 5.2. Table 4.1
corpus
Dates
TDT- 1 TDT-2 Train TDT-2 Dev TDT-2 Eval
3.2
07/94 01/98 03/98 05/98
-
06/95 02/98 04/98 06/98
TDT corpora and topic information Number of sources stories 2 6 6 6
15863 20404 20462 22443
Words/ story 460 341 314 333
Number of events rels briefs 25 35 25 34
1132 4159 608 1883
250 1103 78 472
EVALUATION MEASURES
Text classification effectiveness is often based on two measures. It is common for Information Retrieval experiments to be evaluated in terms of recall and precision, where recall is the percent of relevant instances classified correctly, and precision is the percent of relevant instances in the set of stories returned to the user. In TDT, system error rates are used to evaluate text classification. These errors are system misses and false alarms, and the accuracy ofa system improves when both types of errors decline. In first story detection, misses occur when the system does not detect a new event, and false alarms occur when the system indicates a story contains a new event when in truth it does not. It is often desirable to have one measure of effectiveness for cross-system comparisons, or to tune a system for maximum effectiveness. Unfortunately, no measure above uniquely determines the overall effectiveness characteristics of a classification system. Several definitions for single-valued measures have been proposed, and many are reviewed by van Rijsbergen, 1979. One prevalent approach is to evaluate text classification using F1-Measure (Lewis and Gale, 1994), which combines recall and precision as 2PR/(P + R). In TDT-2, a cost function was used to analyze detection effectiveness. The general form of the TDT cost function is Cost = costfa * P(fa) * (1 – P(topic)) + costm * P(m) * P(topic)
(4.7)
where P(fa) is the system false alarm rate, P(m) is miss probability, and P(topic) is the prior probability that a story is relevant to a topic. In TDT-2, cost was defined with P(event) = 0.02, and the constants costfa = costm = 1.0. Because only 25–35 topics in each corpus were judged, we use an evaluation methodology for first story detection that expands the number of experimental
110
ADVANCES IN INFORMATION RETRIEVAL
trials (this approach was developed for the TDT pilot study). The methodology uses 11 passes through the stream of stories. Assuming that there are 25 topics available for a particular experiment, the goal of the first pass is to mark the first story of each of the 25 topics as “first” and the rest as “not first.” Systems process all of the unjudged stories, too, but are not evaluated on their effectiveness on those stories. The second pass excludes those 25 first stories and the goal is to detect the new first story (previously the second story) for each of the 25 topics. The process repeats to skip up to 10 stories for each topic. If a topic contains fewer stories than the number of stories to be skipped in the pass, the topic is excluded from evaluation in that pass. Most of our experiments used the holdout method (Cohen, 1995; Kohavi, 1995) for parameter setting. Here, the topics from the TDT-1, TDT-2-Train, and TDT-2-Development corpora are used for training, and testing is done on the topics and stories from the TDT-2-Evaluation corpus. We chose a two-tailed sign test with confidence α = 0.05 to determine significance.
4
EVENT CLUSTERING
In this section, we test our classifier approach for event clustering. We evaluate the on-line versions of single- and average-link strategies. We also test the effectiveness of the time component E > 0) in the threshold model. We evaluate event clustering effectiveness using a standard TDT methodology: we match each truth cluster (topic) with the generated cluster that contains that most matching stories, considering the extraneous stories if necessary. That is, we maximize recall and then use precision within the generated cluster to break ties. The match between truth and generated clusters can then be compared using recall, precision, and F1-Measures. The parameter optimization process involves finding appropriate values for T and E for our threshold model (Equation 4.5). We tested a range of values for each parameter and vector dimensionality over the training topics with known judgments. In the experiments that follow, we used the TDT-1 corpus as well as the automatic speech recognition (ASR) versions of the TDT-2-Train, and TDT-2-Development corpora to find parameters. We evaluated the effectiveness of our systems using the ASR version of the TDT-2 evaluation corpus and its topics. During training, we found that the average-link strategy was comparable in effectiveness to the single-link strategy with the time-based threshold. Both of those strategies were more effective than the single-link strategy without the time factor. No particular classifier dimensionality appears to yield a significant increase in effectiveness. For example, on the TDT-1 corpus using 50-feature classifiers, the on-line average-link approach had an optimal F1 of 0.81, and using 200 features, the on-line single-link+time strategy had optimal F1 of 0.79.
Topic Detection and Tracking
111
These measures are comparable to others reported for on-line clustering on this corpus (Yang et al., 1998; Allan et al., 1998a) The TDT-2 corpora contain news headers and trailers that are not full news stories (e.g., “up next.. . ”). Those short “stories” were omitted from the evaluation—i.e., they were not considered part of the topic, let alone a first story—so it was important that we detect them. We selected the simple strategy of assuming that any story containing fewer than 55 words was a header or trailer, and treating them as if they were not in the collection. (They were assigned a system score indicating non-relevance to all topics.) We selected thresholds for event clustering by choosing the parameter values that gave optimal pooled average F1-measures in the training data. The evaluation was run using those parameters on the TDT-2-Evaluation corpus. It resulted in the values reported in Table 4.2. Optimizing for TDT-2 cost results in similar numbers, though the cost values are slightly lower, not surprisingly. Table 4.2
Event clustering results on TDT-2-Evaluation corpus.
On-line Strategy
F1-Measure SW TW
single-link+time single-link average-link
0.61 0.54 0.58
0.71 0.71 0.68
TDT-2 Cost SW TW 0.0056 0.0077 0.0108
0.0062 0.0072 0.0062
In Table 4.2, pooled or story-weighted (SW) measures, as well as the mean or topic-weighted (TW) measures, are reported for both F1 and TDT-2 cost. The on-line single-link+time approach appears to cluster event-based topics more effectively than the on-line single-link strategy, which suggests that using the time component of the threshold model is effective. In addition, the singlelink+time approach appeared to be the most effective in terms of pooled F1measure and even more so in terms ofpooled TDT-2 cost. However, the averagelink approach resulted in the same topic-weighted cost as the single-link+time approach, which suggests that both methods are effective for event clustering.’ These results are comparable to those obtained by other sites participating in the TDT-2 evaluation (DARPA, 1999). We are interested in how event clustering can help first story detection. In the context of clustering news, the problem of first story detection is to find the first story in each cluster—i.e., the stories that become cluster seeds. Our results suggest that, on average, on-line event clustering puts most of the stories 1In TDT-3, new evaluation measures for the detection task are being considered that may provide more insight into the quality of the resulting clusters. There is concern that the matching process between truth and generated clusters does not appropriately measure the capabilities of a system.
112
ADVANCES IN INFORMATION RETRIEVAL
about a topic in the appropriate cluster. In the next section we evaluate on-line clustering as a basis for first story detection, and determine how well different comparison methodologies produce cluster seeds.
5
FIRST STORY DETECTION
Recall that the problem of first story detection is to identify the stories in a stream of news that contain discussion of a new topic, that is, a topic whose event has not been previously reported. In this section, we present on-line solutions to first story detection in which the system indicates whether the current news story contains or does not contain discussion of a new topic before processing the subsequent story. In what follows, we describe the details of our algorithm and experimental results using the corpora and evaluation methodology developed as part of the TDT initiative. The motivation for our approaches to the problem is to incorporate the salient properties of broadcast news. In particular, we identify the property of time as a distinguishing feature of this domain. We posit that modeling the temporal relationship between stories should result in improved classification. Our event clustering results suggest that this hypothesis is true, and we showed that our approach to event clustering is more effective when this temporal relationship is modeled. Another property of news is that its content includes the names of the people, places, dates, and things, i.e., the who, what, when, and where that are the focus of an topic. Our intuition is that the words in proper noun phrases are important to include in a classifier when using a word co-occurrence model for text classification. We test this intuition by augmenting the classifier formulation process with a natural language parser that finds proper noun phrases in each story. Our results suggest that identifying these phrases leads to improved classification. In what follows, we use the same representation and classification model we used for event clustering. The contents of each story are formulated into a classifier which is compared to subsequent stories. Our approach to first story detection is that if no classifier comparison results in a positive classification decision for the current story, then the current story has content not previously encountered, and thus it contains discussion of a new topic. The main difference between our approach to first story detection and event clustering is that the emphasis is placed on finding the start of each topic, not grouping stories by topics.
5.1
FIRST STORY DETECTION ALGORITHM
We use a single-pass algorithm for first story detection using the representation and model described earlier for event clustering. We represent the content of each story, which we assume discusses some event-based news topic, as a
Topic Detection and Tracking
113
classifier. If any existing classifier results in a positive classification of the current story, the story is assumed to discuss the topic represented by the classifier; otherwise the current story contains a new topic. The approach we use for first story detection does not actually merge the stories in a cluster in any way. In our algorithm, a story contains a new topic if the story does not result in a positive decision score with respect to any of the existing classifiers—where there is one classifier for every earlier story. Our first story detection algorithm is therefore similar to the on-line singlelink+time strategy for event clustering that we evaluated above. The data in those experiments suggested that different comparison strategies result in different groupings of stories. This implies that the number of clusters, and thus the number of new topics identified, is different. If we view the creation of a cluster seed as equivalent to detecting a new topic, we find that on-line single-link strategies tend to return more new topics than on-line average-link strategies at optimal parameter settings, which is advantageous in terms of misses, but not for false alarms. In the sections that follow, we compare event clustering strategies and evaluate their effectiveness as approaches to first story detection.
5.2
FIRST STORY DETECTION EXPERIMENTS
In this section, we evaluate our first story detection system varying the comparison strategies between stories and classifiers, and varying the dimensionality of the classifiers. We ran an analogous set of retrospective experiments to those we ran for event clustering, in which we compared on-line single-link strategies and on-line average-link strategies. Recall that in clustering, comparison values are determined for each cluster using the maximum or average decision scores resulting from the current story being processed. The classifier for the current story becomes a member of the cluster with highest comparison value. If no existing cluster results in a positive comparison value, then the story’s classifier initiates a new cluster and the story is assumed to discuss a new topic. At optimal average effectiveness, first story detection is similar using different numbers of features for classifiers. However, a topic-level comparison across dimensionality revealed that some classifiers benefit from using fewer features while others benefit from more. This analysis also suggested that choosing the appropriate dimensionality for each topic would yield over 50% improvements in F 1 -measure on the TDT-2-Train and TDT-2-Development corpora. We have yet to find a methodology for determining the appropriate dimensionality for an individual topic automatically, and we look forward to studying this problem in our future work. Our training runs suggested that the on-line single-link strategy is better at detecting new event-based topics when E > 0, that is, when the time component of the threshold model is used. The on-line single-link strategies, in general,
114
ADVANCES IN INFORMATION RETRIEVAL
appear to be more effective than the average-link strategy using optimal parameters. However, all the strategies had lower accuracy on the TDT-2-Train corpus. We believe this was caused by the “Monica Lewinsky Scandal” and the “Asian Financial Crisis” topics, two topics that contained heavy news coverage comprising 10% of the stories in the corpus. We evaluated our first story detection system on the ASR version of the TDT2-Evaluation corpus, and the pooled-average results from the 1 1-pass evaluation methodology are listed in Table 4.3. We used the parameters that on average gave rise to optimal effectiveness in the training data, using 50-feature classifiers. We set T = 0.2 and E = 0.0005 when time is factored into the threshold model, and T = 0.3 and E = 0 otherwise. The results from Table 4.3 indicate a 7.1% improvement in F1-measure, and a 15.6% improvement in TDT-2 cost when the temporal relationship between stories is modeled2. This suggests the desirability of using the time component of the threshold model for an actual application. Table 4.3 Comparison of clustering strategies for first story detection (Pooled averages, TDT2-Evaluation corpus, ASR and newswire text). On-line Strategy
Miss Rate
F/A Rate
Recall
Prec
Fl
TDT-2 Cost
single-link+time single-link average-link
66% 63% 73%
1.69% 2.31% 4.22%
34% 37% 27%
26% 22% 10%
0.30 0.28 0.15
0.0297 0.0352 0.0559
We also tested the comparison strategies and parameter settings that gave rise to the optimal event clustering results. In the following experiments we evaluate event clustering as the first story detection task. The threshold parameter for these experiments are those obtained from the optimization process for event clustering using F1 as the target utility measure. In the following experiments, we used T = 0.26 and E = 0.001 for the on-line single-link+time strategy, and T = 0.33 and E = 0 for the on-link single-link strategy. For on-line average-link, 6’ = 0.15. The results for these experiments are listed in Table 4.4. The parameters for the on-line clustering strategies were different, and generally lower for first story detection, and we see an improvement in miss rates, The TDT cost measures in Tables 4.3 through 4.7 are worse than those obtained by a heuristic that decides that no story contains a new topic. For example, this simple approach would yield an F1-measure of 0.0, but a cost of 0.0200 using the TDT-2 cost function parameters applied to Equation 4.7. It should be noted that the cost function parameters were intended for evaluating TDT tracking and detection data, and that different parameters are currently being considered in TDT-3 to address this issue. 2
Topic Detection and Tracking
115
Table 4.4 First story detection using good parameters for event clustering (Pooled averages, TDT-2-Evaluation corpus, ASR and newswire text). On-line Strategy
Miss Rate
F/A Rate
Recall
Prec
FI
TDT-2 Cost
single-link+time single-link average-link
41% 35% 65%
7.50% 10.09% 6.53%
59% 65% 35%
12% 10% 9%
0.20 0.17 0.14
0.0818 0.1059 0.0769
but a significant increase in false alarm rates when applying the lower effective clustering parameters. The improvement in F1 and the decrease in cost resulting from using parameters optimized for first story detection are listed in Table 4.5. Table 4.5 Percent improvement using best first story detection threshold parameters over best event Clustering parameters. On-line Strategy
FI
TDT-2 Cost
single-link+time single-link average-link
50.0% 64.7% 7.1%
-63.7% -66.8% -27.3%
The data in Tables 4.3 to 4.5 suggest that on average, first story detection improved when different threshold parameters were determined explicitly for the task. These data suggest that good clustering strategies do not necessarily lead to effective first story detection: each task focuses places different emphasis on whether or not the first story is identified. Recall that the on-line averagelink strategy was shown to be an effective event clustering strategy, but does not result in finding many new event-based topics. In addition, the on-line single-link strategies need different parameter settings for event clustering and first story detection.
5.3
IMPACT OF ASR TECHNOLOGY
In the following experiments, we evaluate the effects of using text from the automatic speech recognition (ASR) process as compared to human-generated transcripts from closed captioning (CCAP). The ASR process has an expected word error rate of 15%. We substituted the ASR transcriptions with cleaner CCAP transcription. The newswire (NWT) stories remained the same. We used the same parameters for the CCAP+NWT data that were used on the
116
ADVANCES IN INFORMATION RETRIEVAL
ASR+NWT data above. The pooled-average results from the 11-pass evaluation methodology are listed in Table 4.6 below. Table4.6 Comparison of clustering methodologies for first story detection (pooled averages, TDT-2-Evaluation corpus with closed caption source).
Type
Miss Rate
F/A Rate
Recall
Prec
F1
TDT-2 Cost
single-link+tirne single-link average-link
61% 61% 73%
1.53% 1.73% 3.02%
39% 39% 27%
31% 29% 14%
0.35 0.33 0.18
0.0271 0.0291 0.0442
The on-line single-link strategies were more effective than were the online average-link strategy for first story detection for both the CCAP and ASR sources. Furthermore, using the on-link single-link+time strategy gives rise to the most effective pooled-average results for F1-measure and TDT-2 cost. In Table 4.7, we summarize the percent improvements in effectiveness realized by replacing ASR sources with CCAP sources. For both F1-measure and TDT-2 cost, the CCAP sources result in relatively high percent improvements in effectiveness over the ASR sources for both on-line single- and average-link strategies. In general, we would expect the ASR technology to give rise to more out-of-vocabulary (OOV) words than the manual transcription process, and thus we expect the OOV words to be the cause of an increase in classification error. This analysis suggests that the ASR technology hinders new topic classification using our approaches. Table 4.7
5.4
Percent improvement using CCAP vs. ASR transcriptions. On-line Strategy
F1
TDT-2 Cost
single-link+time single-link average-link
16.7% 17.8% 20.0%
-9.4% -17.3% -20.9%
USING PHRASES
A retrieval system that uses single-word features has its drawbacks. For example, a user interested in stories about The World Bank is not necessarily interested in retrieving stories about the merger that created the world’s largest bank. In situations such as that, specifying the query as a phrase would assist in discriminating between relevant and non-relevant stories. Previous research
Topic Detection and Tracking
117
in story classification extends the feature space by extracting natural language phrases and more general multi-word features. The utility of multi-word features and their effects on retrieval have gotten mixed reviews in the previous literature. Lewis has documented some of the earlier work pertaining to representation and ambiguity issues arising from the use of phrases. He shows that “[t]he optimal effectiveness of a text representation based on using simple noun phrases... will be less than that of a word-based representation” (Lewis, 199 1). In addition, TREC routing and filtering systems using multi-word features do not appear to significantly outperform systems that use only single-word features (Robertson et al., 1995). These negative conclusions regarding multi-word features are offset by several positive results that have been reported. For example, Fagan reports 2%22% improvements in average precision using phrasal indexing (Fagan, 1987). Strzalkowski and Carballo, 1996, describe improvements in ad hoc retrieval using natural language phrases. Boolean features comprising multiple words have been used to improve precision by Hearst and Pedersen, 1996. Papka and Allan, 1998, showed that in the context of massive query expansion (queries with several hundred features), retrieval effectiveness improves by using proximity operators on pairs of words. There are several methods for extracting phrases. N-gram models based on mutual information metrics are used to find sets of adjacent words that are likely to co-occur within sentences (Brown et al., 1990). Part-of-speech tagging using pre-specified syntactic templates or more complex parsing (Fagan, 1987; Tzoukermann et al., 1997) gives rise to related multiple words comprising noun, verb, and prepositional phrases. Riloff and Lehnert, 1994, use information extraction techniques that build multi-word features as an integral part of their message understanding system. In what follows we embed a statistical natural language parser into our first story detection system (Charniak, 1996; Charniak, 1999). We use the parser for its ability to detect proper nouns and dates. Our goal was to select features associated with the who, what, where, and when of the story. For example, in the “O.J. Simpson murder trial” topic, we might get the names O. J. Simpson and Nicole Brown Simpson, as well as locations such as Chicago, Brentwood section, as well as dates and the names of objects. We tried a series of experiments on the TDT-1 data where we parsed each story and extracted proper nouns and dates. Our hypothesis is that if proper noun phrases and dates are important lexical features, then increasing their weight in a classifier should result in an effectiveness increase for first story detection, and decreasing their weights should result in an effectiveness decrease. We augmented the classifier formulation process using single-word lexical features. After formulating a classifier for each story, we identified the singleword features that were contained in the proper noun phrases contained in the
118
ADVANCES IN INFORMATION RETRIEVAL
story from which it was formulated. We initially set the feature weight using the usual weight assignment, and then we increased and decreased the feature weights only for words that appeared in the proper noun phrases collected from the parse. For an experiment using the TDT- 1 corpus 3 and 200-feature classifiers, we determined that 47% of the words that were contained in the proper noun phrases extracted from the parse appeared in the classifiers. In addition, 21% of the lexical features in the classifiers mapped to words in the proper noun phrases. In general, we saw that classification improved at false alarm levels below 5% when the weights of the lexical features in the proper noun phrases are doubled. When the weights of these features are halved, classification tends to get worse at false alarm levels below 5%. Unfortunately this improvement does not appear to be stable throughout the entire range of false alarm values. We also observed that effectiveness was reduced when all the words associated with proper noun phrases were removed from the classifier by setting their weights to zero. We did not see that more gains resulted from quadrupling weights. We also tested this approach using 25-feature classifiers, and found no apparent improvement from increasing weights in the features mapping to the words of proper noun phrases. Several feature selection issues associated with natural language phrases became apparent from this experiment. For example, we found that many proper noun phrases were already represented in the classifier. Another problem is that in broadcast news there are many references to news correspondents’ names. Reporters use their own names for communication cues within live coverage to signal transitions, so a reporter’s name that appears often in the text is likely to be selected for the classifier. It could be that a reporter’s name indicates the type of coverage (e.g., Wolf Blitzer often covered politics from the White House) but in general, names of reporters are not good features, and should be removed from a classifier. In addition, there are many dates and proper noun phrases that have no significant bearing on the content of the story, so using all of these features blindly will most likely hurt rather than help effectiveness. Even if these problems were solved, we believe that a more effective use of the parse data needs to be established in order to show significant gains and justify the direct application of this technology to our first story detection system.
3Our attempts to use the parser on the TDT-2 corpora failed. The major problem was that the automatic speech recognition data lacked the punctuation needed for the parser to work. The closed caption data, on the other hand, had too much punctuation. We found many spurious periods that appeared to be used to indicate a pause in the flow of the news story. This additional punctuation also prevented the parser from working.
Topic Detection and Tracking
6
119
DISCUSSION OF FIRST STORY DETECTION
We presented our algorithm for first story detection, and analyzed extensions to our classification model that incorporate the properties of broadcast news. In particular, we showed a method for modeling the temporal relationship between stories, and a method for incorporating proper noun phrases into the classifier formulation process. Our results suggest that these extensions can be modeled efficiently and lead to improved classification accuracy for first story detection. We evaluated the use of on-line clustering as an approach to first story detection, and we found that the on-line single-link+time strategy appeared to be more effective than the other on-line clustering strategies tested. Our analysis suggests that good strategies and parameter settings for event clustering do not necessarily result in effective first story detection. In addition, we showed that the ASR technology negatively affects classification.
6.1
THE GOOD NEWS
The effectiveness of our approach to first story detection is sufficient for certain applications. We find that on the TDT-2 data, we correctly classify between 35% and 60% of the new event-based topics from broadcast news with relatively low false alarm rates. At the lowest reported false alarm rate of 3.2%, our results suggest that a user will need to read only 650 of the 20,000 stories available over a two-month period. This implies that our system can significantly reduce the workload of someone searching for new topics in large amounts of news.
6.2
THE BAD NEWS
Though our approach appears to be comparable to other approaches to first story detection, classification using these methods is far from perfect. From the TDT-2 assessment process, we know that assessors of the same topic were in agreement on 90% of the assessments (Graff, 1999), so we would like to see miss rates of 10% at much lower false alarm levels than ones we measured. A major problem is that even with several extensions to the general word-cooccurrence model, the improvements we are experiencing from the various techniques are incremental. With the exception of modeling the temporal relationship between stories, other approaches, including average-link clustering and additional feature selection, did not appear to have a significant impact on first story detection effectiveness. Another major problem is the limitation of the word-co-occurrence model. When we analyze system misses, which occur when stories containing new topics are labeled as “not new”, we find that at low dimensionality, misses occur because important topic features are not appearing in the classifier, or
120
ADVANCES IN INFORMATION RETRIEVAL
are not sufficiently weighted in the classifier. For example, we found the first story about the “Crash of US Air Flight 427”, resulted in a positive decision score from a classifier formulated for an earlier story about the “Crash of US Air Flight 1016”. The distinguishing lexical feature is the flight number, which does not always have a high term frequency in a relevant story. We had similar problems with other disaster events. For example, the first story discussing the “Oklahoma City bombing” resulted in a positive decision score from a classifier formulated from a story discussing the earlier “World Trade Center bombing”. At higher dimensionality, the two bombing topics were separable, but the airline crashes were not. However, as we mentioned previously, choosing the correct dimensionality automatically for a classifier is a hard problem. Other problems appear to be associated with topics that are heavily covered in the news. These topics often span several hundred days and contain significantly more stories than other event-based topics in the corpora. The more stories discussing a topic, the more its effectiveness is weighted in the pooled measures, and also the greater prior probability that false alarms will occur. For example, errors in the “Monica Lewinsky Case”, which has the most stories of all the topics in the TDT-2 corpora, are a major cause of poor effectiveness on the TDT-2-Train corpus. We also noticed that some domains, such as courtroom coverage, led to errors. For example, using the best parameters on the TDT-1 corpus, the system could not distinguish between stories from the “O. J. Simpson Trial” and stories pertaining to other court cases. Different topics in the same country are also problematic. For example, stories related to various topics in Bosnia caused our system to miss “Carter’s Visit to Bosnia”. These examples indicate that the system was unable to detect certain topics that are discussed in the news at different levels of granularity.
7
CONCLUSION
First story detection is an abstract story classification task that we have shown can be reasonably well solved using a single-pass (on-line) approach. Our algorithm for this problem is based on a notion of event identity. Our approach is to represent the event-based topics discussed in each story with a query and a threshold, which together form a classifier for the content of the story. When a story appearing on the news stream is not classified as a positive instance by any of the existing classifiers, we assume the story discusses a new topic. If the story is classified as a positive instance, then we assume the story is similar in content to a previously processed story, and therefore does not discuss a new topic. Our results indicate that we find between 35%-60% of the stories discussing new topics at relatively low false alarm rates of 1.27%-3.2%.
Topic Detection and Tracking
121
Our approach to first story detection is very similar in nature to on-line clustering algorithms. The step in a clustering algorithm in which the decision is made to assign a story to an existing cluster or to initiate the formation of a new cluster is equivalent to deciding whether a story discusses a new topic. With this in mind, we explored different cluster comparison strategies which included on-line single-, average-, and complete-link strategies. Each of these strategies gives rise to a different on-line clustering of the stories, and thus a different set of new topics. We found that our implementation of complete-link did not work well for either first story detection or event clustering. While the average-link approach worked well for clustering, but not for first story detection, the on-line single-link+time strategy worked best for both tasks. The motivation for exploiting the property of time was based on an analysis of the TDT corpora which suggested that stories closer together on the stream were more likely to discuss the same topic than stories further apart. Time was incorporated as a component of our threshold model, which resulted in improved effectiveness for the on-line single-link strategy when applied to first story detection and event clustering. In addition, we explored a process that extracts the lexical features that capture the who, what, when, and where contained in news. The process involved a natural language parser that was used to extract proper noun phrases and dates from the stories in the news stream. Experiments with the parser on the TDT-1 corpus indicated that many of the words in the proper noun phrases were already in the classifiers that were being formulated. When we increased the weights of the classifier features that appeared in proper noun phrases and dates, classification accuracy improved, and when we decreased their weights, classification accuracy declined. This suggests that identifying proper noun phrases and dates can help first story detection. The automatic speech transcription (ASR) process impacted the effectiveness of our approach to first story detection and event clustering. We found that effectiveness improved between 9.4% to 20.9% for both single- and averagelink approaches when the ASR data was replaced by CCAP transcriptions. The impact of the ASR technology was minimal for the TDT Tracking task. The majority of our experimental runs for all three problems involved an exhaustive search for system parameters, including classifier dimensionality and the parameters for our threshold model. Our current solution for first story detection uses constant dimensionality and two global threshold parameters. Our experiments indicated that a good set of values existed for these parameters in 80% of the topics tested. In addition, we found that if it had been possible to determine optimal dimensionality automatically, we would have realized an over 50% effectiveness improvement in the classification of new topics. Unlike ranked retrieval, true text classification tasks require that a system make hard decisions about a story’s relevance. Therefore, threshold parameter
122
ADVANCES IN INFORMATION RETRIEVAL
estimation becomes an integral part of the classification approach. We found that the threshold parameters that optimized pooled average effectiveness measure were similar across the TDT corpora, and thus an averaging of optimal parameters resulted in comparably effective classification relative to other systems evaluated at TDT-2. However, when we compared optimal threshold parameters that worked well for event clustering against those we determined for first story detection, we found that good parameters for first story detection were consistently lower than good parameters for event clustering. We therefore believe that a good clustering approach does not necessarily lead to a good solution for first story detection (at least, when “good” is measured using TDT’s cost functions). Additional evidence that suggests this hypothesis is true was our application of on-line average-link clustering, which was effective for event clustering, but not for first story detection. We presented several experiments for the event clustering problem, where we compared single-pass clustering solutions that included on-line single-link and average-link cluster comparison strategies. The data suggest that augmenting on-line single-link clustering with a time component was the most effective approach when using automatic and manual transcriptions for broadcast news sources. Other clustering experiments suggested that average-link is comparable to single-link+time when audio sources for news are manually transcribed. However, our single-link+time approach resulted in the lowest story-weighted cost realized by the systems evaluated at TDT-2, and we suggest using singlelink+time for on-line clustering of broadcast news because it is both efficient and effective. In retrospect, the comparisons between our approaches and those of other TDT participants indicate that different views of the tasks lead to different retrieval models which however result in similar effectiveness. Our systems, in general, were comparable in effectiveness to the best systems for each of the problems, and we attribute this to our efforts in parameter estimation. In addition to similar overall effectiveness, the common element among the systems is the underlying model of word-co-occurrence used to determine when two stories discuss the same topic. We believe this model is the key to the event-based topic classification solutions, and that improvements in effectiveness will come more from modeling the properties of topics than from modifying existing retrieval models.
8
FUTURE WORK
Our plan for future work involves improving the feature selection and extraction methodologies of our approach to first story detection. In several experiments, we found that classifiers did not contain all the features that distinguish an event-based topic from its more general subject-based topic. For example,
Topic Detection and Tracking
123
distinguishing lexical features such as a flight number and accident location were not necessarily included in each classifier formulated from a plane crash story. Other problems included over-weighting a feature, such as Bosnia, in classifiers that were used to track topics about more specific news coverage. We plan to extend our classifier representation with a model that distinguishes event-level features from subject-level features, perhaps by explicitly incorporating the “rules of interpretation” that define topic scope. Other aspects of feature selection to develop include finding a more effective use of the proper noun phrase data, which should lead to additional effectiveness gains and justify the direct application of natural language parsing technology to our first story detection system. Our results suggest that cleaner data improves classification effectiveness. In the first story detection and event clustering experiments, we saw improvements in all cluster comparison strategies using the cleaner closed caption data. In addition, removing broadcast news header and trailer snippets resulted in improved effectiveness for both problems. There would appear to be several applications for text pre-processing steps that may result in further improvements to classification accuracy, such as removing reporters’ names and other text that is specific to the broadcast news program. Another example is a processing step that would resolve some of the word errors resulting from the automatic speech recognition process. There appear to be several applications for event-based classification, and the interest in Topic Detection and Tracking is increasing with each phase of TDT. We expect several new approaches to the problems we discussed here to emerge from this research effort, and we look forward to future TDT comparisons, which may help us understand how improvements in text classification can be made.
Acknowledgments We thank Victor Lavrenko and Daniella Malin for their help with some of the experiments discussed in this work. This material is based on work supported in part by the National Science Foundation, Library of Congress and Department of Commerce under cooperative agreement number EEC9209623, the Air Force Office of Scientific Research under grant number F49620-99-1-0138, and SPAWARSYSCEN-SDunder grant number N66001-99- 1-8912. Any opinions, findings and conclusions or recommendations expressed in this material are the authors’ and do not necessarily reflect those of the sponsor.
References Allan, J., Carbonell, J., Doddington, G., Yamron, J., and Yang, Y. (1998a). Topic detection and tracking pilot study: Final report. In Proceedings of DARPA
124
ADVANCES IN INFORMATION RETRIEVAL
Broadcast News Transcription and Understanding Workshop, pages 194– 218. Allan, J., Papka, R., and Lavrenko, V. (1998b). On-line new event detection and tracking. In Proceedings of ACM SIGIR, pages 37–45. Brown, P., Cocke, J., Pietra, S. D., Pietra, V. D., Jelinek, E, Lafferty, J., Mercer, R., and Roossin, P. (1990). A statistical approach to machine translation. Computational Linguistics, 16(2):79–85. Callan, J., Croft, B., and Broglio, J. (1994). TREC and TIPSTER experiments with INQUERY. Information Processing and Management, 31(3):327–343. Can, F. and Ozkarahan, E. (1990). Concepts and effectiveness of the covercoefficient-based clustering methodology for text databases. ACM Transactions on Database Systems, 15(4):483–517. Charniak, E. (1996). Tree-bank grammars. Technical Report CS-96-02, Department of Computer Science, Brown University. Charniak, E. (1999). Personal communication. Cohen, P. (1995). Empirical Methods for Artifcial Intelligence. The MIT Press, Cambridge, Massachusetts. Croft, W. and Harper, D. (1979). Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 5(3):285– 295. DARPA, editor (1999). Proceedings of the DARPA Broadcast news Workshop, Herndon, Virginia. Department of Defense (1997). Proceedings of the TDT workshop. University of Maryland, College Park, MD (unpublished). Fagan, J. (1987). A Comparison of Syntactic and Non-Syntactic Methods. PhD thesis, Department of Computer Science, Cornell University. Graff, D. (1999). Personal communication. Hearst, M. and Pedersen, J. (1996). Reexamining the cluster hypothesis: Scatter/ gather on retrieval results. In Proceedings of ACM SIGIR, pages 76–84. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of International Joint Conference on Artifcial Intelligence, pages 1137–1443. Lewis, D. (1991). Representations and Learning in Information Retrieval. PhD thesis, Department of Computer and Information Science, University of Massachussetts. Lewis, D. and Gale, W. (1994). A sequential algorithm for training text classifiers. In Proceedings of ACM SIGIR, pages 3–13. Martin, A., Doddington, G., Kamm, T., Ordowski, M., and Przybocki, M. (1997). The DET curve in assessment of detection task performance. In Proceedings of EuroSpeech'97, pages 1895–1898.
Topic Detection and Tracking
125
Papka, R. and Allan, J. (1998). Document classificiation using multiword features. In Proceedings ofACM International Conference on Information and Knowledge Management, pages 124–131. Papka, R., Allan, J., and Lavrenko, V. (1999). UMass approaches to detection and tracking at TDT2. In Proceedings of the DARPA Broadcast News Workshop,pages 111–116. Ponte, J. and Croft, W. (1997). Text segmentation by topic. In Proceedings of the First European Conference on Research and Advanced Technology for Digital Libraries, pages 113–125. Provost, F. and Fawcett, T. (1998). Robust classification systems for imprecise environments. In Proceedings of the Fifth National Conference on Artificial Intelligence (AAAI98), pages 3–13. Riloff, E. and Lehnert, W. (1994). Information extraction as a basis for highprecision text classification. ACM Transactions on Information Systems, 12(3):296–333. Robertson, S., Walker, W., Jones, S., Hancock-Beaulieu, M., and M.Gatford (1995). Okapi at TREC-3. In Proceedings of TREC-3, pages 109–126. Salton, G. (1989). Automatic Text Processing. Addison-Wesley Publishing Co., Reading, MA. Schultz, J. and Liberman, M. (1999). Topic detection and tracking using idfweighted cosine coefficient. In Proceedings of the DARPA Broadcast News Workshop,pages 189–192. Strzalkowski, T. and Carballo, J. P. (1996). N