Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
6107
Hamish Cunningham Allan Hanbury Stefan Rüger (Eds.)
Advances in Multidisciplinary Retrieval First Information Retrieval Facility Conference, IRFC 2010 Vienna, Austria, May 31, 2010 Proceedings
13
Volume Editors Hamish Cunningham University of Sheffield Dept. of Computer Science Sheffield S1 4DP, UK E-mail:
[email protected] Allan Hanbury Information Retrieval Facility 1040 Vienna, Austria E-mail:
[email protected] Stefan Rüger Knowledge Media Institute The Open University Milton Keynes, MK7 6AA, UK E-mail:
[email protected]
Library of Congress Control Number: 2010926587 CR Subject Classification (1998): H.3, I.2.4, H.5, H.4, I.2, C.2 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13
0302-9743 3-642-13083-6 Springer Berlin Heidelberg New York 978-3-642-13083-0 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180
Preface
These proceedings contain the refereed papers and posters presented at the first Information Retrieval Facility Conference (IRFC), which was held in Vienna on 31 May 2010. The conference provides a multi-disciplinary, scientific forum that aims to bring young researchers into contact with industry at an early stage. IRFC 2010 received 20 high-quality submissions, of which 11 were accepted and appear here. The decision whether a paper was presented orally or as poster was solely based on what we thought was the most suitable form of communication, considering we had only a single day for the event. In particular, the form of presentation bears no relation to the quality of the accepted papers, all of which were thoroughly peer reviewed and had to be endorsed by at least three independent reviewers. The Information Retrieval Facility (IRF) is an open IR research institution, managed by a scientific board drawn from a panel of international experts in the field whose role is to promote the highest quality in the research supported by the facility. As a non-profit research institution, the IRF provides services to IR science in the form of a reference laboratory, hardware and software infrastructure. Committed to Open Science concepts, the IRF promotes publication of recent scientific results and newly developed methods, both in traditional paper form and as data sets freely available to IRF members. Such transparency ensures objective evaluation and comparability of results and consequently diversity and sustainability of their further development. The IRF is unique in providing a powerful supercomputing infrastructure that is exclusively dedicated to semantic processing of text. It has at its heart a huge collection of patent documents representing the global archive of ideas and inventions. This data is housed in an environment that allows large-scale scientific experiments on ways to manage and retrieve this knowledge. This collection of real data allows IR researchers from all over the world to experiment for the first time on a realistic data corpus. The quality of search results is reviewed and evaluated in many fields but especially those specialising in patent search. IRF conferences wish to resonate in particular with young researchers, who are interested in discussing results obtained using the IRF infrastructure and data resources; learning about complementary technologies; applying their research efforts to real business needs; and joining the international research network of the IRF. The first IRFC aimed to tackle four complementary research areas: – – – –
information retrieval semantic web technologies for IR natural language processing for IR large-scale or distributed computing for the above areas
We believe that this first conference has achieved most of these aims and we look forward to many more instances of the IRFC.
VI
Preface
Acknowledgements. It is never easy to make a conference happen. Our sincere thanks go out to: – the IRF executive board: Francisco Eduardo De Sousa Webber, Daniel Schreiber and Sylvia Thal, for their inspiration, for getting the ball rolling and for exceptional organisational talent – the professional team at the IRF and Matrixware for their help in preparing the conference and this volume: Marie-Pierre Garnier; Katja Mayer; Mihai Lupu; Giovanna Roda; Helmut Berger – the IRF scientific board and John Tait (IRF CSO) for their guidance – Niraj Aswani for his help in preparing the proceedings – the conference programme committee for their hard work reviewing and commenting on the papers: • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
Yannis Avrithis, CERTH, Greece Leif Azzopardi, University of Glasgow, UK Ricardo Baeza-Yates, Yahoo! Research, Spain Jamie Callan, Carnegie Mellon University, USA Paul Clough, University of Sheffield, UK W. Bruce Croft, University of Massachusetts, USA Norbert Fuhr, University Duisburg-Essen, Germany Wilfried Gansterer, University of Vienna, Austria Charles J. Gillan, Queen’s University Belfast, UK Gregory Grefenstette, Exalead, France Preben Hansen, Swedish Institute of Computer Science, Sweden David Hawking, Funnelback Internet and Enterprise Search, Australia Joemon Jose, University of Glasgow, UK Noriko Kando, National Institute of Informatics (NII), Japan Philipp Koehn, University of Edinburgh, UK Wessel Kraaij, TNO, The Netherlands Udo Kruschwitz, University of Essex, UK Dominique Maret, Matrixware Information Services, Austria Marie-Francine Moens, Catholic University of Leuven, Belgium Henning M¨ uller, University of Applied Sciences Western Switzerland, Switzerland Walid Najjar, University of California Riverside, USA Arcot Desai Narasimhalu, Singapore Management University, Singapore Fredrik Olsson, Swedish Institute of Computer Science, Sweden Miles Osborne, University of Edinburgh, UK Andreas Rauber, Vienna University of Technology, Austria Magnus Sahlgren, Swedish Institute of Computer Science, Sweden Mark Sanderson, University of Sheffield, UK Frank J. Seinstra, Vrije Universiteit, The Netherlands John Tait, IRF, Austria Benjamin T’sou, City University of Hong Kong, China Christa Womser-Hacker, University of Hildesheim, Germany
Preface
VII
– the keynote speakers Mark Sanderson, University of Sheffield, UK, and David Hawking, Funnelback Internet and Enterprise Search, Australia, for providing excellent and inspiring talks – the sponsors: • Matrixware Information Services • ESTeam • The University of Sheffield • STI International • Yahoo! Labs – the Austrian Federal Ministry of Science and Research and Wien Kultur for their support – BCS — The Chartered Institute for IT for endorsing the conference – Matt Petrillo for understanding what Hamish was talking about (at least some of the time) We hope you enjoy the results! May 2010
Hamish Cunningham, University of Sheffield http://www.dcs.shef.ac.uk/˜hamish General Chair Allan Hanbury, Information Retrieval Facility http://www.ir-facility.org/about/people/staff Publications Chair Stefan R¨ uger, The Open University http://people.kmi.open.ac.uk/stefan Programme Chair
VIII
Preface
Supported by
Endorsed by
Table of Contents
Scaling Up High-Value Retrieval to Medium-Volume Data . . . . . . . . . . . . . Hamish Cunningham, Allan Hanbury, and Stefan R¨ uger
1
Sentence-level Attachment Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M-Dyaa Albakour, Udo Kruschwitz, and Simon Lucas
6
Rank by Readability: Document Weighting for Information Retrieval . . . Neil Newbold, Harry McLaughlin, and Lee Gillam
20
Knowledge Modeling in Prior Art Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erik Graf, Ingo Frommholz, Mounia Lalmas, and Keith van Rijsbergen
31
Combining Wikipedia-Based Concept Models for Cross-Language Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Benjamin Roth and Dietrich Klakow
47
Exploring Contextual Models in Chemical Patent Search . . . . . . . . . . . . . . Jay Urbain and Ophir Frieder
60
Measuring the Variability in Effectiveness of a Retrieval System . . . . . . . . Mehdi Hosseini, Ingemar J. Cox, Natasa Millic-Frayling, and Vishwa Vinay
70
An Information Retrieval Model Based on Discrete Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alberto Costa and Massimo Melucci Logic-Based Retrieval: Technology for Content-Oriented and Analytical Querying of Patent Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Iraklis Angelos Klampanos, Hengzhi Wu, Thomas Roelleke, and Hany Azzam Automatic Extraction and Resolution of Bibliographical References in Patent Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Patrice Lopez An Investigation of Quantum Interference in Information Retrieval . . . . . Massimo Melucci Abstracts versus Full Texts and Patents: A Quantitative Analysis of Biomedical Entities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bernd M¨ uller, Roman Klinger, Harsha Gurulingappa, Heinz-Theodor Mevissen, Martin Hofmann-Apitius, Juliane Fluck, and Christoph M. Friedrich Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
84
100
120 136
152
167
Scaling Up High-Value Retrieval to Medium-Volume Data Hamish Cunningham1 , Allan Hanbury2 , and Stefan R¨ uger3 1
Department of Computer Science, University of Sheffield Regent Court, 211 Portobello Street, Sheffield S1 4DP, UK
[email protected] 2 Information Retrieval Facility Operngasse 20b, 1040 Vienna, Austria
[email protected] 3 Knowledge Media Institute, The Open University Walton Hall, Milton Keynes MK7 6AA, UK
[email protected]
Abstract. We summarise the scientific work presented at the first Information Retrieval Facility Conference [3] and argue that high-value retrieval with medium-volume data, exemplified by patent search, is a thriving topic in a multidisciplinary area that sits between Information Retrieval, Natural Language Processing and Semantic Web Technologies. We analyse the parameters that condition choices of retrieval technology for different sizes and values of document space, and we present the patent document space and some of its characteristics for retrieval work.
1
How Much Search Can You Afford?
Information retrieval (IR) technology has proliferated in rough proportion to the expansion of knowledge and information as a central factor in economic success. How should end-users choose between the many available options? Three main dimensions condition the choice: – Volume. The GYM big three web search engines (Google, Yahoo, Microsoft) deliver sub-second responses to hundreds of millions of queries daily over hundreds of terabytes of data. At the other end of the scale, desktop search systems can rely on substantial computing resources relative to a small data set. – Value. The retrieval of high-value content (typically within corporate intranets or behind pay-for-use turnstiles) is often mission-critical for the business that owns the content. For example, the BBC allocates a skilled staff member for eight hours per broadcast hour to index their most important content. – Cost. Semantic indexing, conceptual search, linked data and so on share with controlled-vocabulary and meta-data systems a higher cost of H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 1–5, 2010. c Springer-Verlag Berlin Heidelberg 2010
2
H. Cunningham, A. Hanbury, and S. R¨ uger
implementation and maintenance than systems based on counting keywords and hyperlinks. To process web-scale volumes, GYM use a combination of one of the oldest and simplest retrieval data structures (an inverted file that relates search terms to documents) and a ranking algorithm that utilises the link structure of the web. These techniques work much better than was initially expected, profiting from the vast number of human relevance decisions that are encapsulated in hyperlinks. Problems remain of course: first, there is still much data in which links are not present, and second the familiar problems of ambiguity (index term synonymy and query term polysemy) can lead to retrieval of irrelevant information and/or failure to retrieve relevant information. High-value (or low-volume) content retrieval systems address these problems with a variety of semantics-based approaches that attempt to perform conceptual indexing and logical querying. For example, the BBC system cited above indexes using a thesaurus of 100,000 terms that generalise over anticipated search terms. Similarly in the Life Sciences, publication databases increasingly use rich terminological resources to support conceptual navigation (MeSH, the Gene Ontology, Snomed, the unified UMLS system etc). An important research theme in recent years has been to ask to what degree can we have our cake and eat it? In other words, how far can the lowvolume/high-value methods be extended? One area of new high-value methods is being driven from Natural Language Processing. In the proceedings of this IRFC 2010 conference, Roth and Klakow [12] combine Wikipedia-based concept models for cross-language retrieval. They employ Wikipedia as a low-cost resource that is up-to-date and find that standard Latent Dirichlet Allocation can extract cross-language information that is valuable for IR by simply normalising the training data. Albakour et al [1] look at sentence-level analysis of e-mails with a view to predict whether the e-mail should contain an attachment. This is an example for a problem where a finergrained sentence-level approach is preferable to the common IR approach of a bag-of-words of the entire document. Evaluation measures in IR certainly have to adapt to new models and modus operandi. One example is the Newbold et al [11] novel ranking-by-readability approach, which considers the reading ability (and motivation) of the user. Hosseini et al [6] critique the common approach of comparing retrieval systems by their mean performance. They argue that the averaging process of the typical mean average precision measure does not capture all the important aspects of effectiveness. Instead they explore how the variance of a metric can be used to provide a more informative measure of system effectiveness than mean average precision. Other contributions to potential future high-value methods come from the forefront of theory: Melucci [9] looks at van Rijsbergen’s quantum mechanics framework as a new model for IR. He first discusses a situation in which that framework, and then quantum probability, can be necessary in IR and then describes the experiments designed to this end. Costa and Melucci [2] present a new framework based on the Discrete Fourier Transform (DFT) for IR. Their
Scaling Up High-Value Retrieval to Medium-Volume Data
3
model represents a query term as a sine curve and a query as the sum of sine curves, which is then transformed from the time domain to the frequency domain through DFT. Each document of the collection corresponds to a set of filters in this interpretation, and the retrieval operation corresponds to filtering the result of the query DFT (aka as spectrum). Costa and Melucci’s initial small-scale experiments with spectral methods indicate a competitive retrieval performance. The science and technology reported in this volume also concerns pushing the boundary of scale for the low-volume methods in the general context of patent documentation, to which we now turn.
2
Patent Retrieval
The number and economic significance of patent search professionals (and others who must search patent data) has grown rapidly in recent decades. This growth shows no signs of slowing. There are several schools of thought as to whether this is a good thing, but even in the negative camp many would agree that the load involved in searching prior art is a waste of economic resources. Further, techniques like deliberate obfuscation, defensive patenting and cross-licencing all add to the burden on the most dynamic sectors of industry, and this is especially problematic for small to medium enterprises. Several other factors have increased the value of patents and the criticality of patent searching: 1. Patent applications have been rising at around 7% yearly for more than a decade (plus large increases in the abbreviated Patent Cooperation Treaty process applications) 2. Globalisation, specialisation, the fashion for startups in high-tech, and crosslicencing as a corporate defence mechanism 3. Stronger enforcement In parallel, the publication space of prior art has mushroomed in volume (to take only one of many possible examples, Medline was “growing at the rate of more than 400,000 new entries per year” at the start of the last decade1 . In sum, patents have become “the currency of the knowledge economy” [5]. How do patents differ from Web documents? They are harder in some respects, easier in others. Patent documents are harder because patent searchers require very high recall (web searchers want high precision) and because patents don’t contain hyperlinks (though they do contain references). On the other hand, patent documents are easier because of the lower volume, their more regular structure and because there there are some sub-language effects (turns of phrase and terminology that are particular to the genre and which can aid automated processing). The average value of patent documents is also much higher than the average value of web pages. All in all, patent search is a challenging task for IR researchers, and the strength and diversity of the work reported at IRFC 2010 is testament to the 1
http://www.medscape.com/viewarticle/440693 3
4
H. Cunningham, A. Hanbury, and S. R¨ uger
energy with which the scientific community is addressing this area: Graf et al [4] explore the benefits of integrating knowledge representations in prior art patent retrieval, where they utilise human judgement available in the form of classifications assigned to patent documents. Klampanos et al [7] apply logic-based retrieval to patent searching. They see an initial document search only as the starting point of a chain of searches and decisions that need to be made by patent searchers. They argue that keyword-based retrieval is not suitable for modelling comprehensive retrieval strategies while database-like and logical approaches are better suited to model strategies, reasoning and decision making. Lopez [8] describes experiments with Conditional Random Fields (CRF) for extracting bibliographical references in patent documents. M¨ uller et al [10] contrast abstracts versus full texts and patents in a quantitative analysis of biomedical entities. They research and uncover the density and variety of relevant life science terminologies in Medline abstracts, PubMedCentral journal articles and patents from the TREC Chemistry Track. In yet another patent area Urbain and Frieder [13] explore the development of probabilistic retrieval models for integrating term statistics with entity search using multiple levels of document context to improve the performance of chemical patent search. In the way of conclusion, we believe that the IRFC 2010 conference has demonstrated how cutting-edge-research continues to make inroads in the effort to scale up high-value retrieval to medium-volume repositories.
References 1. Albakour, M.D., Kruschwitz, U., Lucas, S.: Sentence-level attachment prediction. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 6–19. Springer, Heidelberg (2010) 2. Costa, A., Melucci, M.: An information retrieval model based on Discrete Fourier Transform. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 88–103. Springer, Heidelberg (2010) 3. Cunningham, H., Hanbury, A., R¨ uger, S. (eds.): IRFC 2010. LNCS, vol. 6107. Springer, Heidelberg (2010) 4. Graf, E., Frommholz, I., Lalmas, M., van Rijsbergen, K.: Knowledge modeling in prior art search. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 32–47. Springer, Heidelberg (2010) 5. Guellec, D., van Pottelsberghe de la Potterie, B.: The Economics of the European Patent System — IP Policy for Innovation and Competition. Oxford University Press, Oxford (2007) 6. Hosseini, M., Cox, I.J., Millic-Frayling, N., Vinay, V.: Measuring performance volatility of retrieval systems. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 72–87. Springer, Heidelberg (2010) 7. Klampanos, I.A., Wu, H., Roelleke, T., Azzam, H.: Logic-based retrieval: Technology for content-oriented and analytical querying of patent data. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 104–118. Springer, Heidelberg (2010) 8. Lopez, P.: Automatic extraction and resolution of bibliographical references in patent documents. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 119–134. Springer, Heidelberg (2010)
Scaling Up High-Value Retrieval to Medium-Volume Data
5
9. Melucci, M.: An investigation of quantum interference in information retrieval. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 135–150. Springer, Heidelberg (2010) 10. M¨ uller, B., Klinger, R., Gurulingappa, H., Mevissen, H.T., Hofmann-Apitius, M., Fluck, J., Friedrich, C.: Abstracts versus full texts and patents: A quantitative analysis of biomedical entities. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 151–164. Springer, Heidelberg (2010) 11. Newbold, N., McLaughlin, H., Gillam, L.: Rank by readability: Document weighting for information retrieval. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 20–31. Springer, Heidelberg (2010) 12. Roth, B., Klakow, D.: Combining Wikipedia-based concept models for crosslanguage retrieval. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 48–61. Springer, Heidelberg (2010) 13. Urbain, J., Frieder, O.: Exploring contextual models for chemical patent search. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 62–71. Springer, Heidelberg (2010)
Sentence-Level Attachment Prediction M-Dyaa Albakour1,2, Udo Kruschwitz1 , and Simon Lucas1 1
School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe Park, Colchester, CO4 3SQ, UK {malbak,udo,sml}@essex.ac.uk 2 Active Web Solutions Ltd., Broomvale Business Centre, Bramford Road, Ipswich, IP8 4JU, UK
[email protected]
Abstract. Attachment prediction is the task of automatically identifying email messages that should contain an attachment. This can be useful to tackle the problem of sending out emails but forgetting to include the relevant attachment (something that happens all too often). A common Information Retrieval (IR) approach in analyzing documents such as emails is to treat the entire document as a bag of words. Here we propose a finer-grained analysis to address the problem. We aim at identifying individual sentences within an email that refer to an attachment. If we detect any such sentence, we predict that the email should have an attachment. Using part of the Enron corpus for evaluation we find that our finer-grained approach outperforms previously reported documentlevel attachment prediction in similar evaluation settings. A second contribution this paper makes is to give another successful example of the ‘wisdom of the crowd’ when collecting annotations needed to train the attachment prediction algorithm. The aggregated non-expert judgements collected on Amazon’s Mechanical Turk can be used as a substitute for much more costly expert judgements.
1
Introduction
Email has certainly become one of the most effective communication tools in today’s organisations. At the same time email has also become one the most challenging digital materials to manage and studies show that current email systems do not fully satisfy users in managing their growing mail boxes, a phenomenon known as email overload [17], [7]. One of the common pitfalls when sending emails is to forget to attach a document resulting in a sequence of emails between the sender and the recipients. We are interested in solving this problem by trying to predict a necessity for an attachment so that the user is notified before sending the message. Previous work using bag-of-word approaches has shown promising results on building such a system by analysing the content of the email message and the user profile [5], [6]. Here we explore performing a finer-grained analysis by looking at the sentences that comprise the email body rather that the whole email. We aim to identify H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 6–19, 2010. c Springer-Verlag Berlin Heidelberg 2010
Sentence-Level Attachment Prediction
7
sentences that refer implicitly or explicitly to a document attached to the email message. This will enable us to know more about the attachment which might further be useful for suggesting documents to the user. Finding out what sentence in an email relates to an attachment can also be useful in other contexts, e.g. when extracting knowledge from emails and associating extracted facts with the appropriate attachments [1]. Any supervised learning approach requires annotated data. For our purpose we need a realistic set of emails that are annotated to reflect which sentence within each email refers to an attachment. Collecting such annotations can be a costly exercise. Therefore we suggest using Amazon’s Mechanical Turk1 to collect annotations cheaply and quickly from paid non-experts. We will of course have to show that we can get high quality annotations. In summary, this works tries to answer the following main research question: Can sentence-level prediction of attachments beat state-of-the-art methods that are based on a more coarse-grained document-level approach? Beyond this main question we will try to answer two further research questions: – Are non-experts annotations good enough in the task of identifying sentences that refer to attachments in email messages? – Can they substitute expert annotations and therefore be used as a gold standard? The paper is organised as follows, in Section 2 we look at related work. Sections 3 and 4 describe the dataset used for the experiments. Section 5 explains the task of collecting non-expert annotations through Mechanical Turk while Section 6 explains how we developed a sentence classifier and used it to predict emails with attachments. Discussion and future work are presented in Section 7.
2
Related Work
2.1
Attachment Prediction
Attachment prediction in emails is the task of identifying whether an email message contains any files attached to it. It can be useful for notifying users that an email message is missing an attachment before hitting the send button. Commercial applications are available as plugins for common Email Clients such as MS Outlook and Mozilla Thunderbird. For example Forgotten Attachment Detector2 allows users to specify certain phrases that can be looked at to alert them before sending an email that an attachment file is missing. Such systems can handle a limited number of cases. Dredze et al. [5],[6] studied the problem and 1 2
http://mturk.com/ http://www.officelabs.com/projects/forgottenattachmentdetector/ Pages/default.aspx
8
M-D. Albakour, U. Kruschwitz, and S. Lucas
introduced a significantly better system in terms of accuracy. In their work they used a supervised machine learning approach where they trained the classifier on bag-of-words features of the body of the email message and other features such as user profile (their history in sending messages). To our knowledge this is the only published work that studies the problem in sufficient detail and we are therefore using that approach as a baseline. 2.2
Mechanical Turk for NLP and IR
Supervised learning for text categorisation and classification requires annotated examples which can be an expensive and tedious task. Recently Amazon’s Mechanical Turk has proved to be a cheap and a fast platform to obtain document annotation for natural language processing and information retrieval tasks. It has been used recently by different researchers in Information Retrieval and Natural Language Processing as in [16], [3], [13]. Snow et al. show that nonexpert annotations can be used as gold standards for machine learning tasks in various complex natural language processing tasks such as word disambiguation and affect recognition [16]. Callison-Burch [3] explored using Mechanical Turk to evaluate machine translation and found that non-expert judgements are as good as those of experts. In our task we aim to collect finer-grained judgement to mark the specific sentences that refer to an attachment in an email message. We explore using Mechanical Turk for this task. 2.3
Sentence Level Approaches
Sentence-level approaches differ from coarse-grained document-level approaches by breaking the document into sentences and treating each sentence as a learning instance as opposed to considering the whole document. Bennett et al. [2] show the superiority of finer-grained sentence-level methods over coarse-grained document-level approaches in detecting action items in emails. Sentence-level classification in emails has also been successfully applied to text classification [10]. Processing at the sentence level was studied in blogs by Missen et al. [14] to explore the challenges accompanied by sentence-level opinion detection in user generated content. General challenges on the sentence level include the use of informal data which makes sentence detection harder and inaccurate and demonstrates the costly nature of sentence-level annotations.
3
The Corpus
Information Retrieval as a research area has partly been so successful because standard metrics, evaluation regimes and test collections have been developed that make it easy to compare different methodologies. We will use precision and recall (as well as F1) measures. There are only a few publically accessible email corpora that can be considered standard test collections. We will use the Enron
Sentence-Level Attachment Prediction
9
corpus so that we can compare our work with the baseline applied to the same data. The Enron corpus was made public in 2004 during a legal investigation concerning the Enron corporation. It contains 619,446 messages belonging to 158 users [11]. The corpus has triggered a lot of interest in email research. The corpus contains the attachment information but the actual attachments have been excluded from the prepared corpus [11]. Thanks to Dredze et al [5] we obtained a cleaned version of the corpus annotated with attachment information excluding forwarded messages resulting in a corpus of 157,711 messages from 150 users. In this cleaned corpus each email message has an extra header indicating if the message has an attachment or not and an extra header that contains a list of the attachment names if the previous header is positive. A detailed explanation of the cleaning process of the corpus can be found in [5]. We further analysed this cleaned corpus to study the distribution of emails with attachments and their contents in comparison to the rest of the emails in the corpus. We looked at the emails in the ‘sent’ folders of each user as they are more representative of real user-authored emails as emails in other folders might contain promotional content or spam. The ‘sent’ folders contain more than 33,000 emails, out of which around 2500 emails contain attachments. This suggests that usually only 8% of the emails that users send are used to transfer files. Within the emails which contain attachments, we were interested in how many sentences can potentially refer to an attachment. In order to do so we run the dataset through an automatic preprocessing step to clean the body of the emails from noisy text by: – Removing all ‘quoted’ text from messages in the same thread. When a message is a reply to another message in the same thread then email clients would include the body and some header information of previous emails in the thread. We used the heuristics introduced by [19] to eliminate quoted text from the email’s body. – Removing the sender’s signature from the body using simple heuristics. This cleaning step ensures that we only consider text that genuinely represents the email body (the text that the email author has written). After that we split the remaining text into sentences using the OpenNLP3 sentence splitter. Figure 1 illustrates how many sentences are left in an email after the cleaning step. More than 90% of those emails are rather short and contains less than 6 sentences. This observation allows us to estimate the effort and the time needed by annotators to identify sentences referring to an attachment within a single email message. The figure suggests that for a single email message annotators will be often asked to annotate less than six sentences. We also use this ratio when sampling the corpus so that the selected dataset reflects this distribution. 3
http://opennlp.org
10
M-D. Albakour, U. Kruschwitz, and S. Lucas
Fig. 1. Distribution of number of emails with attachments versus number of sentences
4
Data Selection
We will first only look at emails that do contain an attachment. In order to sample a representative data set (representative in terms of users and document length), we performed the following procedure: – Among the 20 users with the largest mail boxes we randomly selected 10 users who sent emails with attachments. – For each user we randomly collected up to 19 email messages so that the sample corpus reflects the distribution of sentences indicated above. As a result we obtained 178 email messages with 765 sentences. The task now is to identify the sentences in those emails which refer to any attached document.
5 5.1
Annotating the Data Amazon Mechanical Turk
Amazon’s Mechanical Turk (AMT) is a place for people to find microwork (a marketplace for crowd sourcing). The work typically pays a few cents, though some tasks take longer and might pay a few dollars. AMT requesters place tasks, which are called HITs (Human Intelligence Tasks), on the site. HITs include things like labelling a product photo with tags or rewriting a sentence in different words. Requesters receive the work done by workers and can choose to reject the work without giving a reason.
Sentence-Level Attachment Prediction
5.2
11
Task Design
The task we are interested in is annotating sentences within emails that refer implicitly or explicitly to an attachment. We used the selected dataset and performed the preprocessing steps explained previously in Section 3. We designed a HIT using the AMT web interface containing the following: – An example of the correct answer. This includes an email message with four sentences in which two are positively referring to an attached document and two negative sentences which are not related to any specific attachment. – The email subject, attachment names and the whole body of the email message to be annotated. – A list of sentences, produced by the OpenNLP sentence splitter on the cleaned body of the email, each of which to be annotated as to whether it refers to an attached document or not. For every task we asked five different unique workers to annotate the sentences as it was shown before that aggregation of multiple independent annotations from non-experts can produce good quality results [16]. We paid $0.05 for each task. We also asked two experts to do the same task for all the emails in the dataset in order to evaluate the quality of the non-expert annotations (experts in this case were researchers in the lab who performed the annotation task independently). Figure 2 shows an example of an email message given to the annotators.
Fig. 2. An example of an email message for the annotation task
12
5.3
M-D. Albakour, U. Kruschwitz, and S. Lucas
Results Processing
We automatically processed the results coming from the AMT workers to eliminate results from workers who did not follow the instructions they were asked when given the task. This is the case when the answer is not in the right format. As a result three workers’ annotations were excluded and their answers were substituted by reproducing the tasks with different workers. 5.4
Quality of Annotations
In order to test whether the non-expert annotations can be used as gold standard in a machine learning context we did some experiments to test: – Whether non-experts agree with each other. This is an indication of how well the non-experts are performing the task. – Whether the combined non-expert annotation agreement with experts is similar to the agreement between experts themselves. Inter-annotator agreement measurement. Fleiss’ Kappa statistical measure [8] was used to assess the reliability of agreement between the multiple annotators. The Kappa measure takes into consideration the probability of agreement by chance and is therefore considered more robust than simple percent agreement calculation. Kappa takes a value in the range of -1 to 1, where -1 indicates perfect disagreement below chance, 0 indicates agreement equal to chance and 1 indicates perfect agreement above chance. Table 1 shows the corresponding Kappa and percentage agreement (PA) values. Agreement among non-experts. As explained earlier we obtained five different judgements from different workers for each sentence in every email in the dataset, resulting in 765 sentences labelled 5 times with a flag indicating if the sentence refers to an attachment or not. The value of κ for these annotations was 0.61 indicating substantial agreement [12]. This is a baseline of good quality work. In the example given in Figure 2, all the non-experts agreed on sentences 1,2,3 to be positive (referring to an attachment), however they disagreed on sentences 4 where only 2 voted for positive and on sentence 5 where only one of the workers voted for positive. Agreement among experts. This measurement was conducted to understand the difficulty of this task in the given dataset. In the case of experts κ is 0.70 as measured between the two experts. As expected, the agreement between the two experts is higher than the agreement between the five non-expert judgements. However, the value is not very close to 1.0 which suggests that the task is not straightforward and in some cases it is difficult to judge if a given sentence refers to an attachment.
Sentence-Level Attachment Prediction
5.5
13
Combining Non-expert Annotations
In this work we followed the approach by Snow et al.[16] to combine the judgements produced by non-experts to produce an aggregation of non-expert annotations. This approach takes the average of the numerical judgement given by workers to produce an aggregated judgement. In our case the binary judgements (Yes/No) for each sentence were aggregated by simply taking the majority of votes, i.e. the sentence is considered positive (in other words, it refers to an attachment) when the number of votes for being positive is larger than those for being negative. We now ask the question of whether these combinations can be used as gold standard for our classification task. In order to do so we treated these aggregated annotations as annotations coming from one annotator and calculated the agreement between this annotator (non-experts combined in Table 1) and each of our experts. As shown in Table 1, the κ value for the agreement between nonexperts and each of the experts (0.73, 0.70) is comparable to the one between the experts themselves which suggest that their judgements are good enough to use in our task. Therefore substituting expert judgements with those of non-expert collected by AMT is justified for our task. Also it is important to realize that such annotations were quicker and cheaper to obtain with AMT. We paid 44 USD to the AMT workers to obtain 765*5=3825 labels in comparison to the time and effort by our two experts and the arrangements needed. Table 1. Inter-Annotator Agreement Inter-annotator agreement Comparison κ Non-expert agreement 0.61 Expert agreement 0.70 Non-experts combined vs. Expert 1 0.73 Non-experts combined vs. Expert 2 0.70
6
PA 0.81 0.85 0.87 0.85
Learning to Predict Attachments
In this section we explain the supervised learning using the gold standard obtained by non-expert annotations to identify sentences that refer to an attachment and therefore identify emails that might contain attachments. We aim to show the superiority of finer-grained sentence level approaches in the attachment prediction problem in comparison with document level approaches used previously. We compare our results for the attachment prediction problem to previous work by Dredze et al.[6] reported on the same corpus using document-level classification.
14
M-D. Albakour, U. Kruschwitz, and S. Lucas
6.1
Experimental Design
The dataset used to collect non-expert judgements contains 178 positive emails (documents) with 765 sentences, out of which 346 where identified by non-experts as positive. The task now is to develop a classifier that can identify positive sentences in an email message. The dataset contains only positive email messages. To make this dataset representative to real world mailboxes we had to add a number negative emails (emails without attachments) that will make the ratio of positive emails realistic, i.e close to 8%. We added 1000 negative emails from the ‘sent’ folders of the same ten users (100 each). Sentences in the additional 1000 negative emails were all treated as negative. We ended up with 1178 email message containing 4440 sentences out of which only 346 are positive. Therefore there is a large number of non-relevant instances and the data is highly skewed. 6.2
N-Gram Features
SVMLight4 , an implementation of Support Vector Machines, was used for classifying sentences. We tried different features to train a linear SVM classifier for the sentence classification. We used the OpenNLP tokenizer to obtain the features and we kept all punctuation marks like ‘.’, ‘!’, ‘?’ etc. We used n-gram features to train the classifier. For n-grams we considered all possible sequences of uni-grams, bi-grams, tri-grams and quad-grams. N-Grams have been shown to be effective in similar classification in emails, namely detecting speech acts [4]. Table 2 shows the performance obtained for sentence classification with different features using the Error Rate measure calculated using standard 10-fold cross validation for each classifier. Based on these results we opted for n-grams as it gives the best performance. Table 2. Performance of the SVM classifier using different features Stems Bi-grams N-grams Error Rate 18.3% 21.3% 16.9%
6.3
Learning with Skewed Data
With the skewed nature of the data, training a sentence classifier is tricky as the non-relevant instances are dominant which means that we need to undersample negative examples [15]. To show the difficulty of learning with skewed data, we experimented with learning on balanced and imbalanced datasets. Table 3 shows the 10-fold cross validation of the sentence classifier on different datasets,
4
http://svmlight.joachims.org/
Sentence-Level Attachment Prediction
15
Table 3. 10-fold cross validation performance results for the SVM sentence classifier on different datasets using N -Grams as features Positive Only All documents Precision 0.835 0.573 Recall 0.777 0.523 F1 0.805 0.547
sentences from only positive emails and sentences from the whole dataset. The performance dropped when using the whole dataset. Actually, recall was very low due to the large number of false negatives as a result of training the classifier on too many negative instances. Another way to handle the situation would be to select the most important features which was argued by Forman [9] to be more important than the actual classification algorithm in highly skewed situations. However the imbalanced nature of the data will affect the feature selection procedure [20]. Therefore we have balanced the data to select features and then used only those features to train the classifier on the whole imbalanced dataset. 6.4
Feature Selection
Which features are the most discriminating ones for our task on a balanced dataset? The motivation behind balancing the data is that feature selection measures can fail to produce good classification accuracy [20]. As argued by Zheng et al. [20], when using one-sided metrics which only take the positive features into account, the negative documents will be very likely to be misclassified and that will dramatically affect the classifier performance due to their dominance. On the other hand, when applying two-sided metrics which take both features into account, it can be difficult for negative features to obtain high scores due to the large number and diversity of negative documents which is the case for negative email sentences. Our approach is to select the features from a balanced dataset of negative and positive sentences (balanced in the sense that we add a small number of emails without attachments to the sample corpus). We created this dataset by taking all the 765 sentences from the 178 positive emails annotated by non-experts through AMT. We then added 70 negative emails taken randomly equally from our ten selected users. We end up with a dataset containing 1023 sentences from positive and negative emails out of which 346 sentences are positive. The resulting sentences contain 41,187 n-gram features. We used Chi Square to rank those features. Chi measures the lack of independence between a feature f and category c [18]. χ2 (f, c) =
¯ c)−P (f,¯ N[P (f,c)P (f,¯ c)P (f¯,c)]2 P (f )P (f¯)P (c)P (¯ c)
16
M-D. Albakour, U. Kruschwitz, and S. Lucas
where N is the total number of documents P (f, c) is the probability of presence of f and membership in c P (f, c¯) is the probability of presence of f and non-membership in c P (f¯, c) is the probability of absence of f and membership in c P (f¯, c¯) is the probability of absence of f and non-membership in c. This results in reducing the number of features to 1045. To give the reader a flavour of the selected features, here is a list of the top bigrams selected: ’here is’, ’is the’, ’please review’, ’, please’, ’the file’, ’attached is’, ’is a’, ’the attached’, ’this is’, ’this spreadsheet’. We then used only these features to evaluate the linear SVM classifier on the whole skewed data. The 10-fold cross validation measuring precision and recall of the classifier gave us 0.809 and 0.549 respectively. Note that these measures reflect how well we can predict sentences that refer to attachments. We will now look at how this can be used to detect emails that expect attachments. 6.5
Experiments on Attachment Prediction on the Document Level
On the document level, an email is considered positive if the classifier identified at least one positive sentence and it is considered negative (i.e. not referring to an attachment) otherwise. We measured the performance on the document level by calculating the confusion matrix of each fold and aggregating the results. We performed a standard 5-fold cross validation on the document level. All sentences in a document are either entirely in the training set or entirely in the testing set for each fold. Each fold contains emails with an equal proportion of positive and negative emails from each user mail box. The first column (‘All’) in table 4 shows the performance on the document level for this setting. These figures shows the improvement of our approach over a bag-of-word approach [6]. Dredze et al. report 67% recall and 83% Precision on a single user setting where the classifier was evaluated on a dataset from the same user. Also in that work additional features such as the user profile were taken into account. In our approach, we obtained a higher recall on a more difficult setting without taking the user profile into account. Furthermore, to compare our method to the cross-user evaluation setting in [6], we performed a 5-fold cross validation on a cross-user setting where we trained the classifier on emails from certain users and tested on emails from other users. The dataset was split into 5 folds on which each contains emails from 2 users out of our ten users.The results are reported in Table 4. This shows the system is also performing much better than previously reported (56% recall, 80% precision, 66% F1) on such settings. F1 is 16% better than previously reported. We also performed a much harder evaluation setting where we took another 600 email messages, not included previously in the initial 1178 emails dataset, from the ‘sent’ folders of our ten users, 100 of which are positive and 500 are negative (equally distributed over all users). We used this dataset as our testing
Sentence-Level Attachment Prediction
17
Table 4. Document-level attachment prediction using sentence level classifiers All Cross-User Precision 0.825 0.835 Recall 0.701 0.704 F1 0.758 0.764
New dataset 0.900 0.630 0.741
set and sentences from our initial dataset as the training set. The column labelled ‘new dataset’ in Table 4 shows the results reported in this setting which suggests that the classifier can scale to a larger dataset.
7
Discussion and Future Work
The main focus of this work was to investigate the usefulness of using sentence-level attachment prediction. We showed that detecting attachment-related sentences in emails can beat state-of-the-art methods in document-level attachment prediction which can be used to inform users that an email is missing an attachment. Although our results consider a difficult cross-user evaluation setting, the emails are specific to the Enron corpus. In a real world setting the system should work for a new user in any environment. Some features identified by the feature selection process might be useful in the Enron corpus but not in other environments. The work also presented yet another validation of the usefulness of Amazon’s Mechanical Turk for non-expert annotation tasks5 . We have shown that the aggregated knowledge collected like that can be used to substitute expert user annotation. This finding has led us to recently collect annotations for the entire cleaned Enron corpus. However due to the complexity arising from larger-scale computation required for the feature selection step we were not able to repeat the experiments required to report the results on the full dataset and we consider this as a future work. Another line of our future work is to link attachments to facts extracted from individual sentences. This is part of a larger project aimed as semantic search over email archives [1].
Acknowledgements This work is funded by a partnership between Active Web Solutions (AWS)6 and the University of Essex and funded by a grant from the Technology Strategy Board under the Knowledge Transfer Partnership scheme, grant number KTP006743, as well as the R&D division of AWS.
5 6
The annotations collected in this study can be obtained by contacting the authors. http://aws.net
18
M-D. Albakour, U. Kruschwitz, and S. Lucas
References 1. Albakour, M.-D., Kruschwitz, U., Blackwell, R., Lucas, S.: Managing collaboration projects with semantic email search. In: Proceedings of the WWW 2009 workshop on Semantic Search (April 2009) 2. Bennett, P.N., Carbonell, J.: Detecting action-items in e-mail. In: SIGIR 2005: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 585–586. ACM, New York (2005) 3. Callison-Burch, C.: Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 286–295. Association for Computational Linguistics (2009) 4. Carvalho, V.R., Cohen, W.W.: Improving “email speech acts” analysis via n-gram selection. In: ACTS 2009: Proceedings of the HLT-NAACL 2006 Workshop on Analyzing Conversations in Text and Speech, pp. 35–41. Association for Computational Linguistics, Morristown (2006) 5. Dredze, M., Blitzer, J., Pereira, F.: sorry, i forgot the attachment: Email attachment prediction. In: CEAS (2006) 6. Dredze, M., Brooks, T., Carroll, J., Magarick, J., Blitzer, J., Pereira, F.: Intelligent email: reply and attachment prediction. In: IUI 2008: Proceedings of the 13th international conference on Intelligent user interfaces, pp. 321–324. ACM, New York (2008) 7. Fisher, D., Brush, A.J., Gleave, E., Smith, M.A.: Revisiting Whittaker & Sidner’s “email overload” ten years later. In: CSCW 2006: Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, pp. 309–312. ACM, New York (2006) 8. Fleiss, J., Levin, B., Paik, M.C.: Statistical Methods for Rates and Proportions, 3rd edn. John Wiley and Sons Inc., Chichester (2003) 9. Forman, G.: An extensive empirical study of feature selection metrics for text classification. J. Mach. Learn. Res. 3, 1289–1305 (2003) 10. Khoo, A., Marom, Y., Albrecht, D.: Experiments with sentence classification. In: Proceedings of the 2006 Australasian Language Technology Workshop (ALTW 2006), pp. 18–25 (2006) 11. Klimt, B., Yang, Y.: The Enron Corpus: A New Dataset for Email Classification Research. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 217–226. Springer, Heidelberg (2004) 12. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977) 13. Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: TurKit: tools for iterative tasks on Mechanical Turk. In: HCOMP 2009: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 29–30. ACM, New York (2009) 14. Missen, M.M.S., Boughanem, M., Cabanac, G.: Challenges for sentence level opinion detection in blogs. In: ACIS-ICIS, pp. 347–351 (2009) 15. Singhal, A., Mitra, M., Buckley, C.: Learning routing queries in a query zone. SIGIR Forum 31(SI), 25–32 (1997) 16. Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 254–263. Association for Computational Linguistics (2008)
Sentence-Level Attachment Prediction
19
17. Whittaker, S., Sidner, C.: Email overload: exploring personal information management of email. In: CHI 1996: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 276–283. ACM, New York (1996) 18. Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: Fisher, D.H. (ed.) Proceedings of ICML 1997, 14th International Conference on Machine Learning, Nashville, US, pp. 412–420. Morgan Kaufmann Publishers, San Francisco (1997) 19. Yeh, J.-Y.: Email thread reassembly using similarity matching. In: CEAS (2006) 20. Zheng, Z., Wu, X., Srihari, R.: Feature selection for text categorization on imbalanced data. SIGKDD Explor. Newsl. 6(1), 80–89 (2004)
Rank by Readability: Document Weighting for Information Retrieval Neil Newbold, Harry McLaughlin, and Lee Gillam University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom
[email protected],
[email protected],
[email protected]
Abstract. In this paper, we present a new approach to ranking that considers the reading ability (and motivation) of the user. Web pages can be, increasingly, badly written with unfamiliar words, poor use of syntax, ambiguous phrases and so on. Readability research suggests that experts and motivated readers may overcome confusingly written text, but nevertheless find it an irritation. We investigate using readability to re-rank web pages. We take an extended view of readability that considers the reading level of retrieved web pages using techniques that consider both textual and cognitive factors. Readability of a selection of query results is examined, and a re-ranking on readability is compared to the original ranking. Results to date suggest that considering a view of readability for each reader may increase the probability of relevance to a particular user. Keywords: Web IR, Content-based filtering, NLP for IR, Evaluation methods and metrics.
1 Introduction The goal for information retrieval (IR) is retrieving documents relevant to a user’s information needs. The well-known Probability Ranking Principle (PRP) [1] states that descending probability of relevance to a query is the most effective way to present retrieved documents to the user. This generally appears to have been interpreted as generic relevance to a query on the basis of information in and across texts (Robertson [1]) rather than relevance to the user. We consider that the PRP should account for, amongst other things, reading ability of the user, which would provide for a ranking relating to the likelihood that a user would be able to understand content on any given page. Currently, systems will largely produce the same results for experts in scientific fields as for novices, proficient readers, children, people with learning difficulties, and so on. There is likely to be considerable potential benefit in assessing appropriateness for each user. If mechanisms can be produced to learn about the reading abilities of the user, systems can filter and/or re-rank results according to each reader ability. Readability research since Kitson [2] has explored matching text to a particular (type of) reader and a number of readability measures claim to assess text for an appropriate reading level. By themselves, these formulae may be used to H. Cunningham, A. Hanbury, and S. Rüger (Eds.): IRFC 2010, LNCS 6107, pp. 20–30, 2010. © Springer-Verlag Berlin Heidelberg 2010
Rank by Readability: Document Weighting for Information Retrieval
21
match results to readers, though many would be concerned about the limited focus of these measures on sentence and word lengths or syllable counts. Recent considerations of readability by Oakland and Lane [3] account for both text factors and cognitive factors that affect a reader’s ability to understand and enjoy written content. This work provides the inspiration for our present research and has led us towards exploring how measures of text and cognitive factors might be usefully incorporated into filtering. In this paper, we explore the relationship between text and reader by evaluating the rankings according to readability in relation to a number of queries. We consider, further, how to associate text with a specific reader. In Section 2, we discuss readability research and relevance to information retrieval; Section 3 describes our present research in relation to readability; Section 4 outlines results of readability-ranking; Section 5 concludes the paper and makes considerations for future work.
2 Background To associate query results with an individual reader, we need to consider the relationship between text and reader. Historically, readability research has focused primarily on producing a numeric evaluation of style of writing based on Kitson’s [2] work, using sentence length and word length measured in syllables. These readability measures indicate some proportion of the population who could comfortably read a text, and comparisons have been made amongst measures to correlate with specific human performances over largely disjoint sets of texts. Further discussion of these formulae can be found elsewhere [4, 5]. More recent considerations of readability account for reader factors, which consider certain abilities of the reader, and text factors, which consider the formulation of the text (Oakland and Lane [3]). Reader factors include the person’s ability to read fluently, level of prior subject knowledge, lexical knowledge or familiarity with the language, and motivation and engagement. Text factors account to some extent for current readability metrics, but also cover considerations of syntax, lexical selection, idea density, and cognitive load. Oakland and Lane’s view of readability suggest that it may be possible to generically measure the difficulty of text as an artifact, but that “text difficulty” necessitates consideration of each reader. Our work elaborates that of Oakland and Lane in identifying difficulties in the apparently neat separation of the factors. In this section, we propose a new framework for readability that builds on Oakland and Lane by making consideration of the relationship between text, reader and author. We explore, subsequently, how IR systems might use such a framework in associating text to reader. 2.1 Matching Text to Readers In writing a document, or undertaking search engine optimization, an author has to be mindful of the needs of his intended audience, particularly if they are to continue reading. There must be some correlation across three principal aspects of a text: the nature and extent of its subject matter, its use of language, and its logical or narrative structure. The audience can be defined by their degree of interest in the subject, how much they already know about it, their reading ability, and their general intelligence.
22
N. Newbold, H. McLaughlin, and L. Gillam
Two kinds of measure are suited to appraising text structure: logical coherence and propositional density. By logical coherence, we mean the extent to which one statement is ordered according to a chain of reasoning, a sequence or chain of events, a hierarchy or a classificatory system. By propositional density we mean the closeness, measured by intervening words, between one crucial idea and the next. The greater the density of ideas, the larger the cognitive load on the reader. For our current considerations, we are interested in similarities that might occur in re-ranking results by reading level, to assess the effects of such re-rankings, and to make comparisons over extant and proposed readability formulae. Our new framework for readability, describing the factors to be considered is presented in Fig. 1. In the remainder of this section, we elaborate these factors.
Fig. 1. Matches needed for easy reading
2.2 Language When matching text to reader, the author needs to consider the level of language and style of writing. We propose that systems examine the vocabulary familiarity and syntactic complexity. We describe each of these below: Vocabulary Familiarity. Readability metrics generally determine the difficulty of a word by counting characters or syllables. However, Oakland and Lane cite factors such as simplicity or familiarity as more effective means. They suggest that word difficulty can be determined by examining whether the word is challenging, unusual or technical, and describe vocabulary as a text factor contributing to text difficulty.
Rank by Readability: Document Weighting for Information Retrieval
23
The process by which readers develop word familiarity is part of language acquisition which concerns the reader’s development of their language capability. Frequency plays an important role in building knowledge of a language so that it is sufficient to understand its written content. Diessel [6] showed that linguistic expressions stored in a person’s memory are reinforced by frequency so that the language user expects a particular word or word category to appear with a linguistic expression. These linguistic expectations help comprehension. Frequency was also found to be fundamental in reading fluency as words are only analyzed when they cannot be read from memory as sight words. A limited knowledge of words affects reading fluency as readers are likely to dwell over unfamiliar words or grammatical constructions. This impedes the reader’s ability to construct an ongoing interpretation of the text. The reading fluency of the reader is dependent on their familiarity with language. When readers find text populated with unfamiliar words it becomes harder for them to read. This is especially prevalent in scientific or technical documents. The terminological nature of specialized documents means that terms will appear with disproportionate frequently throughout the documents in contrast to what one would expect to encounter in everyday language. Anyone unfamiliar with the terminology would find the document hard to understand, though terms can be identified by exploiting this relationship. We can use word familiarity as an indicator of word difficulty by contrasting frequency within documents with familiarity in general language. Syntactic Complexity. Vocabulary does not tend to exist in isolation. The vocabulary may be well-defined, yet included in overly verbose sentences. Existing readability formulae consider that long and complex sentences can confuse the reader. However, long sentences do not cause a problem because of memory limits but because of comprehension skills. Reciting text verbatim, or summarizing, requires only a superficial understanding. People can reproduce text word by word without understanding it. Pearlmutter and Macdonald [7] demonstrated how unskilled readers can have the necessary knowledge to understand text but fail to use it when decoding sentences: they cannot associate their ongoing interpretation of a sentence to their existing knowledge. This was due to poor comprehension skills in the reader which Daneman and Carpenter [8] described as reading span. A series of sentences devised by Miyake et. al [9] differentiated between high- and low-span readers. These sentences had ambiguous words which could not be resolved until several words later, an example of which is shown below: “Since Ken liked the boxer, he took a bus to the nearest pet store to buy the animal.” High-span readers had no problem disambiguating ‘boxer’ but low-span readers were confused unless the sentence ordering was changed. Here, the low-span readers performed identically to high-span readers showing that they understood the different senses of the ambiguous word. The syntactic structure of a sentence affects comprehension for users with lower reading skills. When matching text to reader, syntactic complexity should be examined to considering whether it is appropriate for the reading level.
24
N. Newbold, H. McLaughlin, and L. Gillam
2.3 Subject To learn from text, a reader needs to associate the new information to their existing knowledge. This task can be helped by the reader’s interest level. Kintsch et al. [10] showed that we find stories easier to remember than technical texts because they are about human goals and actions, something to which we can all generally relate. Scientific and technical texts require specific knowledge that is often uncommon, making the texts impenetrable to those outside the domain. This suggests that readability is not merely an artifact of text with different readers having contrasting views of difficulty on the same piece of text. Familiarity with certain words depends on experience: a difficult word for a novice is not always the same as a difficult word for an expert. Reader characteristics such as motivation and knowledge may amplify or negate problems with difficult text. We propose IR systems measure the assumed knowledge of the reader to determine an individual readability score which we describe in detail below: Assumed Knowledge. Many readability metrics do not make distinctions based on the background knowledge of the reader. As discussed in relation to vocabulary, word familiarity can give a better indication of word difficulty than word length. A longer word may only be difficult for a particular reader if unfamiliar, and certain shorter words may even be more difficult to understand. Consider a general reader confronted in text discussing a ‘muon’: This short term would be rated as simple by current readability formulae. However, a majority of people would be unfamiliar with this term, and only physicists are likely to know more about the term, its definition, and related items. One way to measure background knowledge would be to consider the extent of use of known terms in the text with direct consideration of previous documents within the reader’s experience. Entin and Klare [11] showed that more readable text is beneficial for those with less knowledge and interest. In their study, students were presented with written material below their reading level. When the reader’s interest was high, text below their grade level did not improve comprehension. However, when the reader’s interest was low their comprehension was improved by simpler text. This suggests that more readable text improves comprehension for those less interested in the subject matter. We consider the need to capture and analyze the user’s experience with prior documents as a proxy for reader knowledge and motivation. Given a reading history for a user, we might next build their personalized vocabulary with frequency information, and therefore measure familiarity with words on an individual basis. In the same way that an expert is familiar with the terminology of their subject, we can reflect the background knowledge required by a reader to interpret the text correctly. When matching text to reader, word difficulty should be measured, if the information is available, on an individual basis. 2.4 Structure Well-written text requires a structure that readers can readily use to find the information they need and to understand it correctly. Text can become confusing when ambiguities arise from information inappropriately presented. Most sentences can become multiply ambiguous if taken out of context. When we read text, we build a collection
Rank by Readability: Document Weighting for Information Retrieval
25
of concepts described within the text. These concepts are described by words and phrases which we encode using pragmatic, semantic and syntactic features. When frequently combined linguistic expressions develop into a processing unit, many of the linguistic elements are ignored and the whole chunk is compressed and treated as one semantic unit. These units often develop into terms with multiword units representing singular concepts. This relates back to the assumed knowledge of the reader. However, for readers unfamiliar with the terms, we have identified two methods called ‘Propositional Density’ and ‘Lexical Incoherence’ for processing semantic units. Propositional Density. When a significant amount of information is conveyed in a relatively small amount of text, the reader can become confused. Although long collocations form semantic units that reduce conceptual complexity, problems occur when numerous semantic units are described within a short space of each other causing the reader to make numerous inferences. The number of ideas expressed in the text contributes to the work required of the reader to interpret the text correctly. Propositional density may be measurable by examining the quantity of objects within short distances of each other. These objects can be labeled with single nouns or multiword expressions. By measuring the number of unique semantic units, we can approximate the workload required for processing or interpreting the text correctly. Logical Incoherence. Problems with lexical coherence occur when writers present new information to the reader without making clear its relationship to previous information: the writer assumes that they have provided enough information to allow readers to follow their arguments logically. Semantic units can be referred to by a number of different labels and by identifying these different labels -, synonyms - we can find the prominent ideas in the text. Repetition of terms and their synonyms and other referents provides a structure for the reader to connect with. There is a relationship here to work on lexical cohesion (Hoey [12]). If a large number of new, seemingly unrelated ideas are being introduced, low cohesion would be expected and measurable.
3 Readability Based Web Ranking Readability depends on text factors and readers. IR systems generally appear to make limited efforts in gathering information about the abilities of the user that would assist in tailoring results. Familiarity of words and complexity of the sentences should be contrasted with the knowledge of the reader. For technical or specialized texts, an indication would be that a reader had demonstrated an interest in related prior material and may be more familiar with its terminology. We would assume that a searcher is somehow motivated and engaged, if informative texts are required, that these are presented appropriately and cohesively. We have undertaken experiments to explore the relationship between readability and rank. We retrieve the first 100 results from search queries issued using the Google plug-in for, GATE (General Architecture for Text Engineering) and apply our own implementations of a series of readability formulae to evaluate various considerations over ranking. We use Flesch Easy Reading Formula, Flesch Kincaid Formula, FOG Index, SMOG and ARI, described in [4,5] as measures of vocabulary and syntactic
26
N. Newbold, H. McLaughlin, and L. Gillam
complexity, and a familiarity measure, Eqn. 1, based on contrast with word frequency in the British National Corpus (BNC). Eqn. 1 sentence-based familiarity
1.5 × ln(n SL ∑
f SL nGL ) (1 + f GL ) n SL
(1)
f is word frequency, n is word count at sentence level (SL) or corpus level (GL). A document measure is derived from average sentence value. Information about prior experience of words may be available through analysis of texts with which the user has some familiarity; Eqn. 1. can then be used against such a collection initially, and subsequently revert to contrasts with BNC. We measure propositional density using the number of individual semantic units (concepts) in a sentence. We use POS tagging in GATE to identify singular nouns with considerations of adjacency and dependency from Lauer [13] to determine multiword expressions. Larger values indicate a more complex text. Eqn. 2 sentence-based propositional density
1 + u SL 2 + n SL − c SL
(2)
u is the number of semantic units, n is sentence length, c is the number of collocated words. A document measure is derived using average sentence value. It is possible to obtain a measure of coherence by calculating the average frequency of words in a document: the number of tokens divided by the number of types. Such a measure will be variously skewed by stopwords, use of synonyms and hypernyms/ hyponyms and other kinds of references. To avoid such skew, we measure lexical coherence by counting the total number of nouns and verbs in the document, and resolving headword references where possible using WordNet [14] for both keywords and multiword expressions. Large numbers of infrequently encountered and relatively unrelated ideas indicate poor incoherence, resulting in high scores for our lexical incoherence measure and texts likely to either be stuffed with keywords or fragments or generally poor matches for readers with lower reading age. Consideration is also being made of incorporating measures for semantic distance within our work. Eqn. 3 shows our measure for lexical coherence for a document as log e
(un DL + uv DL ) 2 + (tn DL + tv DL ) 2 + s DL n DL
(3)
where un and uv are the number of unique nouns and verbs in document respectively. Similarly, tn and tv are the total number of nouns and verbs. s refers to the number of sentences and n is the total word count at a document level (DL).
Rank by Readability: Document Weighting for Information Retrieval
27
4 Results An initial set of 10 queries was devised to return a variety of results expected to be suited to different audiences. The results returned were expected to range from general knowledge to technical with some queries covering technical subjects that needed to be understood by people with low reading skills. The first 100 results were analyzed for readability by the formulas, Flesch Kincaid, FOG Index, SMOG, ARI, and familiarity (Eqn. 1) to return the years of education required to understand each text. Table 1 shows the average readability score calculated by each formula, for the first 100 texts returned from each query. The results show that queries such as ‘legal aid’ and ‘mental health advice’ which need to be understood by people with little technical knowledge tended to score lower with the familiarity measure. More complex documents, such as those returned by ‘predict u.s. inflation rate’ and ‘muon neutrino’ were deemed the most difficult by all the measures. Table 1. The 10 search queries, with average readability scores for the first 100 results calculated by each of the readability formulas, and a correlation between the familiarity formula and the other readability formulas Search query legal aid henry viii wives mental health advice stonehenge facts stock market crash pop music news children’s problems harry potter sales predicts u.s. inflation rate muon neutrino Correlation
Familiarity 11.81 11.98 11.99 12.12 12.13 12.32 12.34 12.39 12.77 13.11
Kincaid 10.73 9.19 10.19 9.98 9.56 9.73 11.51 10.05 12.00 12.89 0.98
FOG 12.58 11.01 12.08 12.32 11.49 11.43 13.38 11.43 14.31 16.01 1.00
SMOG 11.72 10.60 11.55 11.71 11.14 10.78 12.41 10.87 13.04 14.34 0.93
ARI 10.05 8.56 10.33 9.96 9.37 9.39 11.78 9.90 11.88 12.46 0.76
Whilst our familiarity formula correlates with the other readability formulas, we decided to use the average reading age from all of the readability formulas as a reliable indicator for readability. The first 100 results are re-ranked according to average reading age with the most readable documents ranking first. We next determined the percentage of results suitable for somebody with 10 years of education or fewer, and how many of the first 10 results would be suitable for the same assumed education level: a large proportion of the returned search results would appear unsuitable for certain queries. We used Spearman's coefficient to evaluate the impact of re-ranking in increasing order of readability (low to high) ranking and found little correlation between readability ranking and search ranking (see Table 2). Least readable results occur for ‘muon neutrino’ and ‘predicts u.s. inflation rate’: both returned texts requiring subject matter expertise and containing difficult terms.
28
N. Newbold, H. McLaughlin, and L. Gillam
Table 2. The 10 search queries, with average readability for the first 100 results for these queries, the number of readable results if 10 years of education is assumed, and a correlation between readability ranked and original ranking of the readability Rank Search query
Avg. No. of Top 100 No. of Top 10 Spearman’s Readability suitable for suitable for RHO Reading age 10 Reading age 10 henry viii wives 10.27 64 7 0.309 pop music news 10.73 55 9 0.161 stock market crash 10.74 50 7 0.015 harry potter sales 10.93 49 7 0.095 stonehenge facts 11.22 46 4 0.041 mental health advice 11.23 53 7 -0.006 legal aid 11.38 43 3 0.045 children’s problems 12.28 33 4 0.032 predicts u.s. inflation rate 12.80 17 3 0.014 muon neutrino 13.76 7 0 0.267
1 2 3 4 5 6 7 8 9 10
Our familiarity formula correlates well other readability measures, but to examine this further, we scaled the results of applying these measures and examined frequencies of documents in six ranges. Results for ‘muon neutrino’ are shown in Fig. 2. and suggest that our familiarity formula considers the results from this query to be less generally readable: typical readability metrics produce low scores in the presence of short words; further work is required to evaluate whether our familiarity measure is a better indicator than these measures.
Kincaid FOG SMOG ARI Familiarity
1
2
3
4
5
6
Fig. 2. Scaled frequencies into six ranges for five readability formula scores on the documents returned for the query ‘muon neutrino’
Propositional density was returning unusually high results. Limited pre-processing was provided by the search plug-in, so we manually cleaned the first 5 results for each query and re-calculated readability (Table 3). The query for ‘mental health advice’ scores high on propositional density due to multiword expressions such as ‘borderline personality disorder’, ‘obsessive compulsive disorder’ and ‘telephone self help groups’. Documents returned by ‘harry potter sales’ also tend to score high due to sales-related descriptions - ‘Los Angeles-based Exhibitor relations’; ‘senior box-office
Rank by Readability: Document Weighting for Information Retrieval
29
analyst’ and titles such as ‘Half-blood prince’. These types of documents would be more readable to an expert in the subject matter. Therefore, high propositional density indicates further need to consider reader familiarity, which would reduce an overall readability score. Documents returned for ‘Stonehenge facts’ get the lowest score for lexical incoherence due to repeating themes of ‘rocks’, ‘construction’ and ‘old’. Further results can be seen in Table 3. Documents returned for ‘henry viii wives’ scored low on all measures except for lexical coherence, where they scored as one of the highest. The fact that these documents discuss each of Henry’s wives shows that a lot of different information is being presented resulting in the high score for incoherence. This indicates that a factor of the text is being addressed here which is ignored by the other measures, we can identify the amount of disparate information in the text. Table 3. The 10 search queries with the cleaned corpus Previous Rank
Search query
1 7 8 5 2 6 3 4 9 10
henry viii wives legal aid children’s problems stonehenge facts pop music news mental health advice stock market crash harry potter sales predicts u.s. inflation rate muon neutrino
Average Readability
9.02 10.16 10.32 10.85 11.33 11.58 11.71 11.79 14.19 14.38
Propositional Lexical Density Incoherence 0.48 5.10 0.37 4.26 0.40 4.89 0.49 3.93 0.44 4.61 0.51 4.94 0.45 5.08 0.50 4.68 0.48 5.70 0.47 4.78
5 Conclusions and Future Work The ease of understanding written information, or the readability of a piece of text is becoming more important with the sprawling digital morass of contributory social media and users complaining about the poor quality of written content on the Web. The demand for filtering out confused or verbose text appears to be significant, but does not yet appear to have been embraced. Our research aims at providing for an advanced readability analysis that can account for extant sentence length and word length approaches, and caters for further measures covering familiarity, complexity, knowledge, density and coherence. In this paper, we have demonstrated how readability metrics by themselves would impact ranking. Indeed, results can be filtered according to ability, or desire, of the user. We present a series of formulas that measure different aspects of the text to indicate their suitability for the reader. The user can select their reading age and interest/knowledge in a subject to return documents that more closely match their requests. Our results indicate that results of certain queries may largely be unsuitable for some users, with technical documents beyond their reading ability. In particular, we suggest that the Probability Ranking Principle should account for the reading ability of the
30
N. Newbold, H. McLaughlin, and L. Gillam
user to provide for a ranking relating to the likelihood that a user would be able to understand content. A range of further research is required for the extensive evaluation of these measures and others which we are devising to address text factors and cognitive factors. This future work may also make strong use of relevant related resources such as the Living Word Vocabulary and Google n-grams data in devising particular measures.
References 1. Robertson, S.E.: The probability ranking principle in IR. Readings in Information Retrieval, 281–286 (1997) 2. Kitson, H.D.: The mind of the buyer. Macmillan, New York (1921) 3. Oakland, T., Lane, H.B.: Language, Reading, and Readability Formulas: Implications for Developing and Adapting Tests. International Journal of Testing 4(3), 239–252 (2004) 4. DuBay, W.H.: The Principles of Readability. Impact Information, Costa Mesa (2004) 5. Gillam, L., Newbold, N.: Quality Assessment. Deliverable 1.3 of EU eContent project LIRICS (2007), http://lirics.loria.fr/doc_pub/T1.3Deliverable.final.2.pdf 6. Diessel, H.: Frequency Effects in Language Acquisition Language Use, and Diachronic Change. New Ideas in Psychology 25(2), 108–127 (2007) 7. Pearlmutter, N.J., MacDonald, M.C.: Individual differences and probabilistic constraints in syntactic ambiguity resolution. Journal of Memory and Language 34, 521–542 (1995) 8. Daneman, M., Carpenter, P.A.: Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior 19, 450–466 (1980) 9. Miyake, A., Just, M.A., Carpenter, P.A.: Working memory constraints on the resolution of lexical ambiguity: Maintaining multiple interactions in neural contexts. Journal of Memory and Language 33, 175–202 (1994) 10. Kintsch, W., Kozminsky, E., Streby, W.J., McKoon, G., Keenan, J.M.: Comprehension and recall as a function of content variables. Journal of Verbal Learning and Verbal Behavior 14, 196–214 (1975) 11. Entin, E.B., Klare, G.R.: Relationships of measures of interest, prior knowledge, and readability comprehension of expository passages. Advances in reading/language research 3, 9–38 (1985) 12. Hoey, M.: Patterns of Lexis in Text. OUP, Oxford (1991) 13. Lauer, M.: Designing Statistical Language Learners: Experiments on Noun Compounds. Ph.D. thesis, Macquarie University, Sydney, Australia (1995) 14. Fellbaum, C.: WordNet: An Electronic Lexical Database. Bradford Books (1998)
Knowledge Modeling in Prior Art Search Erik Graf, Ingo Frommholz, Mounia Lalmas, and Keith van Rijsbergen University of Glasgow, {graf,frommholz,mounia,keith}@dcs.gla.ac.uk http://ir.dcs.gla.ac.uk/
Abstract. This study explores the benefits of integrating knowledge representations in prior art patent retrieval. Key to the introduced approach is the utilization of human judgment available in the form of classifications assigned to patent documents. The paper first outlines in detail how a methodology for the extraction of knowledge from such an hierarchical classification system can be established. Further potential ways of integrating this knowledge with existing Information Retrieval paradigms in a scalable and flexible manner are investigated. Finally based on these integration strategies the effectiveness in terms of recall and precision is evaluated in the context of a prior art search task for European patents. As a result of this evaluation it can be established that in general the proposed knowledge expansion techniques are particularly beneficial to recall and, with respect to optimizing field retrieval settings, further result in significant precision gains.
1
Introduction
Identifying relevant prior art, i.e. trying to retrieve documents in patent and nonpatent literature that are closely related to the matter described in a patent document, is probably the most commonly executed task in the patent domain. These searches form an essential part of the process of determining the patentability of a specific invention [2]. In order for an invention to be viable for patenting, no prior record of a similar or identical product or process may exist (See Section B IV 1/1 in [2] for a more detailed description). A prior art search therefore aims at clarifying whether any such records exist in patent and non-patent literature that have been published prior to the filing of the patent application in question. Since the erroneous granting of a patent can result in later litigation costs of hundreds of million Euros, extensive effort is invested into retrieving every relevant document. In this context the search for prior art constitutes a good example of a recall-focused task. In this study we explore in what ways knowledge modeling and the integration of knowledge representations into the prior art retrieval task can be beneficial in light of these requirements. Modeling and representing knowledge has been widely researched in Cognitive Psychology [20] and Artificial Intelligence (AI) [12] as part of their quests to understand and replicate aspects of human cognition. In the context of Information Retrieval (IR) the integration of knowledge has been explored as part of the Intelligent Information Retrieval (IIR) [10] H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 31–46, 2010. c Springer-Verlag Berlin Heidelberg 2010
32
E. Graf et al.
and Associative Information Retrieval (AIR) initiatives [27] as a means of building more effective systems. With respect to recall-oriented tasks, mitigating vocabulary mismatch [14], i.e. to allow for the detection of semantic relatedness of documents where it is not reflected through the mutual occurrence of terms, represented a central aim of these approaches. While initial results obtained in these subdomains of IR have been promising, widespread adoption has been limited by a variety of factors. Of these the prohibitively high costs of manual knowledge representation creation, and lack of success concerning the automation of the process, proved to be most severe. As a result knowledge modeling related research in IR relies on the utilization of available knowledge artifacts such as thesauri [19], ontologies, and citations [27]. In the patent domain, a structure that can be interpreted in this sense is given by the International Patent Classification (IPC) system1 . In this system every new patent application is, with respect to technological aspects of the described invention, assigned to one or more classes within a hierarchy consisting of more than 70,000 different elements. These assignments are conducted by patent examiners based on their in depth knowledge of the respective technologies. As a consequence this structure represents a highly precise hierarchical mapping of technological concepts, and provides an excellent basis for the extraction of knowledge. In light of this, the patent domain can be interpreted as an excellent new testbed for revisiting IIR and AIR related concepts. The focus of this study is placed on evaluating the potential benefit of integrating knowledge representations into the prior art patent retrieval task. More specifically, inspired by research from Cognitive Psychology with respect to hierarchical aspects of memory, we propose to model knowledge in the form of a hierarchical conceptual structure extracted from available IPC information. In our chosen representation each element of the IPC hierarchy is comprised of a set of terms reflecting its characteristic vocabulary. To this cause a method aimed at extracting representative vocabularies for the technological aspects covered by specific IPC elements is developed. Based on this representation, strategies concerning the integration of the extracted knowledge into the retrieval task, with respect to the underlying aim of enabling the identification of similarity between a query and a document even in the absence of mutually shared terms, are investigated. Finally the potential benefit of these techniques is evaluated based on the prior art search task of the CLEFIP 09 collection. The remainder of this paper is structured in the following way. Upon giving an overview of relevant previous research from AIR, IIR, and Patent Retrieval in Section 2, we explore the process of knowledge structure extraction based on the IPC in Section 3. In Section 4 an overview of our strategy concerning the integration of knowledge representations into the retrieval process is provided. Section 5 details the experimental setup chosen for the evaluation of our approach with respect to the prior art search task. In Section 6 we report and discuss the obtained results. In the final section we present our conclusions and provide an outlook of future extensions to this work. 1
http://www.wipo.int/classifications/ipc/en/
Knowledge Modeling in Prior Art Search
2
33
Related Work
In the following we will provide an overview of relevant research concerning Patent Retrieval and previous approaches of knowledge integration in IR. The majority of relevant retrieval research in the patent domain has been pioneered by the NTCIR series of evaluation workshops and further diversified and expanded by the CLEF [1] Intellectual Property (IP) Track 2009 [26]. A task related to the prior art search task is presented by the invalidity search run at NTCIR 5 [13], and 6 [18]). Invalidity searches are exercised in order to render specific claims of a patent, or the complete patent itself, invalid by identifying relevant prior art published before the filing date of the patent in question. As such, this kind of search, that can be utilized as a means of defense upon being charged with infringement, is related to prior art search. Likewise to the prior art search task of CLEF IP 09, the starting point of the task is given by a patent document, and a viable corpus consists of a collection of patent documents. The initial challenge with both tasks consists of the automatic formulation of a query w.r.t. a topic document. Commonly applied techniques [26] are based on the analysis of term frequency distributions as a means of identifying effective query term candidates. In addition to these techniques, the usage of bibliographical data associated with a patent document has been applied both for filtering and re-ranking of retrieved documents. Particularly the usage of the hierarchical structure of the IPC classes and applicant identities have been shown to be highly effective [13]. Concerning the integration of bibliographical data such as the IPC into retrieval, our work differs through its utilization of the IPC solely as a source for extracting term relation knowledge that is applied to mitigate the effect of vocabulary mismatch. As outlined in detail in Section 3 its aim lies in improving the query document matching process and it is not envisioned as a potential replacement, or exclusive of application of IPC based filtering or re-ranking methods. In a retrieval context where such information (i.e. IPC classification of the topic) is available these techniques could be applied on result listings returned by a retrieval setup as described in Section 5. As pointed out before, the majority of relevant research in IR stems from the subdomains of Associative IR (AIR) and Intelligent IR (IIR).In AIR the most commonly applied scheme of applying knowledge consists of the construction of conceptual graphs and the application of spreading activation algorithms [9] as a means of expanding user queries. In IIR, which is focusing on the exploration of the ’overlap of AI and IR’ [10], knowledge representations have been utilized in form of semantic networks [7] and hierarchies of retrieval subtopics [22]. A recent study [4] undertaken to evaluate the benefit of query (QE) and document expansion (DE) in the context of ad hoc-retrieval introduced two novel DE methods based on adding terms to documents in a process that is analogous to QE, and on regarding each term in the vocabulary as a query. The study concluded state of the art QE to be generally beneficial and corpus-based DE in its applied form not to be promising.
34
E. Graf et al.
Our work differs from these previously described approaches through its inherent focus on the patent domain, and by aiming at expanding documents with respect to their ’aboutness’ [17] as expressed through their classification with specific IPC codes, and the resulting ’grouping’ with related documents. In view of this, it can be interpreted as a form of meta-data based document expansion.
3
Constructing Knowledge Representations
This section initially recapitulates on the underlying motivation for the integration of knowledge modeling into patent retrieval. This will be followed by an overview of the knowledge representation utilized in this study. Finally key aspects of the IPC based knowledge extraction process are outlined. As noted before, our proposed integration of knowledge representations into patent retrieval is primarily aimed at increasing retrieval effectiveness in terms of recall. A common approach to reach this goal consists of conceiving strategies to mitigate problems associated with vocabulary mismatch. Furnas et al. coined this concept based on ’the fundamental observation’, ’that people use a surprisingly great variety of words to refer to the same thing’ [14]. Attempting to limit potential negative impact can be interpreted as aiming at detecting semantic relatedness of textual artifacts where it is not reflected through the mutual occurrence of terms. In its most trivial form this could be achieved through the expansion of queries or documents with corresponding synonyms. In a more complex form this can be interpreted as the attempt of mimicking the human ability to infer what documents ’are about’, and to base decisions concerning their relatedness on this ’aboutness’ [17]. This notion can best be illustrated through a basic example. Table 1 depicts a sample document collection consisting of the five documents A to E. In this example each document consists of only four terms. In the following, given the hypothetical query q ’matrix collagenase-3 arthritis’, we will illustrate potential rankings with respect to conventional best matching retrieval strategies and our envisioned knowledge expansion strategy. A result list returned by a best match retrieval function such as TF/IDF or BM25 would consist of the documents A,B,D, and E. Document A would be ranked at 1 as it matches two of the query terms. Rank 2 to 4 would fall to documents B,D, and E. As is evident through the chosen sample texts of the documents and the explicit relevancy statements in Table 1 such a ranking does not represent an optimal outcome. An optimal ranking would take the form A,B,C,D,E. The first three documents are related to ’arthritis’, and in this example deemed relevant to the query. A human expert in possession of the knowledge that ’rheumatoid’ and ’arthritis’ are both terms describing medical problems affecting joints and connective tissue, and that ’collagenase-3’ is a ’matrix metalloprotease’, might deduce such an optimal ranking based on the following argumentation. – A should be ranked first as it contains ’collagenase-3’ and ’arthritis’. – B should be ranked second as ’osteoarthritis’ is a subtype of ’arthritis’ and ’collagenase-3’ is a ’matrix metalloprotease’.
Knowledge Modeling in Prior Art Search
35
Table 1. Sample document collection. Matching query terms with respect to query ‘matrix collagenase-3 arthritis’ are highlighted in bold. Doc. A B C D E
arthritis antibody immunohistochemical chondrosarcoma machine
cartilage matrix nonfibrillated matrix matrix
collagenase-3 metalloproteinase rheumatoid metalloproteinase turing
specimen osteoarthritis metalloproteinase vitro zuse
Rel. Yes Yes Yes No No
– C should be ranked third since the term ’rheumatoid’ is closely related to ’arthritis’ and ’collagenase-3’ is a ’matrix metalloprotease’. – D should be ranked fourth since ’collagenase-3’ is a ’matrix metalloprotease’ and D therefore relates to one aspect of the query. While it is, as evidenced through the term ’chondrosarcoma’ relating to another medical condition, it should still be ranked higher than E that is referring to a completely different topic. Research from cognitive psychology indicates that such reasoning with respect to text is enabled through knowledge based term associating during the process of reading [28]. In order to enable an IR system to, in analogy to this, retrieve document C although it does not contain any of the query terms, two things would be necessary: Firstly the extraction of term relation knowledge (e.g. in this case mapping the relation between ’rheumatoid’ and ’arthritis’), and secondly the integration of such a knowledge representation with the documents in the collection (i.e. to allow for consideration of this relationship during the retrieval phase). With regard to the first point, concerning the question of choosing a suitable knowledge representation to benefit patent retrieval, research from cognitive psychology can provide additional guidance. Specifically the described hierarchical aspects [21] of human memory with respect to natural kinds (e.g. collagenase-3 is a matrix metalloprotease, a matrix metalloprotease is a protease, and proteases are enzymes) and artifacts (i.e. artificial concepts such as hard disk drive and tape being magnetic storage devices) seem relevant in consideration of the technical nature of the patent domain. This is further underlined by the reported role of categorical and hierarchical aspects of memory for inductive inference [8] and higher level extraction of meaning [23]. In view of this the choice of a hierarchical structure for storing knowledge seems sensible in regard to inference of relatedness. Such a structure is also well suited considering automatic knowledge extraction based on the available IPC information, as the IPC itself exhibits a hierarchical order. Following this notion, extracted knowledge from the IPC will be modeled in a hierarchical manner that represents specific elements via sets of descriptive terms. By choosing to represent elements in a bag-of-words fashion, as outlined in detail in Section 4, this representation allows to expand documents in a flexible way that enables direct integration with existing retrieval models in a way that is scalable to realistic collection sizes. Subsequently a methodology
36
E. Graf et al.
for the extraction of such an above outlined representation of technological term relatedness based on IPC classification code assignments to patent documents will be introduced. 3.1
IPC Based Knowledge Representation Extraction
This section is focused on providing an overview of the proposed knowledge representation extraction process. As part of this we will also describe in detail the process of generating representative term sets for specific IPC elements. The IPC is a hierarchical classification system comprised of roughly seventy thousand elements. These elements are spread over five main layers that are exemplary depicted in Figure 2 together with a sample IPC classification code. Each patent document is assigned to one or more elements within this hierarchy with respect to its described technical invention. To extract this knowledge, and allow for its representation in form of a hierarchical structure, a methodology comprised of 4 distinct steps is applied. Based on utilization of IPC codes found on patent documents, the methodology aims at representing elements of the IPC hierarchy by the most descriptive terms w.r.t the technological aspects covered by the documents filed to a common classification element. As will be subsequently described in detail the extraction process is based on the statistical analysis of two observed events: A pair of documents belonging to the same IPC element, and a pair of documents sharing a specific term. An overview of this process is depicted in Figure 1, and its four steps are listed below. 1. Document Pair Formation: For a given element E of the IPC hierarchy a representative set of N document pairs is formed out of the set of all documents assigned to this element. 2. Mutual Term Extraction: On completion of this process for each chosen pair of element E the set of all mutual terms is extracted. Requiring a term to occur in at least two documents belonging to E in order to be considered, represents the first selection within the task of extracting a set of terms representative for all documents of E. 3. LL Ratio Computation: To identify the mutual terms that are most representative of E we then apply the Log Likelihood (LL) ratio test on the basis of the extracted mutual terms of the N document pairs. 4. Term Selection: Finally a representative set of terms for the element E is selected by including all terms exhibiting a LL ratio score higher than a chosen threshold t. The above described procedure is then repeated until a representative set of terms has been generated for each element of the IPC. The LL ratio test performed in step (3) of our methodology is described in detail in the following subsection. 3.2
Extracting Representative Term Sets
The basic idea of our extraction approach is based on the identification of significant diversion in the statistical distribution of term occurrence frequencies
Knowledge Modeling in Prior Art Search
37
Fig. 1. Overview of the knowledge extraction process and a depiction of the representative vocabulary extracted for A62B17/04. The listed IPC description represents the complete descriptive text of the specific element.
Fig. 2. Exemplary overview of the IPC system and mapping of IPC code B81B7/04 to hierarchical level denominations
38
E. Graf et al.
within documents belonging to the same IPC element and documents in the rest of the corpus. In the following we will describe the applied process of estimating how much more likely the occurrence of a mutually shared term within a document pair belonging to the same element is in contrast to its occurrence in a pair in the rest of the collection. To this cause we devise a Log Likelihood Ratio test. Such tests have been widely applied to the task of collocation analysis due to good performance with respect to sparse distributions [6]. One advantage using likelihood ratios lies in their clear intuitive interpretation. For example, in our model a likelihood ratio of 900 expresses that the occurrence of a term within a pair of documents belonging to the same IPC element is 900 times more likely than its base rate of occurrence in document pairs in the rest of the collection would suggest. In the following we outline in detail how the Log Likelihood ratio test is applied w.r.t. our aim of extracting representative term sets. In the space W = {d1 d2 , ..., dj dk } we observe two possible events t and e: – Event t: A pair of documents dj dk mutually contains ti . – Event e: A pair of documents dj dk belongs to the same element. Based on this two hypothesis can be formulated: – Hypothesis 1 (Independence): H1: P (t|e) = p = P (t|¬e) – Hypothesis 2 (Dependence): H2: P (t|e) = p1 = p2 = P (t|¬e) With the log-likelihood ratio defined as: log λ =
L(H1) L(H2)
it can be computed as the fraction of two binomials b:
b(cte ,ce ,p)∗b(ct −cte ,n−ce ,p) log λ = log b(c . te ,ce .p1 )∗b(ct −cte ,n−ce ,p2 )
Where c designates the observed counts of the events t, e, and te obtained in the third step (3) of our extraction methodology, and n represents all possible document pairs that can be formed from all N documents contained in a collection: N! n = (N −2)!∗(2)! . With respect to our goal of identifying representative terms, if the mutual occurrence of a term t in a pair of documents belonging to the same element e results in a large LL Ratio, we deem this term to be representative of the class.
4
Integration Strategy
For the integration of the generated knowledge representations described in the previous section we utilize the concept of document fields. In this approach both the text of internal structural elements of a document such as the title, abstract, or passages, as well as external meta-data can be represented in form of a distinct field. This enables the computation of a separate score for each respective field during retrieval. These scores can then be aggregated under consideration of assigned weights with respect to the benefit of distinct fields to a retrieval task.
Knowledge Modeling in Prior Art Search
39
In this form the approach has been successfully applied to a variety of different domains using document internal (e.g. title and body of an e-mail) and external fields (e.g. anchor text for web documents [29]). Basing our integration strategies on this technique allows us to build up on the large amount of research concerning combination techniques of scores [11] for individual fields and allows us to utilize existing retrieval system functionality. In a similar fashion as HTML documents have been expanded with anchor text in previous research, we propose to expand each patent document in the corpus with representative sets of terms according to its classification within the IPC hierarchy. As exemplary outlined in Figure 3 for each document a field representing the text of the document itself, and a field for each hierarchy level can be created with respect to the assigned IPC code. As a consequence, indexing of the collection results in the creation of an index for each level of the IPC. This approach requires additional computational effort during the indexing phase, but enables the application of field-adapted retrieval models such as BM25F. BM25F [25] is a variant of the BM25 Okapi retrieval algorithm, that allows for the combination of scores from several fields in a way, that does not break the non-linear saturation of term frequency in the BM25 function. This form of aggregation has been shown to deliver strong results in field based experimentation in the news, web, and e-mail domain.
Fig. 3. Integration through mapping extracted knowledge structure hierarchy levels via field indices
In the following section we will now describe how this strategy has been implemented in our experimental setup. Further an overview of the test collection and its main task will be provided.
5
Experimental Setup
This section introduces the experimental setup that was applied to evaluate the integration of our knowledge representation. We fill first provide details concerning the corpus and the associated task of the CLEF IP 09 [26] test collection that was used in our experimentation. Following this we will outline the applied indexing process and provide details of the retrieval models that were applied.
40
5.1
E. Graf et al.
Test Collection
For the evaluation of our approach we used the CLEF-IP 09 collection that formed part of the CLEF evaluation workshop. The collection focuses on a patent retrieval task, and features thousands of topics that were created based on a methodology of inferring relevance assessments from the references found on patent documents [15]. The corpus of the collection consists of 1.9 million patent documents published by the European Patent Office (EPO). This corresponds to approximately 1 million individual patents filed between 1985 and 2000. As a consequence of the statutes of the EPO, the documents of the collection are written in English, French and German. The main task of the CLEF-IP 09 track test collection consists of the search for Prior Art. Performed both, by applicants and the examiners at patent offices, it is one of the most common search types in the patent domain. Three sets of topics labeled as S, M, and XL consisting of 500, 1000, and 10000 topics are provided for this task. Each topic consists of an information need in the form of a patent application document and a set of qrels specifying relevant documents for the application. Based on the text of the patent application, participants of the track were required to infer a query in order to retrieve a ranked list of relevant prior art documents. The inference of effective queries formed the main challenge of the task. In this study we will utilize a methodology applied by participants of the 2009 track that is based on identifying effective query terms based on document frequency [16]. 5.2
Retrieval Setup
Indexing of the collection is performed using the MG4J retrieval system [5]. For our document expansion strategy each document is indexed in the form of a set of field indices. While field index ’0’ represents the text of the document itself, knowledge in the form of representative terms is associated with one field index per hierarchy level. No form of stemming was applied. This decision was based on the fact that the corpus contains a large amount of technical terms (e.g. chemical formulas) and tri-lingual documents. In order to increase indexing efficiency, stop-wording based on a minimum stop-word list was applied. Based on this setup we apply the following retrieval models. As baseline for our experiments the BM25 model is applied to a full text index of the collection. BM25 has shown strong performance in the CLEFIP 09 prior art track [24]. For the knowledge expanded field indices the BM25F model [25] is applied. Essentially identical to BM25 it has shown very good performance in field based scenarios due to the ability to set a field specific normalization parameter b in addition to applying weighting of each index [29]. For the initial retrieval runs and optimization the large training topic set consisting of 500 query documents has been applied. The performance with optimized parameter sets was evaluated based on the medium sized (1000) topic set. As stated before the main challenge of prior art search initially consists of the automatic extraction of effective query terms based on a topic posed in form
Knowledge Modeling in Prior Art Search
41
of a several pages long patent application. State of the art automatic query formulation methods rely on choosing query terms with respect to the distribution of term features. One such feature that has been applied to the task of automatic query formulation consists of the global document frequency (df) of terms. It has been found, that effective queries can be formulated by including only those terms of a patent application that occur in a low percentage of the documents in the collection. A query selection parameter called percentage threshold is defined as df N ∗ 100, where N denotes the total number of documents in the collection. A percentage threshold of 0.5% therefore denotes, that all terms from a topic document are included in a query that appear in less than 0.5% of the documents in the collection. In light of this the experimental evaluation of the knowledge integration is divided into two parts: – Query Dependency Analysis: Induced by the lack of one definitive set of queries as for example encountered in Web domain based tracks such as HARD [3], a first step in estimating the benefit of knowledge integration consists of an evaluation of the performance of various knowledge representations with respect to a varied set of generated queries. This step is aimed at generally clarifying if, and in what ways, knowledge based expansion can impact the prior art retrieval task. – Parameter Set Optimization: In order to more precisely estimate the potential benefit of the integration, we propose to conduct parameter set optimization for query generation settings that have shown promising results in the first experimentation phase. The need to optimize the parameter sets of BM25F based retrieval attempts in order to allow for best performance is discussed and outlined in detail in [29]. The results of these investigations are outlined in the subsequent section.
6
Experimental Results and Discussion
In the following the results for both experimentation phases will be outlined. 6.1
Query Dependency Analysis
The results of the query dependent analysis with respect to three generated knowledge representations are depicted in Figure 4 and 5. The graphs outline performance in terms of Recall and MAP w.r.t. the association of knowledge representations extracted based on LL Ratio thresholds of 15, 300, and 900. A baseline is provided by the BM25 model operating on a full text index. In these initial experiments only the lowest level of the IPC (subgroup) is considered for the expansion. Evident from Figure 4 is that the expansion with subgroup level fields benefits recall over all query generation parameters. The observation of a positive effect on recall being retained among the full spectrum of query formulations
42
E. Graf et al.
is a very promising result. Since queries created with a df-threshold of 0.25 exhibit an average query term length of 103.838 in contrast to an average length of 259.548 terms for a 3.25 query-threshold, the applied set of queries represents not only a large spectrum with respect to length, but also in regard of the document frequency of the contained query terms. In light of this it seems reasonable to assume, that the observed positive effect and its robustness might also apply to related query generation methods such as TF/IDF, and potentially also to manually created prior art queries. Further it can be deduced from Figure 4 that the LLRT 15 based representation exhibits the highest amount of variance with regard to the observed performance. Responsible for these results may be application of a comparatively low threshold of 15 in order to select knowledge representation terms. Generally this will lead to the inclusion of more general terms within the representative term set of the modeled subgroup level elements. It seems likely that the inclusion of more general knowledge terms raises the probability of topic drift occurrence. The stricter term selection criteria set by LLRT likelihoods of 300 and 900 in contrast exhibit more robust performances. The negative impact of this observation becomes also evident by studying the MAP related performance shown in Figure 5. While the contained vocabulary of the LLRT 15 based knowledge representation seems still descriptive of the general technological aspects, as expressed in the higher recall with respect to the baseline, substantial noise seems to be introduced, resulting in a clearly visible negative impact on precision. This is not exhibited by the more strict LLRT300 and LLRT 900 knowledge representations, which again show lower variance in their observed performance. Promising with respect to precision is the exhibited strong MAP performance of the LLRT 300 and LLRT 900 based runs for percentage-threshold values of 0.5 and 0.75. In order to estimate their full potential benefit, linear optimization of the BM25 and BM25F parameter sets was performed based on the large training topic set (500) of the Clef-IP 09 collection. 6.2
Parameter Optimization for BM25F
Based on the above reported initial results a complete linear optimization of BM25 and BM25F parameters was performed for a percentage-threshold value of 0.75. Training of the parameters for BM25F followed the strategy of dividing the optimization of k1, and index specific b and w parameters into several smaller optimization tasks as outlined in Zaragoza et al. [29]. Table 2 lists the results. As can be seen from the table the optimization results in a much improved performance for the knowledge expanded BM25F runs. MAP as well as recall are substantially and statistically significantly increased for both the LLRT300 and LLRT900 based knowledge representation. This constitutes an especially promising result in light of the sparse exploration with respect to optimization of the LLRT threshold space.
Knowledge Modeling in Prior Art Search
43
Fig. 4. Recall over various query length and LLRT thresholds for 1 field expansion
Fig. 5. MAP over various query length and LLRT thresholds for 1 field expansion
44
E. Graf et al.
Table 2. Optimized results BM25 baseline versus BM25F knowledge expansion. b1 and w1 constitute the parameter values for the full text index; b2 and w2 represent parameters for the subgroup based knowledge field (** strong statistical significance). Query 0.75 BM25 BM25F,LLRT300 BM25F,LLRT900
7
MAP 0.1036 0.1073 0.1091
% change / 3.57** 5.31**
Recall 0.5377 0.5887 0.5819
% change / 9.48** 8.22**
Bpref 0.5544 0.6083 0.6038
k1 1.2 1.0 0.2
b1 0.4 0.4 0.4
b2 / 0.65 0.7
w1 / 1.0 1.0
w2 / 0.3 1.0
Conclusion and Future Outlook
A first study of modeling knowledge within the task of prior art patent retrieval was presented. Initial results are very promising as it is shown that the proposed knowledge association is beneficial in terms of recall, and very robust with regard to query variation. Given prior optimization of the BM25F parameter space the knowledge association results in significant improvement of both recall and precision in comparison to an in the same manner optimized BM25 baseline. This is specifically encouraging, since in the introduced work only the lowest level of the extracted knowledge representation, corresponding to the subgroup IPC hierarchy level, has been utilized. Integration of higher hierarchy levels constitutes a logical next step and could potentially result in further improvements. Moreover a more fine-grained exploration of the term extraction parameters, and the application of varying methods for representative term selection, merit extensive additional investigation. Further an exploration of n-gram representations, proximity, and document structure exploitation within the extraction and retrieval process should be considered. Finally an evaluation of the potential benefit of the introduced knowledge representations with respect to other tasks such as classification and clustering and the feasibility of applying the introduced techniques to other domains form interesting long term aspects of this research.
Acknowledgments The authors would like to thank Matrixware Information Services2 and the Information Retrieval Facility3 (IRF)for their support of this work. Mounia Lalmas is currently funded by Microsoft Research/Royal Academy of Engineering.
References 1. The Cross-Language Evaluation Forum (CLEF) 2. Guidelines for Examination in the European Patent Office (December 2007) 2 3
http://www.matrixware.com http://www.ir-facility.org/
Knowledge Modeling in Prior Art Search
45
3. Allan, J.: HARD track overview in TREC 2004: High accuracy retrieval from documents. In: Proceedings of the thirteenth Text REtrieval Conference (TREC 2004), no. Ldc, NIST, pp. 1–11 (2004) 4. Billerbeck, B., Zobel, J.: Document expansion versus query expansion for ad-hoc retrieval. In: Proceedings of the 10th Australasian Document Computing Symposium (2005) 5. Boldi, P., Vigna, S.: MG4J at TREC 2005. In: The Fourteenth Text REtrieval Conference (TREC 2005) Proceedings, number SP, Citeseer, vol. 500, p. 266 (2005) 6. Bordag, S.: Elements of Knowledge-free and Unsupervised lexical acquisition. PhD thesis (2007) 7. Cohen, P.: Information retrieval by constrained spreading activation in semantic networks. Information Processing & Management 23(4), 255–268 (1987) 8. Coley, J.: Knowledge, expectations, and inductive reasoning within conceptual hierarchies. Cognition 90(3), 217–253 (2004) 9. Crestani, F.: Application of spreading activation techniques in information retrieval. Artificial Intelligence Review 11(6), 453–482 (1997) 10. Croft, W.B.: Approaches to intelligent information retrieval. Information Processing & Management 23(4), 249–254 (1987) 11. Croft, W.: Combining approaches to information retrieval. Advances in information retrieval 7, 1–36 (2000) 12. Davis, R., Shrobe, H., Szolovits, P.: What is a knowledge representation? AI magazine 14(1), 17 (1993) 13. Fujii, A., Iwayama, M., Kando, N.: Overview of Patent Retrieval Task at NTCIR-5. In: Proceedings of NTCIR-5 Workshop Meeting (2005) 14. Furnas, G., Landauer, T., Gomez, L., Dumais, S.: The vocabulary problem in human-system communication. Communications of the ACM 30(11), 971 (1987) 15. Graf, E., Azzopardi, L.: A methodology for building a patent test collection for prior art search. In: Proceedings of the Second International Workshop on Evaluating Information Access, EVIA (2008) 16. Graf, E., Azzopardi, L., van Rijsbergen, K.: Automatically Generating Queries for Prior Art Search 17. Hutchins, W.: The concept of aboutness in subject indexing. Aslib Proceedings 30(5), 172–181 (1978) 18. Iwayama, M., Fujii, A., Kando, N.: Overview of Classification Subtask at NTCIR-6 Patent Retrieval Task. In: Proceedings of NTCIR-6 Workshop Meeting, pp. 366– 372 (2007) 19. Jing, Y., Croft, W.: An association thesaurus for information retrieval. In: Proceedings of RIAO, vol. 94, Citeseer, pp. 146–160 (1994) 20. Kintsch, W.: The role of knowledge in discourse comprehension: a constructionintegration model. Psychological review 95(2), 163–182 (1988) 21. Markman, E., Callanan, M.: An analysis of hierarchical classification, pp. 325–366. Erlbaum, Hillsdale (1980) 22. McCune, B., Tong, R., Dean, J., Shapiro, D.: RUBRIC: a system for rule-based information retrieval. Readings in information retrieval 9, 445 (1997) 23. Medin, D.L., Rips, L.J.: Concepts and categories: Memory, meaning, and metaphysics. Cambridge Univ. Press, Cambridge (2005) 24. Piroi, F., Roda, G., Zenz, V.: CLEF-IP 2009 Evaluation Summary (2009) 25. Robertson, S., Zaragoza, H., Taylor, M.: Simple BM25 extension to multiple weighted fields. In: CIKM 2004: Proceedings of the thirteenth ACM international conference on Information and knowledge management, pp. 42–49. ACM Press, New York (2004)
46
E. Graf et al.
26. Roda, G., Tait, J., Piroi, F., Zenz, V.: CLEF-IP 2009: retrieval experiments in the Intellectual Property domain. In: CLEF working notes 2009 (2009) 27. Salton, G.: Associative document retrieval techniques using bibliographic information. Journal of the ACM (JACM) 10(4), 440–457 (1963) 28. Wharton, C., Kintsch, W.: An overview of construction-integration model. ACM SIGART Bulletin 2(4), 169–173 (1991) 29. Zaragoza, H., Craswell, N., Taylor, M., Saria, S., Robertson, S.: Microsoft Cambridge at TREC-13: Web and HARD tracks. In: Proceedings of TREC 2004, Citeseer (2004)
Combining Wikipedia-Based Concept Models for Cross-Language Retrieval Benjamin Roth and Dietrich Klakow Spoken Language Systems, Saarland University, Germany
[email protected],
[email protected]
Abstract. As a low-cost ressource that is up-to-date, Wikipedia recently gains attention as a means to provide cross-language brigding for information retrieval. Contradictory to a previous study, we show that standard Latent Dirichlet Allocation (LDA) can extract cross-language information that is valuable for IR by simply normalizing the training data. Furthermore, we show that LDA and Explicit Semantic Analysis (ESA) complement each other, yielding significant improvements when combined. Such a combination can significantly contribute to retrieval based on machine translation, especially when query translations contain errors. The experiments were perfomed on the Multext JOC corpus und a CLEF dataset. Keywords: Latent dirichlet allocation, explicit semantic analysis, crosslanguage information retrieval, machine translation.
1
Introduction
Dimensionality reduction techniques have traditionally been of interest for information retrieval as a means of mitigating the word mismatch problem. The term concept model is more general than dimensionality reduction and denotes a mapping from the word space to another representation that provides a smoother similarity measure than word statistics and is often induced from cooccurrence counts on paragraph or document level. Such a representation may for example be obtained by matrix approximation [7], by probabilistic inference [20] or techniques making use of the conceptual structure of corpora such as Wikipedia [9]. Cross-language information retrieval can be viewed as an extreme case of word mismatch, since for any two texts the vocabulary is in general disjoint if the languages are not the same. In order to have a cross-lingual similarity measure, it is necessary that concept spaces of different languages are aligned, which is often achieved by extending the notion of co-occurrence to pairs of translated or thematically related texts. While some work has been done on multilingual concept modeling [5, 8, 15–19], often the focus is on one method and a comparison with other methods is missing (but see [4]). One reason for this might be that concept models require adaptations for multilinguality that do not seem to be easily implemented. We will show that the adaptations can in H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 47–59, 2010. c Springer-Verlag Berlin Heidelberg 2010
48
B. Roth and D. Klakow
fact be minimal and on the data side only. Another question that has not been investigated so far is how different multilingual concept models can contribute to each other and how they can be combined with word models, an approach that is standard for the monolingual case. The rest of the paper is structured as follows: In section 2 we summarize standard and multilingual Latent Dirichlet Allocation, a probabilistic concept model, and Explicit Semantic Analysis, a more recent explicit concept model. In section 3 we outline how we think the forementioned methods can be applied more profitably. Our experiments are described in section 4. In section 4.1 we explore both LDA and ESA on a mate retrieval task and observe much better results for LDA than reported so far and obtain consistent improvement for their combination. In section 4.2 we show how concept models can improve word-based cross-language retrieval. We end with an outlook on future work and conclusion.
2 2.1
Related Work Latent Dirichlet Allocation
Probabilistic Model. Latent Dirichlet Allocation (LDA) [2, 10, 20] is a latent variable model that gives a fully generative account for documents in a training corpus and for unseen documents. Each document is characterized by a topic distribution, words are emitted according to an emission probability dependent on a topic. The main difference to pLSA [11, 12] is that both topic distributions and word emission distributions are assumed to be generated by Dirichlet priors. It is common to parameterize the Dirichlet prior uniformly with parameters α = α1 = · · · = αT and to direct only the “peakiness” of the multinomials drawn from it. The LDA model describes the process of generating text in the following way: 1. For all k topics generate multinomial distributions ψ (zk ) = p(wj |zk ) ∼ Dir(β). 2. For every document d: (a) Generate a multinomial distribution θ(d) = p(zk |d) ∼ Dir(α). (b) Generate a document length, and topics zi ∼ θ(d) for every position i in the document. (c) Generate words wi ∼ ψ (zi ) for every position in the document. Usually, no generative account for the length of the document is given. Practical Issues. The first approach to estimate such a model [2] was to represent and estimate ψ and θ explicitly, resulting in different inference tasks to be solved and combined. Later approaches concentrate on getting a sample of the assignment of words to topics instead by Gibbs sampling [10]. To determine the similarity between two documents, one can compare either their sampled topic vectors or the probability vectors obtained from them [10].
Combining Wikipedia-Based Concept Models for Cross-Language Retrieval
49
When other variational EM estimation techniques are applied, also other parameter vectors might be available and used [2, 5]. The comparison between these vectors can be done either by taking the cosine similarity of their angles or by using probability divergence measures. For language model based information retrieval, one is interested in the probability of a query, given a document. Wei and Croft [22] interpolate a language model based on LDA with a unigram language model directly estimated on the document. Whenever a sampling technique is used, one wants to be sure that the estimates are stable. This could be a problem when only one topic is sampled per position for short documents or queries. The most natural way to overcome this problem is to average the results of several sampling iterations. Multilingual LDA. LDA has been extended for several languages [16] (see also [15] for an investigation of the semantic clustering of this model and [6] for cross-lingual news-linking with it). Most of the components remain the same, the main difference is that for each language ls a different word emission distribution ψls is assumed. Depending on the language of a position in a document, a word is generated conditioned on the corresponding distribution. The model does not use the topic variable to estimate the language, as it could be the case for a monolingual model applied to a multilingual document without any adaptions on either the data or the model side. A theoretically sound model does not mean that it also provides a good bridging between two languages. It is crucial [15] how many “glue documents”, i.e. documents that indeed have counterparts in all compared languages, are available: Although the model does not try to capture the language, the latent variables might tend to structure the monolingual spaces without semantic alignment between the languages (imagine a multilingual text collection with only one glue document as an extreme case). 2.2
Explicit Semantic Analysis
Explicit Semantic Analysis (ESA) [5, 9, 17–19] is another scheme to overcome the word-mismatch problem. In ESA, the association strength of words to the documents in a training collection is computed, and vectors of these asscociations are used to represent the words in the semantic space spanned by the document names. These word representations can be used to compare words, they can also be combined for a comparison of texts. See [1] for the relation to the Generalized Vector Space Model [23]. Formalization. Several formalizations are possible in this setting. The fundamental ingredients that determine an implementation are: – Word vectors: For every word w a vector w, indicating its association strength to the documents is computed. – Text vectors: For a new document (or query) d a vector representation d is computed from the word vectors. – Similarity function: Giving two documents d1 and d2 the similarity is computed using a similarity function on their text vectors.
50
B. Roth and D. Klakow
The word vectors that were used in [9] and found to be optimal in [19] are obtained by taking the respective columns of the tf.idf-weighted document-term matrix A of the training collection. We use this choice of word vectors in our experiments. For the text similarity, several settings have been proposed. In [9] a weighting of the word vectors is used, they are multiplied with scalars equal to, again, an tf.idf weighting of the terms, and then summed. Sorg and Cimiano [19] explore further combination schemes, including the sum of the elements of either the multiset (considering term frequency) or of the set (not considering term frequency) of word vectors. They find that the set combination works best, yielding preliminary text vectors of the form: ˆ= d
w
w∈d
It is beneficial to truncate the resulting vectors at a certain threshold. The thresholding that turned out to be most successful [19] was to retain the 10000 ˆ Again, we use this parametrization in biggest non-zero values of the vectors d. all following experiments that involve ESA. As a similarity function the cosine is suggested [19] and used by us. Multilingual ESA. The application of this model in a multilingual setting is straightforward. For L languages consider document term matrices A(1) · · · A(L) . Construct the matrices in a way that the document rows correspond. For all lan(·) guages each of the rows An contains documents about the same topic across the languages. Therefore only documents can be included that are available in all of the considered languages. For each document the mapping to text vectors is performed using a monolingual matrix corresponding to its language. As the documents are aligned, similarities can be computed across languages. Because the relative frequency is used in the tf.idf-weighting, all documents are normalized and no bias occurs for documents longer in one language than in another.
3
Making Use of Concept Models for CLIR
Making Use of LDA. In our experiments we want to generalize the intuition given for the multilingual LDA model [15]: not only should a large number of glue documents exist, good bridging documents should optimally be of equal length. Experimentation with a small fraction of the training data indicated that the multilingual LDA model and a monolingual LDA model on documents normalized to equal length on both language sides yield about the same performance, while a monolingual model on unnormalized data performs considerably worse. Moreover, highly optimized and parallelized toolkits that allow us to perform training on all Wikipedia articles have only been developed for standard LDA. We believe, therefore, that it is a promising approach to normalize the data suitably to be processed with a standard monolingual LDA model.
Combining Wikipedia-Based Concept Models for Cross-Language Retrieval
51
Wikipedia is used as a parallel training corpus: corresponding articles are concatenated, their length is normalized to match the length of the counterpart in the other language. We propose two methods of length normalization: First, to cut off every document at a certain length. Second, to retain for the longer language side of an article only a random sample of size equal to the smaller language side. A resizing with a scalar is not possible because the sampling process requires integer counts. The vocabulary is uniquely identified for every language by attaching suitable prefixes (en , de ) to the words. Similarity is measured between two texts after inference by taking the cosine between their vectors of sampled topic statistics. Making Use of ESA. ESA is applied as described in section 2.2, which is as closely as possible as reported in [19]. Making Use of Machine Translation. The machine translation retrieval model translates the queries with a standard Moses [14] translation model trained on Europarl [13]. Translated queries and text are then compared by the cosine of their tf.idf-weighted word-vectors. For the document-term matrix D of the target collection with N documents, we use the commonly used weighting function Dn,w log N w ∈W Dn,w
tf.idf (w, n) =
n =1
N 1Dn ,w >0
Here, 1Dn ,w >0 is an indicator that equals to one if word w has appeared in document dn at least once, and that equals to zero otherwise. Model Combination. We use a simple scheme to combine models by concatenating L2 -normalized vectors. Let u be a m-dimensional vector and v be a n-dimensional vector which represent the same document in two different models. Then the model combination with interpolation weight α represents this document by an m + n-dimensional vector w: u α i if 1 ≤ i ≤ m , wi = v|u| (i−m) (1−α) otherwise. |v| This way any models can be combined as long as they are in vector representation. This the case for all models mentioned above, although they rely on very different principles. Similar combinations have been proven effective in the case of pLSA for monolingual retrieval [11].
4
Experiments
We are using two datasets, the Multext JOC corpus for the task of finding translations, and a CLEF 2000 query-based retrieval collection. These datasets and the experiments are described in detail in the next section.
52
B. Roth and D. Klakow
4.1
Mate Retrieval on Multext JOC
There is only one publication [5] known to us that compares LDA for crosslanguage information retrieval with ESA. Interestingly, our experiments on the same dataset will suggest a distinctively different assessment of the potential of LDA for the same task. The basis of the evaluation is the Multext JOC corpus1 which consists of 3500 questions to the European Parliament and of answers to these questions. As in [5] we use the concatenation of a question together with its answer as a query in one language to search the collection of translations in another language for its counterpart. Our experiments were done with English as the query language and German as the target language. Only preprocessing steps that are clear and easy to reproduce were performed. Exactly those questions were retained that to which an answer was assigned and had the same id in English and German. This resulted in a set of 3212 texts in each language, 157 more than were used in [5]2 . Sequences of characters in Unicode letter blocks were considered words. Words with length = 1 or length > 64 and words contained in the Snowball stopword list were ignored. All other words were stemmed with the publicly available Snowball stemmer3 . In contrast to [5], no compound splitting was done. For the training collection all pairs of Wikipedia articles4 were used that have bidirectional direct cross-language references. All markup was stripped off by using the same filter as in a publicly available ESA implementation5 . Wikipedia articles of less than 100 words in either language were ignored and words with a Wikipedia document frequency of 1 were filtered out. The final training corpus consists of 320000 bilingual articles. Performance of retrieval was measured in mean reciprocal rank (mrr). The ESA retrieval experiment was performed using the same parametrization as discribed before and the result of [5] was reproduced to a difference of 1% (in our experiments we obtained a score of mrr = 0.77 compared with mrr = 0.78). As for the LDA experiments, we were interested in the effect of length normalization of the training documents. We compare two methods: First, every document was cut off at a length of 100 words. Second, the method of down sampling the longer language side to the length of the smaller one was applied. We marked each word with a prefix indicating its language and retained a vocabulary size of roughly 770 thousand and 2.1 million for the cut-off method and for the downsampling method respectively. Both training collections were embedded with 125, 250 and 500 dimensions, and additionally with 1000 dimensions for the cut-off corpus (the vocabulary size was the limiting factor with respect to our computing facilities). The Google plda package [21] was used with the sug50 gested parameters (α = and β = 0.01). With the trained model, topics #topics were inferred for the monolingual Multext documents. In order to get a stable 1 2 3 4 5
http://www.lpl.univ-aix.fr/projects/multext The exact document selection criterion of their experiments is unknown to us. http://snowball.tartarus.org/ We used the German snapshot of 2009-07-10 and the English snapshot of 2009-07-13. http://code.google.com/p/research-esa/
Combining Wikipedia-Based Concept Models for Cross-Language Retrieval
53
0.7 lda, cutoff lda, downsampled 0.65
0.6
mrr
0.55
0.5
0.45
0.4
0.35
0.3 100
200
300
400
500
600
700
800
900
1000
dim
Fig. 1. Performance of LDA models estimated with dimensions (=numbers of topics) equal to 125, 250, 500 and 1000. Thick lines indicate combinations of all models up to the dimension on the x−axis. Drops in the perfomance curve on single points may be due to local sampling optima. Table 1. Performance of LDA on Multext LDA method number of topics Cimiano et al. 500 length downsampling 500 length cut-off 500 length downsampling 125 + 250 + 500 length cut-off 125 + 250 + 500 + 1000
mrr .16 .42 .53∗∗ .55∗∗ .68∗∗
estimate, the statistics of 50 sampling iterations were averaged. Similarity in the LDA setup was measured by taking the cosine similarity between the sampling statistics. A drastic improvement over non-normalized LDA can be observed: while [5] report a score of mrr = 0.16 for their 500-dimensional LDA model, we get mrr = 0.53 with the cut-off corpus. We suppose that the reason for this difference is that a non-multilingual LDA model applied to a comparable corpus estimates the predominant language of a document rather than its semantic content. Another improvement can be observed by combining the results of different models, a technique that is usually applied for pLSA [12]. In this case, the samping statistics of runs with different dimensional models were L2 -norm normalized and concatenated without further weighting. This yielded a score of mrr = 0.68 for the cut-off model, showing performance in the same order of magnitude as ESA. Figure 1 and Table 1 give a survey of the results obtained with LDA. Scores significantly better than in the respective line above having
54
B. Roth and D. Klakow 0.9
α lda + (1-α) esa
0.85
mrr
0.8
0.75
0.7
0.65 0
0.2
0.4
0.6
0.8
1
α
Fig. 2. Combining LDA and ESA on the Multext corpus. The improvement over ESA alone is significant with p 0.005 for .1 ≤ α ≤ .7.
p 0.005 in the paired t-test are marked with ∗∗. (Of course we could not test against scores reported in [5], for lack of the original numerical data.) In order to determine how different the ESA and the LDA models are and how much can they contribute to each other, we combined the vector representations of both models by different interpolation factors 0 ≤ α ≤ 1. A stable improvement in performance with maximum mrr = 0.89 was achieved for giving the cut-off LDA model a weight of 0.4 and the ESA model a weight of 0.6. See Figure 2. 4.2
Query-Based Retrieval with CLEF2000
Mate retrieval experiments can be criticized as being an unrealistic retrieval scenario. Therefore, a second evaluation was done on the CLEF6 German-English ad-hoc track of the year 2000. The target corpus consists of about 110000 English newspaper articles together with 33 German queries for which relevant articles could be pooled. For our experiments the title and description fields of the queries were used and the narrative was ignored. A common strategy for cross-language retrieval is first to translate the query and then to perform monolingual retrieval. While the translation process would have taken prohibitively long for the Multext corpus, we performed query translation on the CLEF2000 queries with a standard Moses translation model trained on Europarl. Retrieval with the translated queries was done by comparing the cosine of the tf.idf-weighted word-vectors. 6
See http://www.clef-campaign.org/; The evaluation packages are available via the ELRA catalogue http://catalog.elra.info, The CLEF Test Suite for the CLEF 2000-2003 Campaigns, catalogue reference: ELRA-E0008.
Combining Wikipedia-Based Concept Models for Cross-Language Retrieval
55
0.26 0.24 0.22 0.2
map
0.18 0.16 0.14 0.12 0.1 α esa α lda, cutoff α (lda + esa) mt tfidf
0.08 0.06 0
0.2
0.4
0.6
0.8
1
α
Fig. 3. Interpolation of concept models (having interpolation weight α) with machine translation (weighted 1 − α) on CLEF2000. The improvement given by the LDA-ESA is significant with p < 0.005 for .2 ≤ α ≤ .6.
We evaluated both the machine translation model and the concept models trained on Wikipedia. In addition to the most commonly used mean average precision score (map) we also evaluated by geometric mean average precision (gmap), which rewards stable results for hard queries. The ESA and the cut-off LDA models with dim = 500 perform equally well for map, while the combination of LDA dimensions gets a considerably better score. This is in contrast to the findings in the mate-retrieval setup. The reason for that, we suspect, may be that the parameters of ESA have been found in order to optimize such a setting. For gmap, LDA consistently outperforms ESA. For the combined LDA-ESA model a slight improvement could be observed using the combination with α = .5 (henceforth referred to as combined concept model ). The machine translation model (map = .203) performed better than LDA and ESA. When interpolated with the machine translation model all three concept models (LDA, ESA and combined) achieved improvements. The biggest and most stable improvement was achieved by the interpolation of machine translation and combined concept model yielding a score up to map = .245 with equal weight for the concept model and the machine translation model. Figure 3 and Table 2 show an overview of the results. Scores significantly better than in the respective line above, having p < 0.05 and p < 0.005 in the paired t-test, are marked with ∗ and ∗∗ respectively. The evaluation results of the combined system lie well within the range of the participating systems in CLEF 2000 for the same track7 . Particularly, no manually edited lexicon and no compound-splitter is used in our case. 7
For licensing reasons, we are not allowed to make a direct comparison, but see [3] for a survey.
56
B. Roth and D. Klakow Table 2. Query-based retrieval on CLEF2000 method parameters map gmap ESA .071 .003 LDA, cut-off d = 500 .071 .010 LDA, cut-off d = all .155∗ .043∗ LDA+ESA α = .5 .176 .054 MT tf.idf .203 .061 concept+MT α = .5 .245∗∗ .128∗∗
Table 3. Improvement of map through the ESA+LDA concept component compared with error rates. The 4th column indicates the change in comparison to machine translation alone (3rd column). The 5th column contains one + per unknown word, the last column the percentage of junkwords per query. id 18 9 7 4 26 11 12 17 5 15 38 21 14 28 33 16 13 40 31 32 20 36 22 37 19 1 30 24 10 29 39 34 3
query title Unf¨ alle von Brandbek¨ ampfern Methanlagerst¨ atten Doping und Fußball ¨ Uberschwemmungen in Europa Nutzung von Windenergie Neue Verfassung f¨ ur S¨ udafrika Sonnentempel Buschbr¨ ande bei Sydney Mitgliedschaft in der Europ¨ aischen Union Wettbewerbsf¨ ahigkeit der europ¨ aischen... R¨ uckf¨ uhrung von Kriegstoten Europ¨ aischer Wirtschaftsraum USA-Tourismus Lehrmethoden f¨ ur nicht-englischsprachige... Krebsgenetik Die Franz¨ osische Akademie Konferenz u ¨ber Geburtenkontrolle Privatisierung der Deutschen Bundesbahn Verbraucherschutz in der EU Weibliche Priester Einheitliche europ¨ aische W¨ ahrung Produktion von Oliven¨ ol im Mittelmeerraum Flugzeugunf¨ alle auf Start- und Landebahnen Untergang der F¨ ahre Estonia Golfkriegssyndrom Architektur in Berlin Einsturz einer Supermarktdecke in Nizza Welthandelsorganisation Krieg und Radio Erster Nobelpreis f¨ ur Wirtschaft Investitionen in Osteuropa oder Rußland Alkoholkonsum in Europa Drogen in Holland
scores errors mt ap Δap unknowns junk(%) 0.000 ap = 0.033 + 0.71 0.000 ap = 0.017 ++ 0.50 0.004 +361.36% 0.33 0.007 +209.09% 0.17 0.071 +167.40% 0.13 0.093 +111.95% 0.09 0.009 +109.57% + 0.50 0.138 +96.09% ++ 0.29 0.060 +89.38% 0.08 0.177 +88.51% 0.10 0.045 +86.84% +++ 0.33 0.165 +73.38% 0.22 0.015 +67.72% + 0.17 0.072 +61.43% + 0.29 0.382 +52.46% + 0.38 0.084 +46.84% 0.33 0.137 +46.68% 0.17 0.015 +46.10% 0.25 0.222 +46.03% 0 0.234 +34.99% 0.22 0.218 +24.13% 0 0.269 +23.29% 0.11 0.094 +19.76% + 0.22 0.926 +4.13% 0.17 0.704 +1.44% 0.25 0.651 +1.27% 0.17 1.000 −.01% + 0.13 0.516 −6.94% 0 0.015 −11.47% 0 0.072 −11.99% 0 0.041 −16.99% 0.15 0.060 −24.18% 0 0.185 −55.66% 0
Combining Wikipedia-Based Concept Models for Cross-Language Retrieval
57
Error Analysis. A querywise error analysis is difficult as the inner workings of quantitative methods are often opaque to human analysis. However, the machine translation output is the most contributing source and it is accessible to examination. We sorted the machine translation output by how much it profited by the concept models in the best performing setting. In Table 3 we report the score that is obtained by machine translation and the increase when combined with the concept models. We analyzed how often a word was obviously unknown by the machine translation system trained on Europarl and therefore wrongly just copied over. It would be possible to recognize this type of error automatically. In addition, for every translated query we counted how many words in it had no semantic meaning related to the purpose of the query and were therefore useless (these words are hence called junk words). Junk words are, for example, function words not filtered by the stopword list, machine translation errors of several kinds and artefacts from the query formulation (e.g. “Gesucht sind Dokumente, die ... liefern” in the description part of query 4). The junk word error type would be more difficult to detect. Although the analyzed data basis is small, we conjecture that the concept model makes such queries more robust which induce one of the two errors, while it might be less useful where a good translation is present and the terms are weighted well: In the cases where the concept models contributed there were, on average, 0.53 unknown words for machine translation and 0.24% junk words, in contrast to 0.14 unknown words and 0.04% junk words in the cases where the concept model decreased the score. For future experiments it might be interesting to test whether a trade-off weighting between translation and concept model conditioned on a reliability score of the machine translation improves performance.
5
Future Work
Our experiments have been done in a vector space retrieval framework, because this made model combination straightforward and allowed direct comparison to other experimental setups reported in the literature. However, it would be interesting to include the cross-lingual LDA-model in a language model setup, similar to [22] in the monolingual case. The inclusion of ESA in such a model would be more complicated. We also leave experiments with more language pairs for future work. While we have indicators to when concept models are beneficial for word-based retrieval, the effects that take place when combining LDA and ESA would be more difficult to uncover. Research in this direction could focus on the influence of term-weighting in ESA and on that of disambiguation by context in the case of LDA.
6
Conclusion
For cross-language retrieval, it is essential to have a cross-language bridge that is immediately available, in the best case for many languages and with up-to-date
58
B. Roth and D. Klakow
vocabulary. To work in practice, a method to extract such bridging information should not rely on specialized algorithms, but on approved techniques for which robust implementations exist that run on large computing facilities. In this work we have shown that Wikipedia is a valuable bridging source and that standard (monolingual) LDA can be applied to multilingual training data when care is taken for suitable length normalization. Thus, we get an improvement of 325% mrr compared to a non-competitive score previously reported for LDA with non-normalized Wikipedia data on the Multext corpus. A second finding is that simple model combinations reliably increase performance. For retrieval of document translations (mate retrieval) the combination of ESA and LDA achieves scores 16% mrr better than reported so far for ESA alone. Concept models based on Wikipedia are also complementary to word based retrieval using machine translation output, here we observe an increase by 21% map compared to a machine translation base-line. While ESA performs better than LDA for mate retrieval, this ranking is reversed for the more relevant task of query-based retrieval. This may be because commonly used ESA-parameters have been tuned for retrieval of document translations.
Acknowledgements We want to thank Matt Lease and the Spoken Language Systems QA-group for helpful discussions.
References [1] Anderka, M., Stein, B.: The ESA retrieval model revisited. In: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pp. 670–671. ACM, New York (2009) [2] Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. The Journal of Machine Learning Research 3, 993–1022 (2003) [3] Braschler, M.: CLEF 2000-Overview of results. In: Peters, C. (ed.) CLEF 2000. LNCS, vol. 2069, p. 89. Springer, Heidelberg (2001) [4] Carbonell, J.G., Yang, Y., Frederking, R.E., Brown, R.D., Geng, Y., Lee, D.: Translingual information retrieval: A comparative evaluation. In: International Joint Conference on Artificial Intelligence, Citeseer, vol. 15, pp. 708–715 (1997) [5] Cimiano, P., Schultz, A., Sizov, S., Sorg, P., Staab, S.: Explicit vs. Latent Concept Models for Cross-Language Information Retrieval. In: Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI 2009 (2009) [6] De Smet, W., Moens, M.F.: Cross-language linking of news stories on the web using interlingual topic modelling. In: The 2nd Workshop on Social Web Search and Mining, SWSM 2009 (2009) [7] Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. Journal of the American society for information science 41(6), 391–407 (1990)
Combining Wikipedia-Based Concept Models for Cross-Language Retrieval
59
[8] Dumais, S.T., Letsche, T.A., Littman, M.L., Landauer, T.K.: Automatic crosslanguage retrieval using latent semantic indexing. In: AAAI Spring Symposuim on Cross-Language Text and Speech Retrieval, pp. 115–132 (1997) [9] Gabrilovich, E., Markovitch, S.: Computing semantic relatedness using wikipediabased explicit semantic analysis. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 6–12 (2007) [10] Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proceedings of the National Academy of Sciences 101(90001), 5228–5235 (2004) [11] Hofmann, T.: Probabilistic latent semantic indexing. In: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 50–57. ACM, New York (1999) [12] Hofmann, T.: Unsupervised Learning by Probabilistic Latent Semantic Analysis. Machine Learning 42(1), 177–196 (2001) [13] Koehn, P.: Europarl: A parallel corpus for statistical machine translation. In: MT summit, vol. 5 (2005) [14] Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al.: Moses: Open source toolkit for statistical machine translation. In: Annual meeting-association for computational linguistics, vol. 45 (2007) [15] Mimno, D., Wallach, H.M., Naradowsky, J., Smith, D.A., McCallum, A.: Polylingual Topic Models. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pp. 880–889 (2009) [16] Ni, X., Sun, J.T., Hu, J., Chen, Z.: Mining multilingual topics from wikipedia. In: Proceedings of the 18th international conference on World wide web, pp. 1155– 1156. ACM, New York (2009) [17] Potthast, M., Stein, B., Anderka, M.: A wikipedia-based multilingual retrieval model. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, p. 522. Springer, Heidelberg (2008) [18] Sorg, P., Cimiano, P.: Cross-lingual information retrieval with explicit semantic analysis. In: Working Notes of the Annual CLEF Meeting (2008) [19] Sorg, P., Cimiano, P.: An Experimental Comparison of Explicit Semantic Analysis Implementations for Cross-Language Retrieval. In: Proceedings of the 14th International Conference on Applications of Natural Language to Information Systems, NLDB 2009 (2009) [20] Steyvers, M., Griffiths, T.: Probabilistic topic models. Handbook of Latent Semantic Analysis, 427 (2007) [21] Wang, Y., Bai, H., Stanton, M., Chen, W.-Y., Chang, E.Y.: Plda: Parallel latent dirichlet allocation for large-scale applications. In: Proc. of 5th International Conference on Algorithmic Aspects in Information and Management (2009), Software available at, http://code.google.com/p/plda [22] Wei, X., Croft, W.B.: LDA-based document models for ad-hoc retrieval. In: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 178–185. ACM, New York (2006) [23] Wong, S.K.M., Ziarko, W., Wong, P.C.N.: Generalized vector spaces model in information retrieval. In: Proceedings of the 8th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 18–25. ACM, New York (1985)
Exploring Contextual Models in Chemical Patent Search Jay Urbain1 and Ophir Frieder2 1
Electrical Engineering & Computer Science Department Milwaukee School of Engineering Milwaukee, WI
[email protected] 2 Department of Computer Science Georgetown University Washington, DC
[email protected]
Abstract. We explore the development of probabilistic retrieval models for integrating term statistics with entity search using multiple levels of document context to improve the performance of chemical patent search. A distributed indexing model was developed to enable efficient named entity search and aggregation of term statistics at multiple levels of patent structure including individual words, sentences, claims, descriptions, abstracts, and titles. The system can be scaled to an arbitrary number of compute instances in a cloud computing environment to support concurrent indexing and query processing operations on large patent collections. The query processing algorithm for patent prior art search uses information extraction techniques to identify candidate entities and distinctive terms from the query patent’s title, abstract, description, and claim sections. Structured queries integrating terms and entities in context are automatically generated to test the validity of each section of potentially relevant patents. The system was deployed across 15 Amazon Web Services (AWS) Elastic Cloud Compute (EC2) instances to support efficient indexing and query processing of the relatively large 100G+ collection of chemical patent documents. We evaluated several retrieval models for integrating statistics of candidate entities with term statistics at multiple levels of patent structure to identify relevant patents for prior art search. Our top performing retrieval model integrating contextual evidence from multiple levels of patent structure resulted in bpref measurements of 0.8929 for the prior art search task, exceeding the top results reported from the 2009 TREC Chemistry track. Keywords: Information Retrieval, Patent Retrieval, Scaleable Database Systems, Dimensional Modeling, Probabilistic Modeling, Chemical IR.
1 Introduction The TREC Chemistry Track for 2009 was organized to evaluate the statistical significance on the ranking of information retrieval (IR) systems and the scalability of IR H. Cunningham, A. Hanbury, and S. Rüger (Eds.): IRFC 2010, LNCS 6107, pp. 60–69, 2010. © Springer-Verlag Berlin Heidelberg 2010
Exploring Contextual Models in Chemical Patent Search
61
systems when dealing with chemical patents [1]. A test collection was assembled from approximately 1.2M patent files (approximately 100G) of full-text chemical patents and research papers to evaluate two ad-hoc retrieval tasks common to patent investigation: technology survey and prior art search. The goals of technology survey search and prior art search are fundamentally different. Technology survey search is similar to ad-hoc retrieval targeting patent documents using a natural language query to satisfy an information need. Systems are required to return a set of documents that is relevant to this information need. The goal of the technology survey evaluation is to identify how current IR methods adapt to text containing chemical names and formulas. Systems for the technology survey task are evaluated using a pooling, sampling, and expert evaluation methodology. The goal of prior art search is to evaluate the validity of a patent claim. In this task, systems attempt to identify prior art that may invalidate a patent claim. The query set for this evaluation consists of 1000 patent files with prior art references removed. Systems are required to return a set of documents relevant to the prior art of claims stated in the patent. Of special interest in this task was to consider three types of topics: full text patents, description only, and claims only. Systems for this task are evaluated automatically using the known references for each patent. The focus of this research effort is on prior art search. Chemical and patent information retrieval are challenging tasks. Chemical IR requires a chemical named entity identification strategy for dealing with large multiword terms, synonyms, acronyms, and morphological variants used for identifying the same chemical concept. For example, Dipeptidyl peptidase-IV inhibitor can also be referred to as Dipeptidyl peptidase-4, DPP4, DPP-4, or dipeptidylaminopeptidase. Guggulsterone can be identified as Pregna-4,17-diene-3,16-dione, Guggulsterone-E, Guggulsterone-Z, trans-guggulsterone, or as the guggulu steroid extract. Many chemical terms also have hierarchical hyponym-hypernym relationships. For example, silver halides, AgX, could include silver bromide (AgBr), silver iodide (AgI), or silver fluorides. The subject matter of chemical patents can also extend far beyond topics in chemistry and law, and include topics, vocabulary, and related data from disparate fields. For example, many patents include chemical reactions related to metabolic processes and include the biological vocabulary of genes, proteins, and metabolic networks. Pharmaceutical patents typically integrate pharmaceutical nomenclature with chemistry, biology, and medical disease processes and treatment. Engineering patents can include chemical compounds and processes for developing effective adhesives to a broad range of topics in materials science. All of these fields contain large, complex vocabularies that can complicate the process of term normalization and entity identification. Chemical patent retrieval clearly requires not only the identification of entities across disparate fields, but the relationships between entities in the context of how they are applied. Patent documents are often large, relatively complex documents with inconsistent structure and semantics. Due to problems with legal standards for creating titles and abstracts, these fields can provide little additional value for search and can obfuscate
62
J. Urbain and O. Frieder
the true nature of an invention [2]. For example, the European Patent Office (EPO) advises applicants to not disclose inventions in titles to prevent early disclosure. The US Patent and Trade Office (USPTO) provides guidelines limiting abstracts to 150 words. In addition, a significant portion of patent abstracts can be taken up with disclosures. Patent text can also be obfuscated by using “word smithing” to not preclude getting a patent, and at the same time not limit the scope of a patent. In addition, the structure of patent documents can inconsistently include claim text in description fields and vice versa often due to OCR (optical character recognition) processing of documents. The structure of patent documents is important for identifying and validating specific claims as they serve as the legal basis of the patent, however descriptions in other portions of the documents may provide important context and alternative nomenclature [3]. Clearly, information retrieval approaches are needed that integrate multiple forms of evidence from throughout the patent application. To meet the needs of chemical patent IR, and the scalability needs of large patent collections, we have developed a distributed search engine based on a dimensional data model. The model supports entity identification and aggregation of term statistics at multiple levels of patent structure including individual words, sentences, claims, descriptions, abstracts, and titles [4]. The query processing algorithm uses information extraction and locally aggregated term statistics to identify and help disambiguate candidate entities in context. Extracted candidate entities are filtered using discriminative measures of how informative an entity phrase is with respect to the patent document. Structured queries are automatically generated based on the relative distinctiveness of individual terms and candidate entities from the query patent's claims, abstract, title, and description sections. We evaluated several probabilistic retrieval functions for integrating statistics of retrieved entities with term statistics at multiple levels of document structure to identify relevant patents. Our primary objective in this research was to develop a scalable and flexible system and evaluate methods for integrating multiple levels of context as a basis for further research. We first describe our distributed indexing model, followed by the system description, indexing process, query processing algorithms, and our retrieval models.
2 Distributed Dimensional Data Model Paragraphs, sentences, and terms, representing complete topics, thoughts, and units of meaning respectively, provide a logical breakdown of document lexical structure into finer levels of meaning and context [5]. We capture these hierarchical relationships within a search index based on a dimensional data model. As shown in Fig. 1, this dimensional model can be logically represented as an n-dimensional cube. Where each patent document is represented as a series of paragraphs (title, abstract, descriptions, and claims). Each paragraph is represented as a sequence of sentences, and each sentence is represented as a sequence of individual terms. Such a model facilitates
Exploring Contextual Models in Chemical Patent Search
63
Fig. 1. Search index based on dimensional model
search for multi-word terms and efficient aggregation of term statistics within multiple levels of patent structure. The model also supports term variants, however no term variants or synonyms from external data sources were used in this study. As shown in Figure 2, we represent the dimensional index as a star schema [6][7] with a dimension table for each level of document structure (document, paragraph, sentence, term) and three central fact tables or postinglists. The “grain”, i.e., the
Fig. 2. Search index based on dimensional model
64
J. Urbain and O. Frieder
smallest non-divisible element of the database, is the individual word as represented in the term postinglist. Pre-aggregated postinglists for sentences, paragraphs, and documents were generated to improve sentence, paragraph, and document query performance respectively. Sentences aggregate words in sequence by position, paragraphs aggregate sentences, and documents aggregate paragraphs. The index can be extended to include additional dimensions, and allows for efficient formulation of SQL search queries. By indexing each individual word, queries can be developed for searching single- and multi-word terms, and term statistics can be aggregated over different levels of document structure.
3 System Description Indexing, retrieval, and analysis applications were developed in Java. The MySQL 5.1 database with the MyISAM storage engine was used for index storage and retrieval. 15 Amazon Elastic Cloud Compute (EC2) m1.small instances based on the Ubuntu Hardy base (ami-ef48af86) machine image were allocated for processing each of 15 database shards [8]. Each shard was roughly equivalent to the Chemical IR track collection distribution. For example, one shard for EP 000000, one shard for EP 000001, one shard for US 020060, etc. Elastic Block Storage (EBS) volumes of 350G were allocated for each compute instance to accommodate the size of the index and the need to insure persistence of the database if a compute instance was restarted. An m1.small EC2 compute unit consisted of 1.7 GB memory, 1 32-bit virtual core, and 160 GB of storage. Experiments with larger dual-core compute instances improved indexing performance 2-fold per instance, but did not significantly improve query performance. It takes approximately 2 days to construct the entire index. Each m1.small instance cost $0.10 per hour. Additional charges are encountered for data loading and storage. Each compute instance performed roughly equivalent to a standard Pentium 4 laptop with 2G of memory. The total cost of the experiment was approximately $1,000 though this included a fair amount of trial and error to get things running.
4 Indexing Process The indexing process includes the following: 1.
2.
3.
Lexical Partitioning: Documents are parsed into title, abstract, descriptions, and claims. Each is subsequently parsed into paragraphs, and these paragraphs are parsed into sentences. Tokenization: Sentence terms are tokenized, stop words removed, and lexical variants are normalized. Porter stemming [9] is used on each token with the following exceptions: all upper case, mixed case, alpha-numeric terms. Small “s” is also stripped from all upper-case terms. Indexing: Each individual word is stored in the index with positional information and its paragraph type (title, abstract, description, or claim).
Exploring Contextual Models in Chemical Patent Search
65
5 Query Processing 5.1 Paragraph Queries For paragraph search, the query patent title, abstract, up to the first 20 claims, and up to the first 20 descriptions (depending on the experiment) are extracted as paragraphs. Using the following as an illustrative example: “We are a new pharmaceutical company that is interested in entering the area of Dipetidyl peptidase-IV inhibitors…”, each paragraph is processed as follows: 1. 2. 3.
4. 5.
6.
Sentences are extracted. Part-of-speed tagging is performed. Candidate entities are identified by locating non-recursive noun phrases: [pharmaceutical_NN company_NN], [Dipetidyl_NN peptidase-IV inhibitors_NNS]. Candidate entities are cached in a collection for subsequent filtering. Stop and function words are removed. The top 7 terms by NIDF (normalized inverse document frequency) above a minimum threshold of 0.10 (0.0 to 1.0) is used to generate a query for each paragraph The top 500 paragraphs are retrieved using the probabilistic BM25 retrieval function [10] shown in equation (1). BM25 is implemented using standard SQL.
BM25:
(1)
⎛ ⎞ ⎜ ⎟ ⎛ N − df + 0.5 ⎞⎜ k1 + 1) * tfd ( ⎟⎛⎜ (k 3 + 1) * tfq ⎞⎟ ∑wq ln⎜⎜ df + 0.5 ⎟⎟⎜ ⎟⎜ k 3 + tfq ⎟⎠ docLen ⎠ k1 * (1 − b) + b * ( ⎝ ) + tfd ⎟⎝ ⎜ avgDocLen ⎝ ⎠ Note: We used k1=1.4, k2=0, k3=7, and b=0.75
5.2 Entity Queries From the cached list of candidate entities from across title, abstract, claim, and description paragraphs, entities are selected as follows: 1.
2. 3.
Entities must occur in at least 2 paragraphs and have a NIDF > 0.15. We have found that this drastically reduces the number of spurious entity phrases. Entity phrases are ranked by NIDF and their log frequency of occurrence. The top 20 remaining entity phrases are searched within the context of title, abstract, description, and claims paragraphs of target patent documents.
66
J. Urbain and O. Frieder
The following abbreviated query illustrates entity search for “Dipetidyl peptidase-IV inhibitor”. All queries are distributed across all database shards and results are aggregated: select i1.term, p1.docid, p1.parid, p1.sentid, p1.seq, p1.section, d.docnum from invertedindex_qc i1, postinglist_qc p1, paragraphresults_qc d where i1.term=' dipetid’ ' and i1.termid=p1.termid and p1.docid=d.docid and d.parid=p1.parid and exists ( select * from invertedindex_qc i2, postinglist_qc p2 where i2.term=' peptidas' and i2.termid=p2.termid and p1.docid=p2.docid and p1.parid=p2.parid and p1.sentid=p2.sentid and p1.section=p2.section and abs(p2.seq-p1.seq)<=5 and exists (…
5.3 Document Queries The top 5000 target patent documents are retrieved using a BM25 formulated query of the top 20 individual terms (by NIDF) selected from the top 20 candidate entities. The idea is to select a relatively even distribution of terms across all patent claims. 5.4 Distributed Processing As illustrated in Fig. 3, each paragraph, entity, and document query is distributed across all compute nodes and index shards. Results from each compute node are aggregated and ranked using a probabilistic ranking function. Table 1 shows the similarity coefficients (SC) computed from the paragraph, entity, and document query results for each retrieved patent document. Each individual score is normalized. Table 1. Document Retrieval Function Similarity Coefficients Similarity Coefficient docSc parSc top3parSC top10parSC conceptDocCount parConcept
Definition Top document (normalized BM25) Top paragraph (normalized BM25) Top 3 paragraphs (normalized BM25) Top 10 paragraphs (normalized BM25) Distinct count of candidate entities per document Largest distinct count of candidate entities in a single paragraph
A linear weighted sum (2) is used to generate various retrieval functions by weighting and combining similarity coefficients (SC) for each target document. SCcomposite = w1SC1 + w2SC2 + …+ wnSCn
(2)
Exploring Contextual Models in Chemical Patent Search
67
Fig. 3. Distributed Processing
6 Results Table 2 shows the results from retrieval functions generated by integrating our highest-performing similarity coefficients listed in Table 1. These results use all sections (title, abstract, description, and claim) in target documents, and only title, abstract, and claim sections for generating the query from the query patent. Results are for the first 100 queries of the query set (small query set). It is important to note that no external knowledge sources were used for this evaluation. The document retrieval similarity coefficient (SC) resulted in a bpref measurement of 0.7201, competitive with the top TREC results. The paragraph SC was 0.7503. Integrating document with paragraph SC’s improved the result to 0.7769. Integrating individual entity SC’s further improved the results to 0.7854. This result exceeded the top result from the TREC 2009 Chemical IR track of 0.7432 for the short PA task (100 query patents).These results clearly demonstrate the efficacy of integrating multiple sources of evidence of entities and terms in context. [
68
J. Urbain and O. Frieder Table 2. Results (without descriptions in query patent) runid All docSC parSC top3parSC top10parSC docConcept parConcept docSC + parSC docSC + parSC + docConcept docSC + parSC + parConcept docSC + parSC + docConcept + parConcept
bpref 0.7738 0.7201 0.7503 0.7481 0.7468 0.6697 0.6624 0.7769 0.7814 0.7842 0.7854
Table 3 shows the results using all sections in target documents, and all sections (title, abstract, description, and claim) for generating the query from the query patent. Table 3. Results (with descriptions in query patent) runid docConcept docSC + top3parSC + docConcept + parConcept docSC + parSC + top3parSC + docConcept + parConcept
bpref 0.8624 0.8929 0.8924
By integrating candidate entities and textual context from the description sections of query patents, we were able to further improve our bpref performance to 0.8924. Since the description sections of patents are not part of the legal claim of a patent, we were surprised by this result. Upon further examination, the bref scores were improved primarily by improving the relative ranking of relevant documents in the results by improved term matching. Description sections clearly provide valuable context for improving the ranking of relevant patents. From our work we have noted a number of issues and opportunities for further improvement of these models. •
• •
Identifying candidate entities is a work in progress. There is a fundamental tradeoff in selecting entities that are too distinctive (low recall) versus returning huge result sets from such a large collection. We would like to explore more sophisticated NLP techniques as well as explore chemical named entity resolution and synonym identification by integrating external databases. More refined probabilistic models with adaptable weights for integrating SC’s. Identifying the optimal size and number of shards for indexing and query model performance.
7 Conclusion We explored development of a multidimensional indexing model distributed across a cloud computing cluster to enable efficient search and aggregation of entities and
Exploring Contextual Models in Chemical Patent Search
69
terms at multiple levels of patent context. Several retrieval models integrating entities with term statistics at multiple levels of context were evaluated. Our top performing models exceeded baseline results, and exceeded the top results reported at the TREC forum, demonstrating the efficacy of our approach.
References 1. Lupu, M., Piroi, F., Tait, J.: Overview of the TREC 2009 Chemical IR Track. In: The Eighteenth Text REtrieval Conference Proceedings (TREC 2009), Gaithersburg, Maryland (2009) 2. Adams, S.: The text, the full text and nothing but the text: Part 1 – Standards for creating textual information in patent documents and general search implications. In: World Patent Information, vol. 32, pp. 22–29. Elsevier Ltd., Amsterdam (2010) 3. Fujii, Atsushi, Iwayama, M., Kando, N.: Introduction to the special issue on patent processing. Information Processing and Management 43, 149–1153 (2007) 4. Urbain, J., Frieder, O., Goharian, N.: Probabilistic Passage Models for Semantic Search of Genomics Literature. Journal of the American Society of Information Science and Technology (2008) 5. Urbain, J., Frieder, O., Goharian, N.: A Dimensional Retrieval Model for Integrating Semantics and Statistical Evidence in Context for Genomics Literature Search. Computers in Biology and Medicine (2008) 6. Gray, J., Chaudhuri, S., Bosworth, A., Layman, A., Reichart, D., Venckatrao, M., Pells, F.: Data Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals. Data Mining and Knowledge Discovery 1(1) (1997) 7. Kimball, R.: The Data Warehouse Toolkit: Practical Techniques for Building Dimensional Data Warehouses. John Wiley, Ralph (1996) 8. Amazon Web Services, http://aws.amazon.com/documentation/PubChem, National Center for Biotechnology Information (NCBI), http://pubchem.ncbi.nlm.nih.gov 9. Porter, M.F.: An algorithm for suffix stripping. Program 14, 130–137 (1980) 10. Robertson, S., Walker, S.: Okapi/Keenbow at TREC-8, pp. 246–500. NIST Special Publication (2000)
Measuring the Variability in Effectiveness of a Retrieval System Mehdi Hosseini1 , Ingemar J. Cox1 , Natasa Millic-Frayling2 , and Vishwa Vinay2 1
Computer Science Department, University College London {m.hosseini,i.cox}@cs.ucl.ac.uk 2 Microsoft Research Cambridge {natasamf,vvinay}@microsoft.com
Abstract. A typical evaluation of a retrieval system involves computing an effectiveness metric, e.g. average precision, for each topic of a test collection and then using the average of the metric, e.g. mean average precision, to express the overall effectiveness. However, averages do not capture all the important aspects of effectiveness and, used alone, may not be an informative measure of systems’ effectiveness. Indeed, in addition to the average, we need to consider the variation of effectiveness across topics. We refer to this variation as the variability in effectiveness. In this paper we explore how the variance of a metric can be used as a measure of variability. We define a variability metric, and illustrate how the metric can be used in practice.
1
Introduction
A common practice in a comparative evaluation of information retrieval (IR) systems is to create a test collection comprising a document collection, a set of topics (queries) and associated relevance judgments, and to then measure effectiveness (performance1 ) of retrieval systems over such a collection. A typical evaluation of a system involves computing an effectiveness metric, e.g. average precision (AP), and then averaging across topics, e.g. computing the mean average precision (MAP), to characterize the overall system effectiveness. However, when used alone, averages do not capture all the important aspects of effectiveness. For example, averages may not reveal possibly large variations in effectiveness across topics. We maintain that, in addition to average effectiveness, one needs to consider the variation in effectiveness across topics. In particular, when two systems are not distinguishable based on their average, we can use the variations to contrast them. We refer to the cross-topic variation as the variability in effectiveness. There are various ways in which variability could be measured. In this paper we explore how the variance of IR metrics, in particular, the variance in AP scores, can be used for this purpose. The IR community is, of course, familiar with variance, and uses it routinely to assess whether the difference in the averages 1
In this paper we use effectiveness and performance interchangeably.
H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 70–83, 2010. © Springer-Verlag Berlin Heidelberg 2010
Measuring the Variability in Effectiveness of a Retrieval System
71
of two systems’ effectiveness is significant or not. However, our use of variance is different, and we illustrate this next. Consider a scenario illustrated in Figure 1a, in which we have two systems, A and B, each of which exhibits the same MAP score, but the variance of AP scores for System A is much larger than for System B. If the two systems are compared based on MAP alone, then a paired student t-test will conclude that the two systems are equivalent. However, in practice, users may observe a significant difference between the two systems. Qualitatively, Figure 1a shows that System A either gives very good or very poor responses to a query. In contrast, System B gives ”satisfactory” responses to all queries, i.e. the responses of System B are neither very good nor very bad. Which system would a user prefer? The answer to this question is not entirely straightforward. Consider a scenario in which users require AP scores to exceed a minimum threshold in order to be satisfied with the response of the system. This is depicted by the horizontal line in Figure 1a. In this case, System B always satisfies users, while half the time, System A fails to satisfy users, despite the fact that both systems have the same MAP. In this case, the system with lower variability is preferred. Now consider Figure 1b, in which we again have two systems, C and D, with the same MAP score. However, in this scenario, the MAP is lower than the threshold needed to satisfy users. In this case, the system with lower variance, D, never satisfies users. In contrast, System C, with high variance does satisfy users for some queries. In this case, the system with higher variability is preferred.
Fig. 1. Two IR systems with (a) equal MAP which is larger than a threshold needed to satisfy a user, and (b) two IR systems with equal MAP smaller than the threshold
This example highlights three important points. First, the average of a metric does not always provide sufficient information with which to judge a system. Second, variability can be used not just for significance testing, but also to characterize systems with similar average performance. Third, a preference for systems with high or low performance variability depends on the relative performance of systems in comparison with a user’s satisfaction threshold.
72
M. Hosseini et al.
Of course, this is an artificial example. However it is common for real systems to exhibit statistically identical mean performance, yet exhibit different levels of variance. For example, Table 1 shows two experimental runs2 from the Robust track of TREC 2004 [11]. For each of them we compute MAP across 199 topics (351-450 and 600-700), removing one topic that had no relevant documents in the collection. We also calculate the standard deviation of AP scores as our measure of variability. The two runs have the same MAP value but different standard deviations. Using the paired t-test reveals that there is no significant difference in the MAP values while applying a statistical test to assess the equality of standard deviation (levene’s test, see Section 2.2) confirms that the difference in the standard deviations is statistically significant. Table 1. Two experimental runs from the robust track of TREC 2004. The corresponding MAP values and standard deviations of AP scores, SD (AP), are measured over 199 topics. Runs MAP SD(AP) Paired t-test Levene’s test uogRobSWR5 0.304 0.24 p = 0.96 0.05 p = 0.007 0.05 NLPR04clus10 0.304 0.20
In Section 2.1, we discuss related work and in Section 2.2 two statistical tests are introduced which are used to determine whether the difference between the variances of two systems’ effectiveness is statistically significant. In Section 3, we describe several experiments. We first highlight the problem of using standard deviation (SD) of a metric which has bounded values as a measure of variability. In order to overcome the problem we consider two transformations that have been used in the context of information retrieval. Then we evaluate the systems that participated in two tracks of TREC 2004. When pairs of systems are compared, we find that 26% of these pairs have no statistical difference in their MAP scores. We refer to these systems as ”ties”. However, 44% of the ties have statistically different levels of variability. Finally, we consider the minimum number of topics required to run a robust comparison in terms of variability. Using two test collections, Robust track and Web track of TREC 2004, we observe that 90 topics are required to run a comparison with less than a 5% error rate. We finally discuss our findings in Section 4.
2
Background
In this section, we first briefly discuss related work and then introduce two statistical significant tests used to compare variabilities in systems’ effectiveness.
2
A single retrieval system can have several settings. In TREC each setting is referred to as a run. For our purposes we will treat each run as a search system.
Measuring the Variability in Effectiveness of a Retrieval System
2.1
73
Related Work
The topic of variability in effectiveness has received little attention in IR research. Perhaps, the most of prior work related to variability is to do query expansion. Query expansion methods typically yield good improvements in mean average precision but are unstable and have high variance across queries [2]. CollinsThompson [1] proposed a model of evaluating effectiveness of query expansion methods by using a risk-reward tradeoff where reward was defined as the percentage gain for MAP relative to the original, un-expanded query, and the risk reflected the number of relevant documents that were lost due to the expansion. Such a risk measurement is solely based on the number of relevant documents. In contrast, the percentage MAP gain depends on not only the number of relevant documents retrieved but also the ranks of them. Perhaps the variability in effectiveness, as defined in this paper, can be an alternative measure of risk where both number of relevant documents and corresponding ranks are taken into account. C.T. Lee et al. [5] proposed a novel weighted average (generalized adaptiveweight mean) to rank systems’ effectiveness where the weights reflected the ability of the test topics to differentiate among the retrieval systems. The variance of the AP scores was indirectly incorporated into measuring systems’ effectiveness. In particular, they used the Euclidean distance to characterize the dispersion of AP scores. However, effectiveness of their system ranking and comparison was not evaluated in detail. We observe that the performance scores (AP values) are bounded in [0, 1] and expect that the approach will be affected by the boundary conditions, 0 and 1, as we discuss in Section 3. In this paper, we propose a way to overcome the issues of a bounded score distribution and its effect on the variance. 2.2
Statistical Significance Tests
Tests of statistical significance have been thoroughly discussed in the IR literature. The common statistical significance tests used in IR experiments are student’s paired t-test, wilcoxon signed rank, and sign test. The assumptions which these tests are based on were discussed in [4]. In addition, the use of two sampling-based tests, bootstrap shift method and fisher’s randomization, in IR was discussed in [10]. Sakai [8] also discussed the use of paired bootstrap test in IR which was a combination of the bootstrap shift method and student’s t-test. These tests, for example, make use of the variance of AP scores to determine whether the difference in two MAP scores is statistically significant. Here, we are interested in determining whether the difference in variabilities, as measured by variance or standard deviation, of two systems’ effectiveness is statistically significant. The statistical community has, of course, addressed this and we briefly describe two tests, the F-test and Levene’s test. F-test. This test first defines a ratio of the standard deviations of two systems’ effectiveness measured across a set of topics. Therefore, if σA and σB are the standard deviations of AP scores of systems A and B, the ratio is calculated as:
74
M. Hosseini et al.
F =
σA σB
(1)
In the F-test the null hypothesis and the alternative hypothesis is defined as below: H0 : σA = σB (the null hypothesis) H1 : σA = σB (the alternative hypothesis) The more the ratio deviates from 1, the stronger the evidence for unequal variances. The null hypothesis is rejected if the ratio was larger than a critical value. The critical value is adjusted based on a significance level, e.g. α = 0.05 or α = 0.01. There is a limiting condition in F-test assuming that the distribution of AP scores is normal. However, our experiment in Section 3.3 shows that this assumption is not necessarily true. In order to deal with this restriction, we also consider the Levene’s test which does not have such an assumption. Levene’s Test. Levene’s test is used to assess whether k sample groups have the same standard deviation [6]. Levene’s test does not have the normality assumption. The statistic is obtained from one-way analysis of variance (ANOVA), where each observation, in our case each AP score, is replaced with its absolute deviation from the associated group’s mean. In our case the group mean is the MAP value. Let zij = |APij − M APi |, where APij is the measured AP value of the ith system on the j th query. Levene’s test defines a ratio as: z¯)2 × i (ni − 1) i ni (z¯i − W0 = (2) (g − 1) × ( i j (zij − z¯i )2 ) where g is the number of sample groups which in our case is 2 indicating the number of systems, and ni is the number of observations in the ith group (in our case it is equal to the number of topics): zij zij z¯i = and z¯ = ni ni The null hypothesis is rejected if W0 was larger than a critical value that is adjusted with regard to a significance level. Replacing the group mean, M APij , with the median of observations, median(AP), in forming zij defines W50 . We use W50 instead of W0 when the AP distributions suffer from skewness.
3
Experiments
In Section 3.1, we examine the performance of various systems involved in the Web and Terabyte tracks of TREC 2004. This study reveals a curious phenomenon - systems with average performances, measured by a bounded IR metric, e.g. MAP, near to 0.5 have larger variances than systems with average performances near to each of the two boundaries, 0 and 1. This phenomenon
Measuring the Variability in Effectiveness of a Retrieval System
75
is an artifact of the fact that the metric’s scores are bounded in [0,1]. Section 3.2 proposes two transformations of the metric’s scores in order to eliminate this artifact. Section 3.3 then considers all pairs of systems participating in two test collections of TREC 2004: the Web and Robust tracks. Student t-tests show that 26% and 28% of pairs, respectively, are ties, i.e. there is no statistically significant difference in the averages of transformed AP scores. If the variability in effectiveness of these ties is examined, then the F-test shows that 33% and 34% of ties have statistically significant differences in variance. When Levene’s test is used, 47% and 38% of ties have statistically significant differences in variance. Finally, in Section 3.4, we explore the effect that the size of a topic set has on the system comparison using variability in effectiveness. We observe that one needs to consider a sample of 90 topics to obtain an error rate smaller than 0.05. 3.1
The Variance of a Bounded Metric
Figure 2 plots the standard deviation in AP scores as a function of MAP, for systems participating in the Web and Terabyte tracks of TREC 2004. The Web track involves 74 systems and 225 topics. The Terabyte track involves 70 systems and 49 topics.
Fig. 2. The standard deviation of AP values (SD (AP)) versus MAP. The standard deviation is bounded in a semicircle with centre (0.5, 0.0) and radius 0.5.
We note that some systems have similar MAP values. Thus, it would be beneficial to use additional criteria, e.g., variability in effectiveness, to differentiate their performance. This is discussed shortly. However, the most striking feature in Figure 2 is an unexpected trend: the monotonic relationship between standard deviation and MAP, i.e. the larger the MAP value, the larger the variance in AP scores. We believe this relationship is due to the bounded nature of AP metric, i.e. the fact that the metric’s values fall within [0,1]. This bounds the standard deviation of AP scores to a semicircle as shown in Figure 2 and proven in Appendix A.
76
M. Hosseini et al.
Therefore, the retrieval system with MAP close to one of the boundaries, 0 or 1, are more likely to have a smaller variance than those with MAP near to the center, 0.5. For this reason using the standard deviation of the raw AP scores is not a reliable measure of variability. In fact, this is true for any other bounded IR metrics, e.g. the reciprocal rank, as shown in Figure 3. The figure shows how the variance decreases as the MRR increases above 0.5. Again, this is expected since now the variation above the mean is limited by the upper bound of one on reciprocal rank.
Fig. 3. MRR versus the standard deviation of RR values from: (a) runs participating in the Web track 2004, (b) runs participating in the Terabyte track 2004
In order to overcome this issue, we consider functions that map values from [0,1] to (−∞, +∞). We favor mappings that produce a symmetric distribution in the transformed space, akin to the normal distribution, if possible. We can then define the variability in effectiveness as the variance of the transformed values of a metric. 3.2
The Variability of Transformed Scores
We illustrate our approach by considering two transformations that have been used in IR and observe the properties of the transformed scores. The first is the logit function used by Cormak and Lynam [3] as a parametric estimate to deal with the asymmetric AP distribution. The logit is defined as: logit(x) = x log( 1−x ) for x in (0,1). The boundary points, 0 and 1, are replaced by and 1− respectively, for a small value of > 0, and then transformed, letting → 0. The second transformation is the standardized score or z-score whose use in x) IR was motivated by Webber et al. [13]. It is defined as z = (x−¯ , where x σ is a metric’s score, e.g. an AP score. In addition, x ¯ and σ are the average and standard deviation of a set of scores measured across a set of retrieval systems on a fixed topic. Hence, for a particular topic it is defined as z=
(AP − M ean(AP )systems ) SD(AP )systems
(3)
Measuring the Variability in Effectiveness of a Retrieval System
77
In the following, we observe several properties of the logit(AP) and the standardized z-scores.
Fig. 4. The frequency distributions before and after transformation of three runs in Web track. (a) a run with a low MAP, (b) a run with a medium MAP, (c) a run with a high MAP.
Boundary Values and Score Distributions. Figure 4 shows three runs of the Web track collection: (a) with a low MAP value, (b) with a medium MAP value, and (c) with a high MAP value. The distributions of the AP scores before and after both the logit and z-score transformations are presented as frequency histograms. The logit and z-score transformations differ significantly in the way they handle boundary values. The logit transformation transforms the boundaries to extreme values in the transformed space. This is observed by the extreme values at each end of the distributions in the middle column. In contrast, the z-score transformation disperses the boundaries smoothly as illustrated in the right column. In addition, the z-score transformation helps eliminate the source of variance coming from topic difficulty3 before measuring the variability of system effectiveness itself. We now consider the variability in a system’s effectiveness as the standard deviation of the transformed AP values. Let MLAP refer to the mean of the logit-transformed AP values and let MSAP refer to the mean of the standardized z-transformed AP values. Figure 5 shows the scatterplots of the standard 3
A topic is regarded as difficult if the range of effectiveness scores measured across a set of systems is small and near to zero.
78
M. Hosseini et al.
deviations in transformed AP values as a function of their mean values, MLAP and MSAP. As seen in the figure, the logit and z-score transform the scores in different ranges. In addition, there is no longer a monotonic relationship between the values of mean and variability.
Fig. 5. Variability in effectiveness versus mean of transformed AP values: logit (a) and the z-score transformation (b)
3.3
Variability as a Tie Breaker
We consider all pairs of the top 75% (ordered by MAP) of systems participating in either the Robust or Web track of TREC 2004. We compare systems based on the mean of the standardized z-scores (MSAP). We use the paired t-test to measure the significance of MSAP differences. We set the significance level to 0.05. For all the ties we use the F-test and Levene’s test to investigate the proportion of ties for which the variabilities in effectiveness’s scores are significantly different. Table 2. The variability in effectiveness as a tie breaker: number of pairs, ties and broken ties in two tracks of TREC 2004 Collections Pairs Robust Web
Status
before transformation after transformation before transformation 1485 after transformation 3321
Ties 997 857 469 415
(30%) (26%) (31%) (28%)
Broken ties F-test Levene 0 (0%) 106 (11%) 280 (33%) 404 (47%) 1 (0.002%) 21 (0.04%) 140 (34%) 158 (38%)
As seen in Table 2 for the Robust track, 30% of pairs are considered ties, when using AP score, and 26% are considered ties in the transformed space. Interestingly, before transformation, the F-test cannot distinguish any statistical difference in variability, and the Levene’s test can only break 11% of the ties.
Measuring the Variability in Effectiveness of a Retrieval System
79
In contrast, after transformation into the z-space, the F-test can distinguish between 33% and Levene’s test can distinguish between 47% of the tied pairs. A similar effect before and after transformation is observed for the Web track. 3.4
The Effect of Topic Set Size on Measuring Variability in Effectiveness
If we are to use variability to characterize systems, it will be useful to know how many topics are needed to reliably compare two systems in terms of variability in effectiveness. Indeed, we will need to know how likely a decision would change if we compare systems using a different topic set. This performance variation across topic sets has previously been studied in the context of average performance [12]. We perform the same experiment to compare variabilities in systems’ effectiveness. foreach topic set size c from 10, 20, 30, ... , 100 { set the counters to 0; foreach TREC test collection t { foreach pair of systems A and B from track t { foreach trial from 1 to 50 select two disjoint sets of topics X and Y of size c from t; if ( the difference between the variabilities is significant){ d_X=SD(A,X)-SD(B,X); d_Y=SD(A,Y)-SD(B,Y); increment counter; if(d_X * d_Y < 0) { increment swap counter; } } }} error-rate (c) =swap counter /counter;} Fig. 6. Calculating error rates. SD(A, X) is the standard deviation of AP scores of system A measured on the topic Set X.
In our experiment, we first fix the topic set size, and then compute the variabilities in effectiveness of a pair of systems, A and B. Let us assume that System A is less volatile than System B based on this measurement. We then estimate the probability of a changed decision, i.e. finding System B to be less volatile than System A. We estimate this probability by comparing the two systems across several trials that use different topic sets and then counting how many times the preference decision changes. Finally, to estimate the average probability of changing a decision, we repeat the process on different pairs of systems. This average probability (across systems) is called the error rate. The whole process is repeated for different sizes of topic sets. The algorithm for computing the error rate is shown in Figure 6. It is based on the algorithm described in [12]. In our experiment, we compute the error rate for all the pairs regardless of their absolute differences. We run 50 different trials using different combinations of topics in the two disjoint topic sets.
80
M. Hosseini et al.
Furthermore, as Sanderson and Zobel [9] suggested, we only consider pairs with statistically significant differences in variability, as measured by Levene’s test with a significance level of 0.05. Once again we use the runs participating in the Robust track of TREC 2004 using topics 351-450 and 601-700 (199 topics), and the runs participating in the Web track (225 topics). In this experiment, only the top 75% of systems (ranked by MAP) are considered to prevent the poorly performing runs from having an effect on our conclusion [12]. Thus, our data collection consists of 135 runs and 4806 pairs of runs. Note that we transform AP scores using the z-score before measuring variability. The resulting error rate is shown in Figure 7. As expected, the curve shows that the error rate decreases as the topic set size increases. The experiment indicates that 90 topics are required to obtain an error rate less than 0.05. With 80 topics the measured error rate is 0.052 and with 90 topics it is 0.038.
Fig. 7. Error rate versus topic set size using two TREC test collections: web track and robust track of TREC 2004
4
Summary and Discussion
The average of effectiveness, measured across a topic set, does not capture all the important aspects of effectiveness and, used alone, may not be an informative measure of a system’s effectiveness. We defined variability in effectiveness as the standard deviation of effectiveness scores measured across a set of topics. We proposed that a mean-variance graph helps demonstrate effectiveness in a two-dimensional space rather than ranking systems based on their average effectiveness. Our investigation revealed that the bounded values of a metric yield a curious phenomenon where values of average around 0.5 are accompanied with higher variances. We attributed this to the fact that the metric values fall within [0, 1]. This bounds the standard deviation of the scores to a semicircle as proven in Appendix A. Hence, retrieval systems with average effectiveness close to each
Measuring the Variability in Effectiveness of a Retrieval System
81
of the two boundaries have smaller variances than those with average away from the boundaries. However, there might be also other reasons. For example, when the distribution is not symmetric, standard deviation cannot explain the dispersion properly. In Figure 4 it was shown that the distributions of AP scores were skewed toward the upper boundary, 1, and was completely asymmetric. We used two transformation methods to deal with this problem and showed how they differentiate systems effectiveness with the same average score. We finally discussed the minimum sample size required to estimate the variability in effectiveness. In our experiments we observed that 90 topics were required to obtain an error rate less than 0.05. This paper only considered standard deviation as the measure of variability while it would be interesting to consider other measures, e.g. interquartile range and median absolute deviation. In addition, there are several ways to transform scores in a more symmetric space. For example, one might consider both logit and z-score transformation together. That is, the AP scores are first transformed by logit to (-∞ , +∞) and then z-score is used to deal with extreme values. We also note that the minimum sample size reported in Section 3.4 was averaged across different pairs of systems. As truly shown by Lin and Hauptmann [7], the minimum sample size varies across pairs of systems, and it depends on the difference between two systems’ average effectiveness scores and corresponding variances. Mean and variability can be used to evaluate retrieval systems. One may define a new metric as a function of both mean and variability. Such a metric helps rank systems’ effectiveness in a one-dimensional space by considering both mean and variability in effectiveness. In addition, by a hypothetical scenario we showed that how a threshold of user satisfaction helps make preference between volatile and stable systems. However, we need to at least deal with two issues here. Firstly, in order to measure users’ satisfaction we need to evaluate systems from users’ perspective, i.e. directly asking users to express the amount of satisfaction. Such a user-oriented evaluation method provides accurate results but it is extremely expensive and difficult to do correctly. We can also model users’ satisfaction by using implicit feedbacks of users, e.g. click-through data in a search engine query log. This method is less expensive but inaccurate. Secondly, users’ satisfaction threshold may vary across queries. Indeed, the scenario described in Section 1 was simplified by considering the threshold as a constant value. However, in practice, the threshold varies across queries since it is highly depended on users’ information needs and their expectation of the result set. We will consider these issues for future work.
Acknowledgements The authors thank Jun Wang and Jianhan Zhu of UCL and Stephan Robertson of Microsoft Research Cambridge for useful discussion on earlier drafts of this paper.
82
M. Hosseini et al.
References [1] Collins-Thompson, K.: Robust word similarity estimation using perturbation kernels. In: Azzopardi, L., Kazai, G., Robertson, S., R¨ uger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 265–272. Springer, Heidelberg (2009) [2] Collins-Thompson, K., Callan, J.: Estimation and use of uncertainty in pseudorelevance feedback. In: SIGIR 2007: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 303–310. ACM, New York (2007) [3] Cormack, G.V., Lynam, T.R.: Statistical precision of information retrieval evaluation. In: SIGIR 2006: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 533–540. ACM, New York (2006) [4] Hull, D.: Using statistical testing in the evaluation of retrieval experiments. In: SIGIR 1993: Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 329–338. ACM, New York (1993) [5] Lee, C.T., Vinay, V., Mendes Rodrigues, E., Kazai, G., Milic-Frayling, N., Ignjatovic, A.: Measuring system performance and topic discernment using generalized adaptive-weight mean. In: CIKM 2009: Proceeding of the 18th ACM conference on Information and knowledge management, pp. 2033–2036. ACM, New York (2009) [6] Levene, H.: Robust test for equality of variances. Contributions to Probability and Statistics: Essays in Honor of Harold Hotteling, 278–292 (1960) [7] Lin, W.-H., Hauptmann, A.: Revisiting the effect of topic set size on retrieval error. In: SIGIR 2005: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 637–638. ACM, New York (2005) [8] Sakai, T.: Evaluating evaluation metrics based on the bootstrap. In: SIGIR 2006: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 525–532. ACM, New York (2006) [9] Sanderson, M., Zobel, J.: Information retrieval system evaluation: effort, sensitivity, and reliability. In: SIGIR 2005: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 162–169. ACM, New York (2005) [10] Smucker, M.D., Allan, J., Carterette, B.: A comparison of statistical significance tests for information retrieval evaluation. In: CIKM 2007: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pp. 623–632. ACM, New York (2007) [11] Voorhees, E.M.: The trec robust retrieval track. SIGIR Forum 39(1), 11–20 (2005) [12] Voorhees, E.M., Buckley, C.: The effect of topic set size on retrieval experiment error. In: SIGIR 2002: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 316–323. ACM Press, New York (2002) [13] Webber, W., Moffat, A., Zobel, J.: Score standardization for inter-collection comparison of retrieval systems. In: SIGIR 2008: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 51–58. ACM, New York (2008)
Measuring the Variability in Effectiveness of a Retrieval System
83
Appendix A Lemma: For all data sets like X = {x1 , x2 , ..., xN } where 0 ≤ xi ≤ 1, the ¯ Sx ), are confined within a corresponding mean-standard deviation values, (X, semicircle with center (0.5, 0) and radius r=0.5: ¯ − 1 )2 + S 2 ≤ ( 1 )2 ; (X x 2 2 2 2 ¯ +S ≤X ¯ X x
(4)
Proof: with reference to the mean and variance: N ¯ = 1 X xi N i=1
Sx2 =
(5)
N N N N 1 ¯ 2= 1 ¯ 1 ¯2 = 1 ¯2 (xi − X) x2i − 2 × X( xi ) + X x2 − X N i=1 N i=1 N i=1 N i=1 i
therefore:
N ¯ 2 + Sx2 = 1 X x2 N i=1 i
(6)
x2i ≤ xi because 0 ≤ xi ≤ 1; therefore: N N 1 2 1 ¯ x ≤ xi = X N i=1 i N i=1
considering (6) and (7) together: N N 1 ¯ 2 + Sx2 = 1 ¯ X x2i ≤ xi = X N i=1 N i=1
then we reach to (4): ¯ 2 + Sx2 ≤ X. ¯ X
Fig. 8. The upper bound of standard deviation for scores bounded in 0 to 1
(7)
An Information Retrieval Model Based on Discrete Fourier Transform Alberto Costa1 and Massimo Melucci2 1
2
´ LIX, Ecole Polytechnique, F-91128 Palaiseau, France
[email protected] Department of Information Engineering, University of Padua, I-35131 Padova, Italy
[email protected]
Abstract. Information Retrieval (IR) systems combine a variety of techniques stemming from logical, vector-space and probabilistic models. This variety of combinations has produced a significant increase in retrieval effectiveness since early 1990s. Nevertheless, the quest for new frameworks has not been less intense than the research in the optimization and experimentation of the most common retrieval models. This paper presents a new framework based on Discrete Fourier Transform (DFT) for IR. Basically, this model represents a query term as a sine curve and a query is the sum of sine curves, thus it acquires an elegant and sound mathematical form. The sinusoidal representation of the query is transformed from the time domain to the frequency domain through DFT. The result of the DFT is a spectrum. Each document of the collection corresponds to a set of filters and the retrieval operation corresponds to filtering the spectrum – for each document the spectrum is filtered and the result is a power. Hence, the documents are ranked by the power of the spectrum such that the more the document decreases the power of the spectrum, the higher the rank of the document. This paper is mainly theoretical and the retrieval algorithm is reported to suggest the feasibility of the proposed model. Some small-scale experiments carried out for testing the effectiveness of the algorithm indicate a performance comparable to the state-ofthe-art. Keywords: Discrete Fourier Trasform, Digital Filters, Information Retrieval Models.
1
Introduction
Different views of Information Retrieval (IR) have been proposed and implemented in a series of models in the last four decades, the most famous being the logical models, the vector space models, and the probabilistic models. The variety of combinations of techniques originated from these models produced the significant increase in retrieval effectiveness observed within the Text Retrieval Conference (TREC) initiative [1]. However, the current retrieval technology is far from being optimal within every domain, medium, language or context. For this reason, in addition to the numerous attempts to optimize and experiment the H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 84–99, 2010. c Springer-Verlag Berlin Heidelberg 2010
An Information Retrieval Model Based on Discrete Fourier Transform
85
(a) Spectrum for the query of the example. (b) Detail of the spectrum near point 200. Fig. 1. Spectrum for a query (1a) and a detail (1b)
current retrieval models for different media, languages and contexts, the quest for new theoretical frameworks has been intense. [2, 3, 4, 5] This paper illustrates a new framework which is based on Discrete Fourier Transform (DFT) and is called Least Spectral Power Ranking (LSPR). The LSPR framework provides the conceptual devices for designing and implementing an innovative class of retrieval systems. An intuitive description of the framework is provided in the remainder of this section while the other sections of the paper explain LSPR in detail. Basically in the LSPR model the query is viewed as a spectrum and each document as a set of filters, with one filter for each document term. Each term of the query is associated with a discrete time sinusoidal signal Ai sin (2πfi nT ), where Ai is the weight of the term i in the collection (e.g. IDF) and fi is the signal frequency (not related with the frequency of a term in a document); an example is depicted in Figure 1a. How the signal frequency is chosen for each term is explained in Section 4.1. Thus, the query becomes a sum of sinusoidal signals for which the DFT can be computed. After the DFT module of the query has been computed, the spectrum looks like a curve which has peaks of amplitude proportional to Ai at the frequency fi and the values of the curve monotonically decrease in the neighborhood of fi as depicted in Figure 1b. The behaviour of the spectrum is crucial because the utilization of a sinusoidal signal combined with the variation in the amplitude of the spectrum causes variations in retrieval effectiveness (see Section 4.1 for more details) – the latter phenomenon has been observed in the experiments carried out during the research reported in this paper. In general, the DFT samples are complex numbers; computing the DFT module means substituting each complex sample with its module. To obtain the ranked document list, the spectrum of the query is filtered by the documents, i.e. the filters. As a matter of fact, each document is represented as a set of filters, each of them is similar to the one in Figure 2: the breadth depends on the weight (e.g. TFIDF or BM25) of the term, while the position of the filter is set so that if a query has a term qi , associated with a peak in the frequency fi in the spectrum, and in the document there is qi , the corresponding
86
A. Costa and M. Melucci
Fig. 2. Example of frequency response for the used filter
(a) A fragment of a spectrum with a filter.
(b) The obtained filtered spectrum.
Fig. 3. Representation of the filtering operation
filter has ZL and ZR near fi (as depicted in Figure 3a). The filtered spectrum is depicted in Figure 3b. The documents are then ranked by the power of the spectrum obtained as the sum of the components of the filtered spectrum. If a document is associated with a low power, it is considered highly relevant by the system, hence, the documents are ordered by increasing power. This is why the model is called Least Spectral Power Ranking (LSPR). The paper is organized as follows. In Section 2 an overview of the main frameworks of the current IR models is presented. Section 3 presents the theory and the methods of DFT and digital filters, which is required to understand the rest of the paper, especially for those unfamilar with digital signal processing. Section 4 introduces LSPR. After that, Section 5 shows how LSPR works with an example (three documents and a query), while Section 6 suggests some future research directions.
2
Related Work
One of the quests in IR is whether a theory exists or not. The question is not without merit because if the answer were affirmative this discipline would have a solid framework in which the methodologies and experiments could be developed with the same efficacy as that of the models and experiments used in Physics to interpret the macro- or microscopic world. A model is the methodological
An Information Retrieval Model Based on Discrete Fourier Transform
87
instrument for formulating theories about IR. Currently, IR systems combine a variety of techniques stemming from logical, vector-space and probabilistic models. It is this variety of combinations investigated over the last four decades that has produced a significant increase in retrieval effectiveness in the last almost twenty years as reported in [1]. The models proposed in the literature aim at capturing some of the variables which affect the retrieval process. Although they are somehow complex, these models are however quite oversimplified because they deliberately ignore many other variables, especially those related to the user and to the context in which the end users operate. What the models proposed in the literature have in common is the use of a “metaphor” for describing them in a simplified manner, thus making their experimentation and dissemination possible. The classical Boolean (or logical) model refers to set theory which views terms as document sets (operands) and queries as set operations [6] – non-classical logical models have been investigated by van Rijsbergen since 1980s [7]. The vectorspace models view terms as basis vectors, documents and queries as vectors of the same space as conceived in the 1960s by Salton [8, 9] before being developed into Latent Semantic Indexing [10] which computes alternative basis vectors for expressing complex terms. The probabilistic models for IR refer to the theories of probability (there are many [11]) which view terms or documents as events (or random variables). There have been many contributions to the development of the probabilistic models for IR, to cite but a few [12, 13, 14, 15, 3, 5]. Specific classes of probabilistic models refer to the Bayesian networks [16] and more recently to statistical language models [17, 3] inspired by speech recognition. The recent book by van Rijsbergen attempts to unify the logical, vector-space and probabilistic models into a single framework – the one used to describe quantum mechanics [4] – which views documents and terms as vectors, retrieval operations as projectors and probability as a trace-based function. The LSPR framework proposed in this paper differs from those proposed in the past because the underlying mathematical basis is different, yet there are some points in common. In our opinion, the main difference is the view through which the terms or in general the information content of the documents and of the queries are represented. In LSPR, the terms are sine curves which are a mathematical representation of signals. As these curves are functions, they can be treated algebraically. This property allows us to express queries and documents as functions which can be transformed for capturing salient features. In this paper, the DFT is the transform investigated. The LSPR framework employs the mathematical constructs proposed in [18, 19] which report a complex framework for document retrieval. In particular, [19] addresses the problem of matching queries and documents in which query terms occur at different positions and then with a variety of patterns. As matching documents by considering all the possible patterns is a difficult task, the authors propose to transform the spatial view of the query term occurrences into another domain. Therefore, each term of each document is represented as a signal and each document is divided into components. The term signals are transformed
88
A. Costa and M. Melucci
into term spectra which are a function of the magnitude and of the phase of the term occurring within each component of each document (magnitude and phase are then indexed by term, document and component). Document scores can be obtained by combining the term spectra components across terms under the assumption that a relevant document would have a high occurrence of query terms (implying a high magnitude of components) and a similar position of each pattern of query terms (implying similar phase). Unfortunately, this framework did not produce significant improvements over the standard TFIDF-based vectorspace model. What makes LSPR different from those models is that the spectrum and filtering are computed on the query and that the documents are the filters. Moreover, LSPR models the query terms as signals independently of the problem addressed in [19], the latter being a specific problem of text retrieval. Indeed, LSPR aims at making the sinusoidal signals a modeling paradigm (as the vector spaces, for example, are the paradigm of the vector-space model) rather than the approach of addressing the problem addressed in [19].
3
Background
3.1
Discrete Fourier Transform
The Fourier transform is essential in digital signal processing, because it allows us to analyze signals in the frequency domain in a more effective and efficient way than the time domain analysis [20, 21]. For continuous time signals the Fourier transform is employed, but if the signals are periodic the Fourier transform can be replaced with the Fourier series. In the discrete time domain, there is something similar and the transform related to discrete time periodic signals is the Discrete Fourier Transform (DFT). DFT is the only transform that can be computed numerically. Our interest in DFT derives from the need to analyze periodic sinusoidal discrete signals in the frequency domain. To this end the Fast Fourier Transform (FFT) is an O (N log N ) algorithm [22, 23]. This approach is a “divideand-conquer” algorithm and was conceived by Cooley and Tuckey in 1965.1 In the following, the DFT is more formally introduced. Let x (nT ) be a discrete time signal sample where T represents the distance between two samples. The DFT of this signal is ˜ (k) = T X
N −1
x (nT ) e−i2πkF nT
F =
n=0
Fc N
(1)
√ where i = −1, Fc is called sampling frequency, N is the number of samples and k identifies the k-th sample of the DFT. In the remainder of this paper, Fc = 1/T , therefore F = N1T . In this way, Eq. 1 is ˜ (k) = T X
N−1
x (nT ) e−i
2πn N k
.
n=0 1
Actually, the first who spoke of this algorithm was Gauss, in 1805.
(2)
An Information Retrieval Model Based on Discrete Fourier Transform
89
In order to obtain the power of the spectrum, the module of Equation 2 has to be computed from the complex numbers of the DFT. When x(nT ) is the sampling of a real periodic function, the module of the DFT is symmetric with respect to N/2, hence, we only need to operate with half of the spectrum. 3.2
Spectral Leakage
As stated earlier, the signals used in LSRP are supposed to be periodic, sinusoidal and discrete. The general form for these signals is Ai sin (2πfi nT ) .
(3)
The module of the DFT for the signal (3) has a peak proportional to amplitude Ai at the frequency fi (see Figure 4a), but if the frequency fi of the signal is not a multiple of F , a distorted DFT is obtained; Figure 4b shows this phenomenon, which is called spectral leakage. Usually, in digital signal processing spectral leakage is avoided because it makes the recognition of the correct frequencies of the signals difficult. In contrast in the LSPR model, spectral leakage is kept because the presence of some components with reduced amplitude at the frequencies close to fi makes it possible to better estimate the relevance of a document – the reason will be more clear in Section 4.1 – and introduces some constraints in the choice of F and fi for the signals (3).
(a) DFT of sinusoidal signal with fi (i = 0) multiple of F .
(b) DFT of sinusoidal signal with fi (i = 0) not multiple of F . Fig. 4.
90
A. Costa and M. Melucci
One of the most important theorems in digital signal processing is the sampling theorem (or Whittaker-Nyquist-Kotelnikov-Shannon theorem). There are a lot of equivalent definitions for this theorem. In this paper, the following is used: Theorem 1 (Sampling theorem [20]). Let x(t) be a signal with no frequencies higher than fM Hertz. Then x(t) is uniquely determined by its samples x(nT ) if Fc ≥ 2fM . This theorem is important in the framework proposed in this paper because it gives some conditions for the choice of the frequencies fi of the sinusoidal signals (3), and for the choice of the parameters of DFT. This will be explained better in Section 4. 3.3
Digital Filtering
One of the most important subjects in digital signal processing is digital filter design. A digital filter is a component that transforms an input signal x(nT ) to an output signal y(nT ) through the convolution of the input by the filter. Convolution is quite difficult to compute if the Fourier transform is not used. In contrast, after DFT has been applied to the signals and the filter, the computa˜ ˜ tion is feasible. Let X(k), Y˜ (k) and H(k) be, respectively, the DFT of the input signal, the DFT of the output signal and the DFT of the filter; the relationship among these transforms can be written as: ˜ X(k). ˜ Y˜ (k) = H(k)
(4)
This equation performs filtering in the LSPR model with a simple multiplication between two functions. There are a lot of filters which could be used in this context. In this paper, we used something similar to a notch filter which eliminates the spectrum near
Fig. 5. Module of the transform for a notch filter with cut-off frequency of 10 kHz
An Information Retrieval Model Based on Discrete Fourier Transform
91
a predefined frequency (called cut-off frequency), and keeps the spectrum in corrispondence with the other frequencies unchanged. The design of a filter appropriate for IR is still under investigation. Figure 5 shows an example of the module for a notch filter. Actually, in our tests we used a simplified filter, that is, a filter with a triangular form. This is better explained in Section 4.
4
Least Spectral Power Ranking
In this section Least Spectral Power Ranking (LSPR) which implements the DFT-based framework is introduced. LSPR consists of three main steps: 1. transformation of a query into a spectrum, 2. transformation of a document into a notch filter set, 3. document ranking. Before these three main steps some preprocessing is performed. The preprocessing phase includes all the operations usually done by every IR system, such as stemming and stopwords removal. In this first implementation of the model, we used the well-known TF-IDF weighting scheme for the documents in the collection. Actually, we used the normalized version of TF-IDF: if we call wij the TF-IDF weight for the term i in the document j, the normalized version is wij w ˆ ij = . 2 i wij
(5)
This expression has been used to implement the strength of the filter. The IDF weighting scheme has been used for implementing the amplitude of the peaks in the spectrum. Other weighting schemes can be used as well. 4.1
Query to Spectrum Transformation
The first step is to set the parameters of the DFT (such as F and N ) and of the sinusoidal signals. We choose to set F = 2 Hz, and to set every frequency of the signals (3) as an odd number: in this way it is not possible for a frequency to be an integer multiple of F , and this guarantees the spectral leakage. This choice allows us to write equation (3) in another way. Recalling that T = N1F : πfi n Ai sin (2πfi nT ) = Ai sin . (6) N Before explaining how we choose the frequency of each signal, the parameter N has to be set. It is important to remember that the spectrum is associated with the query, and that for each query term there is a sinusoidal signal. In this way the query is a sum of sinusoidal signals. The DFT of this sum is computed using N samples (N is both the number of samples of the sinusoidal signals used as input for the DFT and the number of points of the spectrum). For each term of
92
A. Costa and M. Melucci
the query there are 300 points of the spectrum and the peak of the relative signal is near point 200. These values are chosen because some experiments showed that the amplitude of the spectrum was less than 1% of the peak value 100 points after and before the peak. The use of 300 points assures that each sinusoidal signal is related to a disjoint interval of the spectrum; for example, if the query has 2 terms, there are 2 × 300 = 600 points. The number of points is the same for each term. Methods for choosing the number of points depending on the terms will be studied in the future. The FFT algorithm requires that this value is rounded at the closest power of two, and that this value has to be doubled due to the symmetry of the DFT (this last operation also has the advantage of fulfilling the conditions of Theorem 1); for example, 600 is rounded to 1024 and finally N = 2048 = 1024 × 2. Fc = N F = 2048 × 2 = 4096 Hz, where the maximum frequency fM among the sinusoidal signals is 1001 Hz, as shown below, so the condition of Theorem 1 (Fc ≥ 2fM ) is verified. In order to have the peak of the sinusoidal signal near to point 200 of the relative 300-point interval and to guarantee spectral leakage, its frequency is set to (300 · (i − 1) + 200) F + 1 Hz (7) where i refers to the i-th term of the query. Finally, the amplitude of the sinusoidal signal is the IDF weight of the term. Now we can show with an example how this method works. Suppose there is a query with 2 terms; for the sake of clarity, suppose also that the amplitudes of the two original terms are equal, and we call this value A. When N = 2048, the frequencies of the 2 signals are, respectively, 401 Hz and 1001 Hz using formula (7). The input for the FFT algorithm is a N -points vector S, and it is calculated as 401πn 1001πn S [n] = A sin + A sin , n = 1, 2 . . . , N. (8) 2048 2048 The FFT algorithm returns the DFT of this vector and the module is something similar to Figure 1a. In this figure we can see that the spectrum is symmetric with respect to N/2 (point 1024, which corresponds to 2048 Hz), and the peaks are near the points 200 and 500. Figure 1b shows more clearly the spectrum near point 200. Another important aspect showed by Figure 1b is the side effect of the spectral leakage. The decrease in importance follows the decrease in amplitude of the spectrum: this is an important hypothesis, and the efficacy of the presented model is based on it. Without spectral leakage, the figure would show a single peak at point 200 and the function would be zero at the other points, removing the decrease in the amplitude and making it impossible to estimate the decrease in importance, thus making futile the effect of the filtering operations. 4.2
Document to Filter Transformation
Each document is transformed into a filter set (one for each document term). As stated before, in LSPR, each filter is similar to a “notch” with triangular form.
An Information Retrieval Model Based on Discrete Fourier Transform
93
To be precise, there are two points, ZL (Zero Left ) and ZR (Zero Right ), where the module is 0, and there is a parameter called breadth such that the value of the filter before ZL−breadth and after ZR+breadth is 1. Finally, to connect ZL (ZR) with ZL−breadth (ZR+breadth) points, there is a linear function. An example of this filter is represented in Figure 2, where Figure 3 shows a fragment of a spectrum with a filter, and the obtained filtered spectrum. Two issues needs some explainations: the selection of the breadth (which represents the strength of the filter) and the position of the filter. The former is the weight TF-IDF of the document term normalized by the document length (w ˆ ij ), multiplied by a constant called selectivity. The value of this constant was set to 24, because some experiments showed that for values greater or smaller than 24, the MAP decreased. Probably, this value depends on the collection, but the interesting fact is that 24 is a sort of maximum for a function that relates selectivity and MAP. This reminds us of BM25 thus suggesting some possible future research directions. The latter is computed as follow: If the term was occurring in the query, ZL and ZR were points 200 and 201 of the relative interval; this means that for example if this term is the second of the query, ZL=500 and ZR=501, while ZL=200 and ZR=201 for the first query term, as in Figure 2. Otherwise, the filter is not created. 4.3
Document Ranking
Suppose a query is given as input to the system by the user and consider the set of documents retrieved by the system; they are reranked by a score computed after applying Equation (4). As a matter of fact, the module obtained by the query-tospectrum transformation is filtered by the filter set obtained after the documentto-filter transformation phase. Finally, the score obtained by each document is the sum of the module of the spectrum (the “power”) after filtering, hence, every document is associated with a score. The documents are ordered by increasing power because the more the filters decrease the power of the spectrum, the more the system consideres the document and the query related. Figure 6 summarizes LSPR. In this algorithm, the notation X ← [] represents the initialization of a vector, while X ← [X, i] is used to add a new element i to the vector X. Basically, the rows from 1 to 4 represent the initialization phase; the rows from 5 to 9 set the values of ZL for the filters (as explained in Section 4.2), and the rows from 10 to 22 describe the filtering operation. At the end, the power is computed and the algorithm returns the ranking list. The Filter function used within the algorithm has as inputs the spectrum, the ZL point of the filter, the breadth of the filter and the index i of the term of the query, whereas it returns as output the filtered spectrum. Basically, this function does the multiplications of the module of the spectrum (e.g. that in Figure 1a) by the frequency response of the filter (e.g. that in Figure 2), as shown in Figure 3.
94
A. Costa and M. Melucci
Algorithm: LSPR Input: query Q, collection of documents Output: ranked document list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Compute the vector S, as described in Section 4.1 spectrum ← |DFT(S )| I ← [] (initialize I) ZL ← [] (initialize ZL) for each term t of the query Q do Retrieve the posting list of t and call it L(t) I ← [I, index of t − 1] ZL ← [ZL, 300 · (index of t − 1) + 200] for each X ∈ L(t) do if it is the first time that document X is selected then f spectrum ← spectrum end if for i = 1 to |I| do breadth ← Round(selectivity · weight of qi in X) f spectrum ← Filter(f spectrum, ZL[i], breadth, I[i] − 1) end for
|f spectrum|
21
power [X] ←
f spectrum [k]
k=1
22 23 24 25
end for end for Order the documents by increasing power and save the list in result return result Fig. 6. An algorithm to summarize LSPR
4.4
Some Preliminary Results
LSPR was tested by using the CACM test collection and compared with two baselines: – A basic vector-space model. – The Divergence From Randomness (DFR) model implemented by Desktop Terrier v.1.1. After stopword removal and stemming, the retrieved list were measured by trec eval. The Mean Average Precision (MAP) of the vector-space model was 0.242, while the MAP of DFR was 0.329. The MAP of LSPR was 0.348, thus indicating a performance comparable to the state-of-the-art.
An Information Retrieval Model Based on Discrete Fourier Transform
5
95
Example
To give a global idea about how LSPR works, in this section we present an example with a collection of three documents and a query. Suppose we have the following documents in the collection (for the sake of clarity, we consider them after stopword removal, and without stemming): – D1 : (retrieval, data, author, book); – D2 : (information, computer, system, storage, information, data); – D3 : (information, retrieval, system, relevance, MAP, precision, recall, relevance). Suppose also that the query is Q : (information, retrieval, relevance). Table 1 represents the index of the collection, with the normalized TF-IDF weighting schema, and the query with the IDF weighting schema. Table 1. Normalized TF-IDF index matrix and IDF weights for query
author book computer data information MAP precision recall relevance retrieval storage system
D1 0.663 0.663 0 0.245 0 0 0 0 0 0.245 0 0
D2 0 0 0.596 0.220 0.440 0 0 0 0 0 0.596 0.220
D3 Q 0 0 0 0 0 0 0 0 0.136 0.585 0.367 0 0.367 0 0.367 0 0.735 1.585 0.136 0.585 0 0 0.136 0
At this point, we show how LSPR creates the spectrum of the query, as described in section 4.1. First, the frequencies of the terms “information”, “retrieval” and “relevance” are set respectively to 401 Hz, 1001 Hz and 1601 Hz, while N is 2048. Now, the vector S is computed as 401πn 1001πn 1601πn S [n] = 0.585 sin + 0.585 sin + 1.585 sin (9) 2048 2048 2048 where n = 1, 2 . . . , N . The vector S is given as input for the FFT algorithm, and we obtain the spectrum shown in Figure 7 (actually, due to the symmetry, we consider only half of the spectrum). Now, the documents are transformed into filters, as explained in section 4.2. Since D1 contains only the term “retrieval” of the query, with weight 0.245, it is transofmed into a filter with ZL=500 (which corresponds to the peak of the term
96
A. Costa and M. Melucci
Fig. 7. Spectrum for the query
(a) D1
(b) D2
(c) D3
Fig. 8. Spectra after the filtering operations performed on the three documents
“retrieval” in the spectrum) and with breadth=Round(24 · 0.245)= 6. Similarly, D2 corresponds to a filter (“information”) with ZL=200 and breadth=Round(24· 0.440)=11. Finally, D3 is transformed into a set of three filters; the first one (“information”) has ZL=200 and breadth=Round(24 · 0.136)=3, the second one (“retrieval”) has ZL=500 and breadth=Round(24 · 0.136)=3, and the third one has ZL=800 and breadth=Round(24 · 0.735)=18. At the end, after the filtering operations on the spectrum represented in Figure 7, we obtain the filtered spectra of Figure 8. The power (sum of the components of the spectrum) of the spectrum of Figure 7 is 13007.091, while the power of the spectra of Figure 8 are respectively 11836.613, 11649.498 and 6919.414; thus, according to the ranking rule of LSPR, the first document retrieved is D3, the second is D2 and the third is D1.
6
Conclusions and Future Work
This paper describes an IR model, called LSPR, that uses DFT. The paper is mainly theoretical because the focus of the paper was on the model whereas
An Information Retrieval Model Based on Discrete Fourier Transform
97
future work will focus on large scale experimentation. Nevertheless, some experiments were performed on a small scale. Preliminary results show that this model works well, and its efficacy is comparable with the efficacy of the state-of-the-art. LSPR was a first implementation of the framework based on DFT and a variety of implementations can be obtained through the series of parameters illustrated in the paper. It is our opinion that there is large room for improvement by setting appropriate values for sample sizes, thresholds and frequencies. At first glance, such an approach may be difficult, but the sound mathematical framework of DFT may provide useful guidance and significant improvements. Future work has three main objectives. First, efficiency needs to be improved. In this first implementation, since the effectiveness evaluation of the model was our focus, efficiency was not in the priority list. In order to test the model with larger collections these aspects are very important, hence, an optimization of the algorithm is necessary. Second, we have to look for better configuration than those reported in this paper. To obtain this, we will tune the parameters of the model. For example, in this implementation we use the TF-IDF weighting scheme (and IDF for the query), but other schemes such as Okapi BM25 can be tested. Another important issue is the similarity between the terms: using some clustering algorithm [24], we can estimate how two terms are related; these informations can be used to increase the efficacy of the model, for example by associating other different filters with the similar terms. In addition, the internal parameter of LSPR, such as the number of points of DFT and the selectivity will be investigated. Finally, the other feature of LSPR is that the functions are defined over the complex field, the latter being a characteristic of Geometry of IR by van Rijsbergen [4] in which no restrictions were placed on the scalars. The use of the complex field is a question of representation and many operations in that framework are always real (thus permitting ranking). The fact that scalars are complex and the descriptors are represented as sine curves, i.e. signals, helps enlarge the research in IR to other media than text by leveraging the extra representation power given by complex numbers. Moreover, the functions used in the Fourier transform are mutually orthogonal. This property recalls the techniques developed within the vector space models (e.g. Latent Semantic Analysis) and the framework developed by [4]. Further investigation will then be carried out on these connections. If the model leads to good results with larger experimental collections, we can test it in other contexts, such as web information retrieval. Actually, we are working on a modified version of LSPR for the recommender systems.
Acknowledgements We would like to thank Alberto Caccin, Albijon Hoxaj, Marco Lonardi, Enrico Martinelli and Giuseppe Soldo for the tests with Terrier. Moreover, we want to thank Emanuele Di Buccio for his precious suggestions. One of the authors (A. C.) is grateful to Digiteo Project 2009-55D “ARM” for financial support.
98
A. Costa and M. Melucci
References [1] Voorhees, E., Harman, D. (eds.): TREC: Experiment and Evaluation in Information Retrieval. The MIT Press, Cambridge (2005) [2] Robertson, S.: Salton award lecture: On theoretical argument in information retrieval. SIGIR Forum 34(1), 1–10 (2000) [3] Croft, W., Lafferty, J. (eds.): Language Modeling for Information Retrieval. Springer, Heidelberg (2003) [4] van Rijsbergen, C.: The Geometry of Information Retrieval. Cambridge University Press, UK (2004) [5] Fuhr, N.: A probability ranking principle for interactive information retrieval. Journal of Information Retrieval 11(3), 251–265 (2008) [6] Cooper, W.: Getting beyond Boole. Information Processing & Management 24, 243–248 (1988) [7] van Rijsbergen, C.: A non-classical logic for Information Retrieval. The Computer Journal 29(6), 481–485 (1986) [8] Salton, G.: Automatic information organization and retrieval. Mc Graw Hill, New York (1968) [9] Salton, G.: Mathematics and information retrieval. Journal of Documentation 35(1), 1–29 (1979) [10] Deerwester, S., Dumais, S., Furnas, G., Landauer, T., Harshman, R.: Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990) [11] Fine, T.: Theories of probability. Academic Press, London (1973) [12] Maron, M., Kuhns, J.: On relevance, probabilistic indexing and retrieval. Journal of the ACM 7, 216–244 (1960) [13] Robertson, S., Sparck Jones, K.: Relevance weighting of search terms. Journal of the American Society for Information Science 27, 129–146 (1976) [14] Robertson, S.: The probability ranking principle in information retrieval. Journal of Documentation 33(4), 294–304 (1977) [15] Robertson, S., Walker, S.: Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR), Dublin, Ireland, pp. 232–241 (1994) [16] Turtle, H., Croft, W.: Inference networks for document Retrieval. In: Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR), Brussels, Belgium (September 1990) [17] Ponte, J., Croft, W.: A language modeling approach to information retrieval. In: Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR), Melbourne, Australia, pp. 275–281. ACM Press, New York (1998) [18] Park, L.A.F., Ramamohanarao, K., Palaniswami, M.: Fourier domain scoring: A novel document ranking method. IEEE Trans. on Knowl. and Data Eng. 16(5), 529–539 (2004) [19] Park, L.A.F., Ramamohanarao, K., Palaniswami, M.: A novel document retrieval method using the discrete wavelet transform. ACM Trans. Inf. Syst. 23(3), 267– 298 (2005) [20] Oppenheim, A.V., Willsky, A.S., Nawab, S.H.: Signals & systems, 2nd edn. Prentice-Hall, Inc., Upper Saddle River (1996)
An Information Retrieval Model Based on Discrete Fourier Transform
99
[21] Mitra, S.K.: Digital Signal Processing: A Computer-Based Approach, 3rd edn. McGraw-Hill, New York (2006) [22] Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 2nd edn. MIT Press/McGraw-Hill Book Company (2000) [23] Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical recipes in C: The art of scientific computing, 2nd edn. Cambridge University Press, Cambridge (1992) [24] Croft, B., Metzler, D., Strohman, T.: Search Engines: Information Retrieval in Practice, 1st edn. Addison Wesley, Reading (2009)
Logic-Based Retrieval: Technology for Content-Oriented and Analytical Querying of Patent Data Iraklis Angelos Klampanos1, Hengzhi Wu2 , Thomas Roelleke2 , and Hany Azzam2 1
University of Glasgow, UK
[email protected] 2 Queen Mary University of London, UK {hzwoo,thor,hany}@dcs.qmul.ac.uk
Abstract. Patent searching is a complex retrieval task. An initial document search is only the starting point of a chain of searches and decisions that need to be made by patent searchers. Keyword-based retrieval is adequate for document searching, but it is not suitable for modelling comprehensive retrieval strategies. DB-like and logical approaches are the state-of-the-art techniques to model strategies, reasoning and decision making. In this paper we present the application of logical retrieval to patent searching. The two grand challenges are expressiveness and scalability, where high degree of expressiveness usually means a loss in scalability. In this paper we report how to maintain scalability while offering the expressiveness of logical retrieval required for solving patent search tasks. We present logical retrieval background, and how to model data-source selection and results’ fusion. Moreover, we demonstrate the modelling of a retrieval strategy, a technique by which patent professionals are able to express, store and exchange their strategies and rationales when searching patents or when making decisions. An overview of the architecture and technical details complement the paper, while the evaluation reports preliminary results on how query processing times can be guaranteed, and how quality is affected by trading off responsiveness.
1 Introduction Patent retrieval has emerged as an important application of information retrieval (IR). Especially, patent searching is likely to be more and more popular as standardised patent data corpora (e.g. MAtrixware REsearch Collection (MAREC)1 ) become available for research communities. In this paper, we discuss a logic-based solution for querying patent data. We maintain that logic-based retrieval can be particularly useful for patent searching due to its abstraction and descriptive power, its capability to be used in a traceable search workflow and the fact that it can be used through query languages similar to the ones used by IP professionals. Additionally, the use of probabilistic logics allows for probabilistic reasoning about entities described by the underlying data, in a dynamic manner. 1
http://www.ir-facility.org/services/data/matrixware-research-collection
H. Cunningham, A. Hanbury, and S. R¨uger (Eds.): IRFC 2010, LNCS 6107, pp. 100–119, 2010. c Springer-Verlag Berlin Heidelberg 2010
Logic-Based Retrieval
101
In particular, by the term “logical-based retrieval” we refer to the following properties: 1. The ranking of objects is based on the probability that the retrieved objects (for instance documents, inventors, etc.) are based on the probability of the objects being implied by the query. 2. The reasoning about objects, that is the retrieval of data sources, leads to a ranking of sources and this ranking affects the ranking of documents, the ranking of documents leads to a ranking of inventors and/or assignees, and so forth. 3. Logical retrieval allows for open-box, high-level abstraction modelling, in contrast to black-box IR systems where the retrieval function is an integral part of the system (library function). The properties of logical retrieval meet the requirements of patent searching in the sense that patent searching is a complex retrieval task that requires reasoning about objects. Furthermore, it also requires modelling IR in such a way that users (patent searchers) can understand and modify the ranking and reasoning process. From private discussions with patent searchers it is understood that them being in full control of the retrieval process is essential for generating the trust needed to move on from Boolean retrieval. As follows from the above properties, logic-based IR fulfills this requirement. The following example underlines the difference between basic, topical retrieval and complex retrieval. For example, after an initial search for patents about semi-conductors (the initial search typically is a basic topical search), the task is to explore the patents of experienced inventors who filed the most relevant patents about semi-conductors. Logical retrieval supports the reasoning process, that is the modelling of vague predicates such as experienced inventor, the post-processing of retrieval results, and, in general, the combination of evidence from multiple data sources and different dimensions (e.g. content dimension, attribute/field dimensions, semantic annotations). Thus, probabilistic logical retrieval supports the decision-making processes faced by searchers who explore several data sources, compare, combine and re-rank results. However, the flexibility and power of abstraction of logic-based IR comes at the cost of efficiency. Scaling logical retrieval to the data volume of the patent corpus is a challenging task. By combining distributed IR and logical retrieval we can overcome the inherent efficiency problems of logic-based IR and achieve efficient processing of probabilistic logical queries. In distributed information retrieval (DIR) the interface between users and content resourcesis typically provided by a single broker that manages all the transactions between the client (the user) and the servers (the IR engines). This broker is responsible for forwarding queries to the most relevant of the providers as well as fusing and returning result-lists to the user. This requires that the broker has some knowledge over the retrievable content of the participating providers before any query routing can take place. DIR can be described as the set of the following challenges[1]: Resource description is the process during which individual providers or IR engines inform the broker/managing node of their content. Resource selection is the process during which, given a query, the broker makes an informed decision as to which providers may have content that is relevant to the query. These will be the providers that will end-up receiving and responding to the query. Final, results’ merging is the merging of result-lists that
102
I.A. Klampanos et al.
the broker has to perform after having received results from the various, previously selected, providers. The broker has to merge the various result lists into a single one before routing them back to the user. The three problems outlined above occur mainly in a distributed system comprised of autonomous retrieval engines. In a patent-searching scenario we can expect the index distribution to occur in a centrally controlled and computationally efficient environment. In our case we assume the existence of a highly parallel computer infrastructure (for instance IRF’s LDC), however our proposed solution is not restricted to such environments. In distributed logic-based IR, resource selection and fusion are the most critical parts of the solution since they can have a significant impact on retrieval effectiveness and efficiency. In this paper we will show how we model resource selection and fusion in our solution. In the same way as other components of a logical IR system, these two are also pluggable and IP professionals could alter or augment them in order to suit their needs. Again, by supporting the distribution process logically, the IP professionals can here too be in control, being able to drive their work through reasoning about resources and merging strategies in the same manner as they can reason about documents and related patent entities.
2 Motivation and Background 2.1 Logic-Based Retrieval With logic-based retrieval, we refer to the approach of applying a logic-based technology to retrieval tasks. Inherent to logic-based technology is the usage of structured languages to represent knowledge and queries. Therefore, logic-based technology has a higher expressive power than the “bag-of-words” approach of traditional free-textoriented IR. Moreover, logical retrieval support “reasoning”. For example, logical languages allow to express strategic and analytical queries. “Find the patents of Kraft and Cadbury; compare the most important patents; which inventors are the dominant ones in the food provision market.” The application of logical retrieval to large-scale, partly poorly structured data, is challenging for a number of reasons. Because of its expressiveness, logical retrieval demands an algebraic evaluation that is significantly more compex than fetching documnt id’s from the posting list of an inverted list. Also, the modelling of selection, ranking, and fusion strategies in a logical language is challenging. However, the open-box, high-level abstraction is precisely what satisfies the requirements of complex (since advanced, strategic and anlytical) search requirements as they occur, for example, in patent search. The properties of logical retrieval meet the requirements of patent searching in the sense that patent searching is a complex retrieval task that requires reasoning about objects. Furthermore, it also requires modelling IR in such a way that users (patent searchers) can understand and modify the ranking and reasoning process. From discussions with patent searchers this aspect of “I need to understand what is going on” is essential for generating the trust needed to move on from Boolean retrieval, which gives the user a sense of being in “full control” of what is going on.
Logic-Based Retrieval
103
The following example underlines the difference between basic, topical retrieval and complex retrieval. For example, after an initial search for patents about semi-conductors (the initial search is the basic, so-called topical search), the task is to explore the patents of the experienced inventors who filed the most relevant patents about semi-conductors. Logical retrieval supports the reasoning process, that is the modelling of vague predicates such as experienced inventor, the post-processing of retrieval results, and, in general, the combination of evidence from multiple data sources and different dimensions (e.g. content dimension, attribute/field dimensions, semantic annotations). Thus, probabilistic logical retrieval supports the decision-making processes faced by searchers who explore several data sources, compare, combine and re-rank results. Scaling logical retrieval to the data volume size of the patent corpus is challenging. In this paper, we discuss how distributed IR and logical retrieval can be combined to achieve the efficient processing of so-called probabilistic logical programs. 2.2 Probabilistic Datalog Probabilistic Datalog (PD) ([2,3]) is a language for probabilistic reasoning, where deterministic Datalog has its roots in deductive databases. PD was extended in [4,5] to improve its expressiveness and scalability for modelling IR models (ranking functions). In particular, in this paper we use the PD language in order to demonstrate possible implementations for source selection, result fusion, and retrieval strategy composition. Alternative languages, such as probabilistic SQL or probabilistic relational algebra, could have been used instead but PD was preferred due to the clarity of its semantics and compact syntax. To introduce what PD offers, an extract of the syntax of PD is shown in Figure 1. In this syntax, expressions between a pair of curly brackets, i.e. ‘{’ and ‘}’, are optional. A PD rule consists of a head and a body. A head is a goal, and a body is a subgoal list. A PD definition is similar in syntax to a rule, apart from the ‘:=’ indicating the definition; a rule is executed if required (delayed execution), whereas a definition is executed when read by the parser (instant execution). From this point of view, definitions can be views as “materialised views” (as known for DB systems). Specific to the PD shown here (based on [4]) and different from the PD in [6], is the specification of aggregation and estimation assumptions in goals and subgoals. The assumption between predicate name and argument list is the so-called aggregation assumption. For example, for disjoint events, the sum of probabilities is the resulting tuple probability. The assumptions ’DISJOINT’ and ’SUM’ are synonyms, and so are ’INDEPENDENT’ and ’PROD’. The assumption in a conditional is the so-called estimation assumption. For example, for disjoint events, the subgoal “index(Term, Doc) | DISJOINT(Doc)” expresses the conditional probability P(Term|Doc) derived from the statistics in relation “index”. Complex assumptions such as DF (for document frequency) and MAX IDF (max inverse document frequency) can be specified to describe in a convenient way probabilistic parameters commonly used in information retrieval. Expressions with complex assumptions can be decomposed in PD programs with traditional assumptions only. However, for improving the readability and processing (optimisation), complex assumptions can be specified. The decomposition of complex assumptions is shown in [4].
104
I.A. Klampanos et al. pdRule ::= head ’:-’ body pdDef ::= head ’:=’ body head ::= goal body ::= subgoalList subgoalList ::= subgoal {’&’ subgoalList} goal ::= basicGoal | aggGoal | estGoal basicGoal ::= NAME ’(’ argList ’)’ aggGoal ::= NAME aggAssumption ’(’ argList ’)’ | estGoal ::= NAME ’(’ argList ’)’ ’|’ {estAssumption} evidenceKey subgoal ::= goal argList ::= arg {’,’ argList} varList ::= VARIABLE {’,’ varList} var ::= VARIABLE arg ::= VARIABLE | constant constant ::= STRING | NAME | NUMBER evidenceKey ::= ’(’ varList ’)’ tradAssumption ::= ’DISJOINT’ | ’INDEPENDENT’ | ’SUBSUMED’ irAssumption ::= ’DF’ | ’MAX IDF’ | ... aggAssumption ::= tradAssumption | ’SUM’ | ’MAX’ | ... estAssumption ::= tradAssumption | comlexAssumption Fig. 1. Extract of the syntax of probabilistic datalog (PD)
The syntax specification is simplified here to improve readability. For example, for reasons of context-free syntax parsing, constants in goal argument lists need to be quoted, whereas in subgoals, bare constants are sufficient, and quoting of constants is optional. The syntax specification does not make explicit this and other implementational details. A rule is evaluated such that the head is true if and only if the body is true. For example, the following rule demonstrates a common term matching strategy in IR. Queries are represented in relation “qterm(T,Q)” (T is a term, Q is a query ID), and the collection is represented in “term(T,D)” (D is a document id). If T occurs in Q, and T is a term in D, then D is retrieved for Q. 1
coord match retrieve(D, Q) :− qterm(T, Q) & term(T, D);
The main aim of this paper is to demonstrate the usefulness and descriptive power of PD for distributed IR and, in particular, resource selection tasks. We maintain that modelling such tasks at a high level gives expert search users, such as IP professionals, flexibility and tractability of search sessions and predictable retrieval effectiveness. 2.3 Distributed Information Retrieval Distributed information retrieval (DIR), also some times referred to as federated search, deals with the problem of locating and retrieving information from a set of databases
Logic-Based Retrieval
105
or IR engines, as opposed to the classical of IR which relies on a single, centralised store. The interface between the user and the information sources2 is typically provided by a single broker that manages all the transactions between the client (the user) and the servers (the IR engines). This broker is responsible for forwarding queries to the most relevant of the providers as well as fusing and returning result-lists to the user. This requires that the broker has some knowledge over the retrievable content of the participating providers before any query routing can take place[7]. Callan [1] accurately describes the issue of DIR as the set of the following problems: Source description: individual providers or IR engines inform the broker of their content. Source selection: given a query, the broker “retrieves” the sources to “retrieve” from, i.e. the broker ranks the sources, and selects the top-n resources for retrieval. Result fusion: the broker fuses the results received from the previously selected providers. The three phases outlined above occur mainly in a distributed system comprised of autonomous retrieval engines. In a patent-searching scenario we can expect the index distribution to occur in a centrally controlled and computationally efficient environment. In fact, at the centre of our solution there lies a highly parallel computer infrastructure, however our proposed solution is not restricted to such environments. As we discuss in Section 4, the existence of such a tightly controlled environment makes the problem of results’ fusion somewhat redundant. Such an environment, along with the adoption of a DB+IR approach, limits the effects of resource description. However, resource selection is still critical, since different strategies are expected to have a significant impact on retrieval effectiveness and efficiency. For these reasons, resource selection is central to this study, where we attempt to showcase the usefulness, versatility and expressiveness of DB+IR approaches.
3 Data Source Selection In this section we present the two well-known problems of distributed IR of resource description and selection and how these can be expressed in PD and consequently run on a DB+IR system. Resource description is the process by which retrieval engines describe their content to a broker node responsible for conducting the retrieval tasks offered by a distributed or parallel IR system. The type of information shared during resource description, as well as its structure and format, may generally affect the quality of searching. This is especially true in distributed systems linked by slow connections where the transferring of whole documents might be prohibitive. Compact descriptions of documents and incremental description accumulation algorithms (such as query sampling) are therefore generally preferred. As we assume the existence of a multi-CPU supercomputer, the problem of resource description is not central to this study. In our case the broker will 2
In this study the terms sources, referring to data sources and the term resources referring to remote/networked resources coincide and so they are used interchangeably.
106
I.A. Klampanos et al.
Term
term index Doc Source
sailing boats sailing greek islands sailing boats sailing sailing boats sailing sailing boats
s1/doc1 s1/doc1 s1/doc2 s1/doc3 s1/doc4 s2/doc1 s2/doc1 s2/doc2 s2/doc3 s2/doc4 s3/doc1 s3/doc2 s3/doc3
source1 source1 source1 source1 source1 source2 source2 source2 source2 source2 source3 source3 source3
df index Term Source 0.500000 0.250000 0.250000 0.250000 0.750000 0.500000 0.666667 0.333333
sailing boats greek islands sailing boats sailing boats
source1 source1 source1 source1 source2 source2 source3 source3
Fig. 2. Illustraction of data representation
have access to the original parallelised indexes directly and so the resource descriptions will stem from these indexes. In the DB+IR approach we follow, the basic relation representing documents is simply defined by the schema “term(Term, Document)”. In a parallel architecture where many independent engines coexist, we can represent the global document collection by the schema “term src(Term, Document, Source)”, where “Source” uniquely identifies the node sharing the given document. This basic form of resource description provides for a solid basis upon which to distribute collections and searching tasks in a manageable fashion. A small example of such a relation can be seen in Figure 2. This toy collection is also used for demonstrating our modular approach to retrieval strategy modelling in the following sections. Resource selection is the process by which, given a query, a main broker node determines the most appropriate nodes to answer the query. This selection of appropriate resources is based on the descriptions previously made known to the broker by the nodes sharing the searchable content. Therefore, instead of having to reason over the whole collection, an effective resource selection process enables the most relevant set of sources to be probabilistically selected, which decreases the space of evidence needed to be considered upon retrieval 3.1 DF-Based Selection A naive approach to resource selection can be based simply on the document frequencies of query terms inside the document collections to be examined. According to this strategy, the collections with more documents containing the query terms are to be preferred over those with fewer documents. The disadvantages of such a selection strategy can be easily deduced. After all, DF characterises a collection as a whole and does not provide any insight into the retrieval units, the individual documents. However, we decided to include this resource selection strategy in this study because it can serve as a
Logic-Based Retrieval
107
baseline for the comparison of resource selection strategies and also for demonstrating the flexibility of PD when it comes to modularising distributed IR tasks. The DF-based score is defined as follows: nL (t, q) NL (q) nD (t, c) PD (t|c) := ND (c) PL (t|q) :=
Score df(c, q) :=
∑ PL(t|q) · PD(t|c)
(1) (2) (3)
t∈Q
where PL (t|q) denotes the location-based probability of a term t given a query q and is defined by the number of locations (tuples) in which t occurs in q, nL (t, q), over the number of locations present in q, NL (q). Similarly, PD (t|c) denotes the documentbased probability of a term t given a document collection c, defined as the number of documents in which t occurs in c, nD (t, c) over the total number of documents present in c, ND (c). Score df(c, q) can then be defined as the sum of the products of PL (t|q) and PD (t|c), which can also be seen as the sum of documents frequencies, weighted on the query terms. The sum is over the terms in set Q, the set of query terms (for non-query terms, PL (t|q) = 0). A DF-based resource selection strategy expressed in PD can be seen in Figure 3.
1 2 3 4 5
7 8 9 10
12 13 14 15
# DF−based term probability per source # P(t|source): based on document frequency df src(Term, ’source1’) | DF() :− source1(Term, Doc); df src(Term, ’source2’) | DF() :− source2(Term, Doc); df src(Term, ’source3’) | DF() :− source3(Term, Doc); # Query term weighting/normalisation # sum t P(t|q) = 1.0 norm qterm(Term, Query) :− qterm(Term, Query) | DISJOINT(Query); # Retrieval/ranking of sources # sum t P(t|q) ∗ P(t|source) selected SUM(Source, Query) :− norm qterm(Term, Query) & df src(Term, Source);
Fig. 3. DF-based data source selection strategy
The first three rules (lines 3-5) utilise the assumption “df” in order to define the global DF-based probability relation for three data sources. In lines 9, 10, we define the relation “norm qterm” which is the normalised version of the query “qterm” (∑t P(t|q) = 1). The “DISJOINT” assumption is an estimation assumption that defines the evidence
108
I.A. Klampanos et al.
probability as the sum of the probabilities of the tuples with the same evidence key (see Section 2.2). In lines 14, 15 we select the data sources that yield the highest DF scores for our query terms by joining the sources with the “norm qterm” relation. The join operation returns the tuples that match the join condition, and the “SUM” assumption selects the specified columns and aggregates the coinciding tuples in order to sum the probabilities. The next section describes a language modelling-based selection strategy. 3.2 LM-Based Selection In [8] the authors employ a selection rule based on LM. According to this rule, for a query q, a source/collection c, and the global collection g (g comprises all sources), the score is defined as follows: Score lm(c, q) = ∏ [λ · P(t|c) + (1 − λ) · P(t|g)]
(4)
t∈Q
where P(t|c) is the term probability in collection c, P(t|g) is the term probability in the global term probability across all the collections being handled by the system, λ is the mixture parameter, and Q is the set of query terms (here we employ the mathematical convention to use lower case letters for events, and upper case letters for sets
1 2 3
5 6 7
9 10
12 13 14 15
17 18 19
21 22
# P(t|c): source−specific term probability p tc SUM(Term, Source) :− term src(Term, DocId, Source) | DISJOINT(Source); # P(t|g): global term probability (over all sources) p tg SUM(Term) :− term src(Term, DocId, Source) | DISJOINT(); 0.7 model(collModel); 0.3 model(globalModel); w p tc(Term, Source) :− model(collModel) & p tc(Term, Source); w p tg(Term, Source) :− model(globalModel) & p tg(Term) & sources(Source); w p tcg mix(Term, Source) :− w p tc(Term, Source); w p tcg mix(Term, Source) :− w p tg(Term, Source); p tcg mix SUM(Term, Source) :− w p tcg mix(Term, Source); selected PROD(Source, Query) :− qterm(Term, Query) & p tcg mix(Term, Source);
Fig. 4. LM-based data source selection strategy
Logic-Based Retrieval
109
and random variables). Given the resource description introduced in Section 2.3, such a selection strategy can be readily expressed in PD, as shown in Figure 4 3 . In this figure, the rules defined in lines 2 and 6 express the main components of the LM-based selection algorithm, P(t|c) and P(t|g) respectively. It is worth noting that the definition of these two expressions is fairly symmetric, apart from the presence of the attribute “Source” in “p tc”. This is because in “p tc” the term probability is sourcespecific, whereas in “p tg” it is defined across all data sources. This demonstrates the ability of PD to express probabilistic relations on their conceptual and logical characteristics. In lines 9 and 10 we express the mixture parameter λ as a new relation “model” containing exactly two tuples with the corresponding probabilities. Lines 17-19 define the relation “p tcg mix”, which aggregates the probabilities of duplicate tuples as appropriate. Finally, in Line 21, we define the list of the selected data sources, given the query at hand “qterm”. The “PROD” assumption aggregates the probabilities through multiplication, as in Equation 4. 3.3 Other Approaches In addition to the DF-based and LM-based source selection models, there are a range of other approaches to rank sources. [10] discusses a model that is based on the probability ranking principle, saying that there is a cost related to reading relevant and non-relevant documents. Based on this cost-based view, a cost estimate can be defined, and a ranking is optimal if costs are minimal. Other approaches include the application of a “TF-ISF” ranking, where TF is the within-source term frequency, and ISF is the inverse source frequency. In essence, this is a ranking of sources derived from the document-oriented TF-IDF approach. In [11] the so-called CORI model is introduces, with the idea to merge rankings where the origin and semantics of the source-specific retrieval status values might not be known, and therefore, need to be normalised for merging or fusion to take place.