Recent Advances in Natural Language Processing IV
AMSTERDAM STUDIES IN THE THEORY AND HISTORY OF LINGUISTIC SCIENCE G...
26 downloads
927 Views
2MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
Recent Advances in Natural Language Processing IV
AMSTERDAM STUDIES IN THE THEORY AND HISTORY OF LINGUISTIC SCIENCE General Editor E.F.K. KOERNER (Zentrum für Allgemeine Sprachwissenschaft, Typologie und Universalienforschung, Berlin) Series IV – CURRENT ISSUES IN LINGUISTIC THEORY Advisory Editorial Board Lyle Campbell (Salt Lake City); Sheila Embleton (Toronto) Brian D. Joseph (Columbus, Ohio); John E. Joseph (Edinburgh) Manfred Krifka (Berlin); E. Wyn Roberts (Vancouver, B.C.) Joseph C. Salmons (Madison, Wis.); Hans-Jürgen Sasse (Köln)
Volume 292
Nicolas Nicolov, Kalina Bontcheva, Galia Angelova and Ruslan Mitkov (eds.) Recent Advances in Natural Language Processing IV. Selected papers from RANLP 2005
Recent Advances in Natural Language Processing IV Selected papers from RANLP 2005
Edited by
Nicolas Nicolov Umbria, Inc.
Kalina Bontcheva University of Sheffield
Galia Angelova
Bulgarian Academy of Sciences
Ruslan Mitkov
University of Wolverhampton
JOHN BENJAMINS PUBLISHING COMPANY AMSTERDAM/PHILADELPHIA
4-
The paper used in this publication meets the minimum requirements of American National Standard for Information Sciences — Permanence of Paper for Printed Library Materials, ANSI Z39.48-1984.
Library of Congress Cataloging-in-Publication Data RANLP 2005 (2005 : Borovets, Bulgaria) Recent advances in natural language processing IV : selected papers from RANLP 2005 / edited by Nicolas Nicolov ... [et al.]. p. cm. -- (Amsterdam studies in the theory and history of linguistic science. Series IV, Current issues in linguistic theory, ISSN 0304-0763 ; v. 292) Includes bibliographical references and index. 1. Computational linguistics--Congresses. I. Nicolov, Nicolas. II. Title. P98.R36 2005 410.285--dc22 2007043413 ISBN 978 90 272 4807 7 (Hb; alk. paper) © 2007 – John Benjamins B.V. No part of this book may be reproduced in any form, by print, photoprint, microfilm, or any other means, without written permission from the publisher. John Benjamins Publishing Co. • P.O.Box 36224 • 1020 ME Amsterdam • The Netherlands John Benjamins North America • P.O.Box 27519 • Philadelphia PA 19118-0519 • USA
CONTENTS Editors’ Foreword
ix
I. COMPUTATION FOR LINGUISTICS John Nerbonne Linguistic challenges for computationalists
1
II. INFORMATION EXTRACTION & INDEXING Ralph Grishman NLP: An information extraction perspective
17
Florian Seydoux & Jean-C´edric Chappelier Semantic indexing using minimum redundancy cut in ontologies
25
Niraj Aswani, Valentin Tablan, Kalina Bontcheva & Hamish Cunningham Indexing and querying linguistic metadata and document content
35
Irina Matveeva, Gina-Anne Levow, Ayman Farahat & Christiaan Royer Term representation with Generalized Latent Semantic Analysis 45
III. PARSING Ming-Wei Chang, Quang Do & Dan Roth Multilingual dependency parsing: A pipeline approach
55
Sandra K¨ ubler How does treebank annotation influence parsing? Or how not to compare apples and oranges
79
Laura Alonso, Joan Antoni Capilla, Irene Castell´ on, Ana Fern´ andezMontraveta & Gloria V´ azquez The SenSem project: Syntactico-semantic annotation of sentences in Spanish 89
vi
CONTENTS
IV. ANAPHORA & REFERRING EXPRESSIONS Robert Dale Generating referring expressions: Past, present and future Erhard W. Hinrichs, Katja Filippova & Holger Wunsch A data-driven approach to pronominal anaphora resolution for German
99
115
V. CLASSIFICATION Nicolas Nicolov & Franco Salvetti Efficient spam analysis for weblogs through URL segmentation
125
Filip Ginter, Sampo Pyysalo & Tapio Salakoski Document classification using semantic networks with an adaptive similarity measure
137
Rada Mihalcea & Samer Hassan Text summarization for improved text classification
147
Caroline Sporleder & Alex Lascarides Exploiting linguistic cues to classify rhetorical relations
157
VI. TEXTUAL ENTAILMENT & QUESTION ANSWERING Milen Kouylekov & Bernardo Magnini Tree edit distance for textual entailment
167
J¨ org Tiedemann A genetic algorithm for optimising information retrieval with linguistic features in question answering
177
Vasile Rus & Arthur Graesser Lexico-syntactic subsumption for textual entailment
187
Courtney Corley, Andras Csomai & Rada Mihalcea A knowledge-based approach to text-to-text similarity
197
CONTENTS
vii
VII. ONTOLOGIES Keiji Shinzato & Kentaro Torisawa A simple WWW-based method for semantic word class acquisition 207 Eduard Barbu & Verginica Barbu Mititelu Automatic building of Wordnets
217
VIII. MACHINE TRANSLATION Stelios Piperidis, Panagiotis Dimitrakis & Irene Balta Lexical transfer selection using annotated parallel corpora
227
Victoria Arranz, Elisabet Comelles & David Farwell Multi-perspective evaluation of the FAME speech-to-speech translation system for Catalan, English and Spanish
237
D´ aniel Varga, P´eter Hal´ acsy, Andr´ as Kornai, Viktor Nagy, L´ aszl´ o N´emeth & Viktor Tr´ on Parallel corpora for medium density languages
247
IX. CORPORA Anne De Roeck The role of data in NLP: The case for dataset profiling
259
Anne De Roeck, Avik Sarkar & Paul H. Garthwaite Even very frequent function words do not distribute homogeneously
267
Lucia Specia, Maria das Gra¸cas V. Nunes & Mark Stevenson Exploiting parallel texts to produce a multilingual sense tagged corpus for word sense disambiguation
277
Francis Chantree, Alistair Willis, Adam Kilgarriff & Anne de Roeck Detecting dangerous coordination ambiguities using word distribution
287
List and Addresses of Contributors
297
Index of Subjects and Terms
303
Editors’ Foreword This volume brings together revised versions of a selection of papers presented at the Second International Conference on “Recent Advances in Natural Language Processing” (RANLP’05) held in Borovets, Bulgaria, 21–23 September 2005. The aim of the conference was to give researchers the opportunity to present new results in Natural Language Processing (NLP) based on modern theories and methodologies. The conference was preceded by three days of tutorials (18–20 September 2005). Invited lecturers were: Jan Hajic, Zdenka Uresova (Charles University, Prague) The Prague Dependency Treebank and Valency Annotation Bernardo Magnini (IRST, Trento) Open Domain Question Answering: Techniques, Systems and Evaluation Rada Mihalcea (University of North Texas) Graph-based Algorithms for Information Retrieval and Natural Language Processing Dragomir Radev (University of Michigan) Information Retrieval John Tait (University of Sunderland ) Image Retrieval Michael Zock (CNRS ) Natural Language Generation: Snaphot of a fast evolving discipline The conference received 160 submissions from more than 20 countries. Whilst we were delighted to have so many contributions, restrictions on the number of papers which could be presented in three days forced us to be more selective than we would have liked (the acceptance ratio for regular papers at RANLP’05 was 13.75% — significantly lower than for the previous conference). From the papers presented at RANLP’05 we have selected the best for this book, in the hope that they reflect the most significant and promising trends (and successful results) in NLP. Keynote speakers who gave invited talks were: • Ido Dagan (Bar-Ilan University, Israel) • Robert Dale (Macquarie University, Australia) • Ralph Grishman (New York University, U.S.A.) • Makoto Nagao (NICT, Tokyo, Japan) • John Nerbonne (University of Groningen, The Netherlands) • Anne de Roeck (Open University, U.K.)
x
EDITORS’ FOREWORD
The book covers a wide variety of nlp topics: datasets, annotation, treebanks, parallel corpora, information extraction, parsing, word sense disambiguation, translation, indexing, ontologies, question answering, document similarity, spam analysis, document classification, anaphora resolution, referring expressions generation, textual entailment, latent semantic analysis, summarization, rhetorical relations, etc. To help the reader find his/her way we have added an index which contains major NLP terms used throughout the volume. We have also included a list and addresses of all contributors. We would like to thank all members of the Program Committee. Without them the conference, although well organised, would not have had an impact on the development of NLP. Together they have ensured that the best papers were included in the final proceedings and have provided invaluable comments for the authors, so that the papers are ‘state of the art’. The following is a list of those who participated in the selection process and to whom a public acknowledgement is due: Eneko Agirre Elisabeth Andre Galia Angelova Amit Bagga Branimir Boguraev Kalina Bontcheva Ant´ onio Branco Eugene Charniak Dan Cristea Hamish Cunningham Walter Daelemans Ido Dagan Robert Dale Thierry Declerck Ga¨el Dias Rob Gaizauskas Alexander Gelbukh Ralph Grishman Walther von Hahn Jan Hajic Johann Haller Catalina Hallett Graeme Hirst Eduard Hovy Nikiforos Karamanis Martin Kay Dorothy Kenny Alma Kharrat
(Univ. of the Basque Country, Donostia) (University of Augsburg) (Bulgarian Academy of Sciences) (Ask Jeeves, Piscataway) (IBM, T. J. Watson Research Center) (University of Sheffield) (University of Lisbon) (Brown University, Providence) (University of Ia¸si) (University of Sheffield) (University of Antwerp) (Bar Ilan Univ. & FocusEngine, Tel Aviv) (Macquarie University) (DFKI GmbH, Saarbr¨ ucken) (Beira Interior University, Covilh˜ a) (University of Sheffield) (Nat. Polytechnic Inst., Mexico) (New York University) (University of Hamburg) (Charles University, Prague) (University of Saarland) (Open University, Milton Keynes) (University of Toronto) (ISI, University of Southern California) (University of Wolverhampton) (Stanford University) (Dublin City University) (Microsoft Natural Language Group)
EDITORS’ FOREWORD
Sandra K¨ ubler Manfred Kudlek Lori Lamel Shalom Lappin Yves Lepage Anke Luedeling Bente Maegaard Bernardo Magnini Nuno Mamede Inderjeet Mani Carlos Martin-Vide Tony McEnery Wolfgang Menzel Rada Mihalcea Ruslan Mitkov Andres Montoyo Rafael Mu˜ noz-Guillena Masaki Murata Makoto Nagao Preslav Nakov Ani Nenkova John Nerbonne Nicolas Nicolov Kemal Oflazer Constantin Or˘ asan Chris Paice Manuel Palomar Victor Pekar Fabio Pianesi Stelios Piperidis John Prager G´ abor Pr´ osz`eky Stephen Pulman James Pustejovsky Jos´e Francisco Quesada Moreno Dragomir Radev Fuji Ren Ellen Riloff Anne de Roeck Antonio Rubio Franco Salvetti Fr´ed´erique Segond Kiril Simov Richard Sproat Steffen Staab Keh-Yih Su
(Indiana University, Bloomington) (University of Hamburg) (LIMSI/CNRS, Orsay) (King’s College, London) (ATR, Japan) (Humboldt University, Berlin) (CST, University of Copenhagen) (ITC-IRST, Trento) (INESC, Lisbon) (Georgetown University) (Univ. Rovira i Virgili, Tarragona) (Lancaster University) (University of Hamburg) (University of North Texas, Denton) (University of Wolverhampton) (University of Alicante) (University of Alicante) (NICT, Japan) (NICT, Japan) (University of California at Berkeley) (Columbia University) (University of Groningen) (Umbria, Inc., Boulder) (Sabanci University, Istanbul) (University of Wolverhampton) (Lancaster University) (University of Alicante) (University of Wolverhampton) (ITC-IRST, Trento) (ILSP, Athens) (IBM, T. J. Watson Research Center) (MorphoLogic, Budapest) (Oxford University) (Brandeis University) (University of Seville) (University of Michigan) (University of Tokushima) (University of Utah) (Open University, Milton Keynes) (University of Granada) (Umbria, Inc., Boulder) (Xerox Research Centre, Grenoble) (Bulgarian Academy of Sciences) (U. of Illinois at Urbana-Champaign) (University of Koblenz-Landau) (Behavior Design Corporation, Hsinchu)
xi
xii
EDITORS’ FOREWORD
John Tait Kristina Toutanova Isabel Trancoso Jun’ichi Tsujii Hans Uszkoreit Karin Verspoor Piek Vossen Yorick Wilks Michael Zock
(University of Sunderland) (Stanford University) (INEC, Lisbon) (University of Tokyo) (University of Saarland, Saarbr¨ ucken) (Los Alamos National Laboratory) (Irion Technologies, Delft) (Sheffield University) (LIF/CNRS, Marseille)
The conference was made possible through the generous financial support of the European Commission through project grant MLCF-CT-2004-013233 for a Marie Curie large Conference. Special thanks go to Galia Angelova (Bulgarian Academy of Sciences) and Nikolai Nikolov (Incoma, Ltd.) and for the efficient local organisation. We believe that this book will be of interest to researchers, lecturers and graduate students interested in Natural Language Processing and, more specifically, to those who work in Computational Linguistics, Corpus Linguistics, Machine Translation. We would like to acknowledge the unstinting help received from our series editor, E.F.K. Koerner, and from Ms Anke de Looper of John Benjamins in Amsterdam. We have built upon our experience from the work on the previous RANLP volumes. Both, E.F.K. Koerner and Anke de Looper have continued to be ever so professional. Nicolas Nicolov produced the typesetting code for the book, utilising the TEX system with the LATEX 2ε package. Umbria Inc. provided computational support. August 2007
Nicolas Nicolov Kalina Bontcheva Galia Angelova Ruslan Mitkov
Linguistic Challenges for Computationalists John Nerbonne University of Groningen Abstract Even now techniques are in common use in computational linguistics which could lead to important advances in pure linguistics, especially language acquisition and the study of language variation, if they were applied with intelligence and persistence. Reliable techniques for assaying similarities and differences among linguistic varieties are useful not only in dialectology and sociolinguistics, but could also be valuable in studies of first and second language learning and in the study of language contact. These techniques would be even more valuable if they indicated relative degrees of similarity, but also the source of deviation (contamination). Given the current tendency in linguistics to wish to confront the data of language use more directly, techniques are needed which can handle large amounts of noisy data and extract reliable measures from them. The current focus in Computational Linguistics on useful applications is a very good thing, but some further attention to the linguistic use of computational techniques would be very rewarding.1
1
Introduction
The goal of this paper is to urge computational linguists to explore issues in other branches of linguistics more broadly. Computational linguistics (CL) has developed an impressive array of analytical techniques, especially in the past decade and a half, techniques which are capable of assaying linguistic structure of various levels from fairly raw textual data. The goal will be to note how these techniques might be applied to illuminate other issues of broad interest in linguistics. My thesis is that CL techniques are capable now of valuable use in many areas of non-computational linguists. Naturally we will keep working at improvement in our computational processing, but the present essay emphasizes applications of CL techniques to areas of linguistics outside the normal CL remit which do not require massive technical improvements in order to be used. 1
We are grateful to the Netherlands Organization for Scientific Research, NWO, for support (project “Determinants of Dialect Variation, 360-70-120, P.I. J. Nerbonne). It was stimulating to discuss the general issue of engineering work feeding back into pure science with Stuart Schieber, who organized a course with Michael Collins of MIT at the Linguistics Institute of the Linguistic Society of America at MIT, Summer 2005 contrasting science and engineering in CL.
2
JOHN NERBONNE
The idea my plea is based on—that there are opportunities for computational contributions to “pure” linguistics—is not absolutely new, of course, as many computational linguists have been involved in issues of pure linguistics as well, including especially grammatical theory. And we will naturally attempt to identify such work as we become more concrete (below). We aim to spark discussion by identifying less discussed areas where computational forays appear promising, and in fact, we will not dwell on grammatical theory at all. It is best to add some caveats. First, the sort of appeal we aim at can only be successful if it is sketched with some concrete detail. If we attempted to argue the usefulness of computational techniques to general linguistic theory very abstractly, virtually everyone would react, “Fine, but how can we contribute more concretely?” But we can only provide more concrete detail on a very limited number of subjects. Of course, we are limited by our knowledge of these subjects as well, but the first caveat is that this little essay cannot be exhaustive, only suggestive. We should be delighted to hear promptly of several further areas of application for computational techniques we omit here. Second, the exhortation to explore issues in other branches of linguistics more broadly takes the form of an examination of selected issues in noncomputational linguistics together with suggestions on how computational techniques might shed added light on them. Since the survey is to be brief, the suggestions about solutions—or perhaps, merely perspectives— of necessity will also be brief. In particular, they will be no more than suggestions, and will make no pretense at demonstrating anything at all. Third, we might be misconstrued as urging you to ignore useful, moneymaking applications in favor of dedicating yourselves to the higher goal of collaborating in the search for scientific truth. But both the history of CL and the usual modern attitude of scientists toward applications convinces me that the application-oriented side of CL is very important and eminently worthwhile. Perhaps you should indeed turn a deaf ear to the seductions of filthy mammon and consecrate yourself to a life of (pure) science, but this is a matter between you and your clergyman (or analyst). You have not gotten this advice here. Fourth, and finally, we might be viewed as advocating different sorts of applications, namely the application of techniques from one linguistic subfield (CL) to another (dialectology, etc.). In this sense modern genetics applies techniques from chemistry to biological molecules to determine the physical basis of inheritance, anthropology applies techniques from nuclear chemistry (carbon dating) to date human artefacts, and astronomy applies techniques from optics (glass) and electromagnetism (radio astronomy) to map the heavens. In all of these case is the primary motivation is scien-
LINGUISTIC CHALLENGES
3
tific curiosity, not utilitarian, and this view is indeed parallel to the step advocated here. Martin Kay received the Lifetime Acheivement Award of the Association for Computational Linguistics in 2005, and his acceptance address to the 2005 meeting ended in cri de coeur to computational linguists to keep studies of language central: “Language [...] remains a valid and respectable object of study, and I earnestly hope that [computational linguists] will continue to pursue it.” (Kay 2005:438). Perhaps the present piece can serve to suggest less likely areas where CL might contribute to progress in the scientific understanding of language. 2
Computational linguistics
Computational Linguistics (CL) is often characterized as having a theoretical and an application-oriented, or engineering side (Joshi 1999, Kay 2002). The theoretical side of CL is concerned with processes involving language and their abstract computational characterization, including processes such as analyzing (parsing), and producing (generating) language, but also storing, compressing, indexing, searching, sorting, learning and accessing language. The computational characterization of these processes involves investigating algorithms for their accuracy and time and space requirements, finding appropriate data structures, and naturally testing these ideas, where possible, against concrete implementations. The application-oriented, or engineering side of the field concerns itself with creating useful computational systems which involve language manipulation in some way, e.g., lexicography tools; speech understanding (in collaboration with speech recognition); machine translation, including translation aids such as translation memories, multilingual alignment, and specialized lexicon construction; speech synthesis, especially intonation; term extraction, information retrieval, document summarization, data (text) mining, and question answering; telephone information systems and natural language interfaces; automatic dictionary and thesaurus access, grammar checking, including spell-checking; document management, authoring (especially in multi-author systems), and conformance to specifications in socalled “controlled language systems”; foreign language aids (such as access to bilingual dictionaries), foreign language tutoring systems, and communication aids (for the handicapped). See Cole et al. (1996) for further discussion of these, and other areas of application for language technology. We have been overly compulsive about listing the engineering activities not only to remind the reader how extensive these are, but also to emphasize that the breadth of these activities would be unthinkable if it were not for a rich “infrastructure” of language technology tools which the field is
4
JOHN NERBONNE
constantly creating. For the most part the techniques we urge you to apply more broadly have been developed in order to build better and more varied applications, as this has been the great motor in the recent dynamics of computational linguistics. But some of the techniques have also been useful in theoretical computational linguistics, and the distinction will play no role here. In fact, perhaps the simplest view is to acknowledge that applications and theory make use of common technology, a sort of technical infrastructure, and to emphasize the opportunities this provides. 3
Dialectology
We shall examine dialectology first because it is an area we have directly worked in, and for which we therefore need to rely less on speculation about the potential benefits of a computational approach. Given the greater amount of direct experience with this work, we may use it to distill some of the characteristics we need to seek in other areas in which computational techniques might be promising. Dialectology studies the patterns of variation in a language and especially its geographic conditioning (Chambers 1998). In London people say [w6t] for ‘water’, with a voiceless [t] and no trace of final [r], in New York most people say [wAR], with a “tapped” [t], and in Boston [wAR]. These dif" ferences are systematic, but not exceptionless, and they appear to involve potentially every level of linguistic structure, pronunciation, morphology, lexicon, syntax, and discourse. Because differences appear to involve exceptions, it is advantageous to process a great deal of material and to apply statistical techniques to the analysis. Fortunately, dialectologists have been assiduous in collecting and archiving a great deal of data, especially involving pronunciation and lexical differences. Once we have agreed that we need to subject a great deal of data to systematic analysis, we have a fortiori accepted the need for automating the analysis, and since it is linguistic material, it would be strange if this did not lead us to computational linguistics. In fact edit distance, wellknown to computational linguists by its wide variety of applications, may be applied fairly directly to the phonetic transcripts of dialect pronunciations (Nerbonne, Heeringa & Kleiweg 1999). The application of edit distance to pronunciation transcripts yields, for each pair of words, at each pair of field work sites, a numerical characterization of the difference. Because pronunciation differences are characterized numerically, we thereby initiate a numerical analysis of data that dialectologists had normally regarded as categorical—with all the advantages which normally accrue to numerical data analysis.
LINGUISTIC CHALLENGES
5
Nerbonne (2003) discusses at greater length the computational issues in analyzing, presenting and evaluating dialectological analyses, including those which go beyond pronunciation. These issues include the use of lemmatizers or stemmers to clean up word-form data for lexical analysis, raising the edit distance from strings to sets of strings in order treat data collections with alternative forms, and the proper treatment of frequency in detection of linguistic proximity. Opportunities for the application of standard CL techniques in computational linguistics abound. Heeringa (2004) summarizes current thinking on measuring dialectal pronunciation differences, including the thorny issue of evaluating the quality of results. Figure 1 illustrates the results of applying these techniques to Bulgarian data (Osenova, Heeringa & Nerbonne, forthcoming).
Fig. 1: In this line map the average Levenshtein distances between 490 Bulgarian dialects are shown for 36 words. Darker lines join varieties with more similar pronunciations; lighter lines indicate more dissimilar ones It is important to report here, as well, that specialists in dialectology—and not only computational linguists—are enthusiastic about the deployment of computational tools. A common remark by dialectologists is that that the new techniques allow a more comprehensive inclusion of all available data, effectively answering earlier complaints that analyses of dialect areas
6
JOHN NERBONNE
and/or dialect continua relied too extensively on the analysts’ choice of material. William Kretzschmar leads the American Linguistic Atlas Projects (LAP), and has collaborated in various analyses and workshops (Nerbonne & Kretzschmar 2003). He has inter alia included a pointer to CL work on the home page of the LAP site he maintains at and he is presently collaborating on a project to publish a second volume of papers focused on computational techniques (Nerbonne & Kretzschmar 2006). Finally, let us note that the computational step may introduce such genuinely novel opportunities that we find ourselves in a position to ask questions which simply lay beyond earlier methodology. Given our numerical perspective on dialect difference, we may e.g., ask, via a regression analysis, how much of the aggregate varietal difference is explained by geography, or whether travel time is a superior characterization of the geography relevant to linguistic variation (Gooskens 2004), or whether larger settlements tend to share linguistic variants more than smaller ones—something one might expect if variation diffused via social contact (Heeringa & Nerbonne 2002). As a further example, we may examine the degree to which the cultural patterns exemplified in dialectal variation correlate with the distribution of genetic variation (Manni, Heeringa & Nerbonne 2006). The introduction of CL techniques enables us to ask more abstract questions in a way we can still link to concrete linguistic analysis. This work also suggests many related paths of exploration. For example, even if a distance measure allows the mapping of the dialectological landscape well, it seems ill-equipped to assay one extreme result of dialect differentiation, i.e., the failure of comprehensibility. The reason for this failure is the fact the comprehensibility is not symmetrical, while linguistic distance by definition is: it may reliably be the case that speakers of one variety understand the speaker of another better than vice versa. For example, Dutch speakers find it easier to understand Afrikaans than vice versa (Gooskens & van Bezooijen 2006). If this is due to language differences, it calls for the development of an asymmetrical measure of the relative difficulty of mapping from one language to another, or something similar.2 The computational work has been successful in dialectology because there were large reservoirs of linguistics data to which analyses could be applied, i.e., dialect atlases, because distinguishing properties resisted simple categorical characterization, and naturally because there were promising computational techniques for getting at the crucial phenomena. As we turn to other areas, we shall ask ourselves whether we are likely to satisfy these desiderata. When even one is missing, the result can be disap2
Nathan Vaillette, Univ. of Massachusetts has explored this problem using relative entropy in unpublished work.
LINGUISTIC CHALLENGES
7
pointing. For example, sociolinguistics has largely succeeded dialectology in attracting scholarly interest. The linguistic issues are not wildly different— different social groups use different language varieties, and these may differ in all the ways in which geographical varieties do (pronunciation, lexicon, etc.). It would be straightforward and interesting to apply the techniques sketch above to linguistic varieties associated with different social groups. But there is no tradition in sociolinguistics like that of the dialect atlas, i.e., collecting speech samples from a large set of sociolects. So the opportunity does not present itself. 4
Diachronic linguistics
Diachronic linguistics investigates how languages change, and, most spectacularly, how a single language many evolve into many related ones. It regularly attracts a good deal of scholarly attention (Gray & Atkinson, 2003; Eska & Ringe, 2004; Dunn, Terrill, Reesink & Levinson, 2005) as computational biologists have applied their techniques for tracking genetic evolution to linguistic data. Although the scholarship is at times forbidding in its expectations about philological expertise, the problem appears to allow neat enough formulations so that one may be optimistic about computational investigations. Essentially, we are given a set of cognate words in several putatively related languages, and we construct hypotheses about the most recent common ancestor—the protolanguage—as well as a simple set of sound changes leading from the protolanguage to the individual descendants. For example, we note that the word for father has an initial /f/ in Germanic (English father ), /p/ in Romance, Greek and Indic (French p`ere, Greek patera, and Hindi pit¯ a ), and no initial consonant in some Celtic languages (Irish athair ). This suggests that we postulate a /p/ in the protolanguage and changes from /p/ to /f/ for Germanic and /p/ to ∅ for the relevant Celtic varieties. But we gain confidence in these postulates only when the same rules are shown to operate on other forms, i.e., when the correspondences recur (as the p/f/∅ definitely does). It is surprising that CL should turn over to the biologists such a well-structured problem in linguistic computation.3 Our community has contributed to this area, especially Brett Kessler, who investigated how to test when sound correspondences exceed chance levels (Kessler 2001), and Grzegorz Kondrak, who modified the edit distance algorithm mentioned above, in order to identify cognates, align them, and on that basis postulate recurrent sound combinations (Kondrak 2002). 3
See also Benedetto et al. (2002) for attempt to reconstruct linguistic history using relative entropy, but especially Goodman (2002) for criticism of Benedetto.
8
JOHN NERBONNE
But these studies deserve follow-ups, tests on new data, and extensions to other problems. Among many remaining problems we note that it would be valuable to detect borrowed words, which should not figure in cognate lists, but which suggest interesting influence; to operationalize the notion of semantic relatedness relevant to cognate recognition; to quantify how regular sound change is; or to investigate the level of morphology, which is regarded as especially probative in historical reconstruction. But we emphasize that there are likely to be interesting opportunities for contributions with respect to detail as well, perhaps in the construction of instruments to examine data more insightfully, to measure hypothesized aspects, or to quantify the empirical base on which historical hypotheses are made. 5
Language acquisition
Studies of children’s acquisition of language are interesting to all sorts of inquiries because language is a defining characteristic of us as humans. They occupy an important position in linguistics due to the linguistic argument that innate, specifically linguistic mechanisms must be postulated to account for acquisition (Pinker 1994:ch. 4). The innate organizing principles of language are postulated to be part of human genetic constitution, and therefore the source of universal properties which all languages share. At the same time psychologists have shown that some acquisition is mediated by sensitivity to statistical trends in data (Saffran, Aslin & Newport 1999). And children naturally need minimally to learn which of all the languages they are genetically predisposed toward is the one in use locally. Finally, CL has explored machine learning techniques extensively over the past decade (Manning & Sch¨ utze 1999). Surely CL is positioned to contribute crucially to this scientific discussion with interesting implemented models of specific phenomena, and in particular with models aimed at broader coverage or so one would think. On the other hand, machine learning techniques do not translate to computational models of acquisition very directly, at least not as normally used by CL, namely to optimize performance on technical tasks that may have no interesting parallel in a child’s acquisition of language, e.g., the task of recognizing named entities, persons, places and organizations. In addition, even idealized simulations of acquisition might wish to impose restrictions on the sort of mechanisms to be used, e.g., that they may apply incrementally, and on the input data, e.g., that it reflect children’s experience. Fortunately, these differences in tasks, mechanisms and input data may be overcome, and CL has not been inactive in examining language acquisition. Brent (1997) is an early collection of articles on computational approaches to language acquisition, including especially Brent’s own work ap-
LINGUISTIC CHALLENGES
9
plying minimal description length to the problem of segmenting the speech stream into words, and using only phonotactic and distributional information (Brent 1999, Brent 1999a). There have been a number of other studies focusing on phonotactics (Nerbonne & Konstantopoulos 2004, Nerbonne & Stoianov 2004), the acquisition of morphophonemic rules (Gildea & Jurafsky 1996, Albright & Hayes 2003), morphology (Goldsmith 2001), and syntax (Niyogi & Berwick 1996). Albright and Hayes’s work is especially worth recommending to a CL audience as it is clear and explicit about linguistic concerns in modeling acquisition computationally. Most relevant to the sort of CL contribution I have in mind is the series of workshops organized by William Gregory Sakas of CUNY, PsychoComputational Models of Human Language Acquisition. The first took place in 2004 in Geneva in coordination with COLING and the second in 2005 in Ann Arbor in coordination with the ACL’s special interest group on natural language learning It is clear from the proceedings of these workshops that new syntheses of linguistic, psychological and computational perspectives enjoy a good deal of interest (Yang 2004). It is also clear that there is an enormous interest in further questions about segmentation, alignment, constituency, local and long-distance relations, modification, and ill-formed input in addition to the usual questions about the generality of solutions wit respect to various language types. Finally, it is worth emphasizing here more than elsewhere that contributions need not take the form of simulations of human learning (even if this is the case for most of the studies cited). There is great potential interest in characterizing easy vs. difficult material, in what happens when second and third languages are learned (contamination), and in how languages are lost. In addition to simulation, we should also be thinking of how to operationalize measures of language proficiency that could use speech as directly as possible. At the moment, extremely crude measures such as mean length of utterance (MLU) and type/token ratio enjoy great popularity, but one suspects that this is due more to their ease of computation than to their reflection of linguistic sophistication. Ideally we should like to automate our detection of the mastery of various linguistic structures, rules and exceptions. That is clearly a long way off in its full generality, but perhaps realizable in some instances with standard techniques. 6
Language contact
Language contact study is an active branch of linguistics focused on recognizing and analyzing the ways in which languages borrow from one another (Thomason & Kaufmann 1988, van Coetsem 1988). It is growing in popularity, perhaps due to increases in mobility and the realization that
10
JOHN NERBONNE
multilingual speakers often, albeit unconsciously, impose the structures of one language on another. Mufwene (2001) urges us to view extreme contact effects such as koin´eization, creolization and pidginization as various degrees to which language mixtures may develop (instead of as the results of very different processes, as earlier scholarship had held). Language contact study is, moreover, linked to second-language acquisition in an obvious way: if second-language speakers habitually impose elements of their native language onto another, then those element are good candidates for long-term borrowing whenever these languages are in contact. It might seem as if we could use the same tools for the study of contact effects that we developed for dialectology. After all, if one variety of a language adopts elements of another, it should become more similar. Indeed given the sort of data in dialect atlases, one can perform these analyses and determine the convergence of some varieties toward a putative source of contamination, at least the convergence with respect to other varieties (Heeringa, Nerbonne, Niebaum & Nieuweboer 2000, Gooskens & Heeringa 2004). Furthermore, one could examine the role of geography in this convergence. But language contact data collections are not usually designed as dialect atlases, with a number of distinct collection sites, and a controlled set of linguistic variables to be assayed. Recently, we obtained data of a rather different sort, and set ourselves the task of developing computational tools for its analysis.4 Watson collected recordings of Finnish emigrants to Australia in the mid 1990’s (Watson 1996), and this group could be divided into adult emigrants and child emigrants, using puberty (16 years old) as the dividing line. The challenge was the development of a technique to determine whether there were significant changes in the syntax of the two groups. Following an obvious tack from CL, we settled on using n-grams of partof-speech tags (POS tags) assigned by the TnT tagger (Brants 2000) as a probe to determine syntactic similarity. In order not to be swamped by fine distinctions we used trigrams of a small tag set (50 tags). Up to this point we were rediscovering an idea others had introduced (Aarts & Granger 1998). To compare one corpus with another, we measured the difference in the two vectors of trigram frequencies using cosine (inter alia). To determine whether the difference is statistically significant, we applied permutationbased statistics, roughly resampling the union of the two data sets (using some complicated normalizations) and checking the degree of difference. A difference is significant at the level p < p′ iff it is among the most extreme p′ fraction of the resampled data. 4
What follows is an informal synopsis of work in progress being conducted with Wybo Wiersma of Groningen and Lisa Lena Opas-H¨ anninen, Timo Lauttamus and Pekka Hirvonen of Oulu University.
LINGUISTIC CHALLENGES
11
Because the technique is still under development, we cannot yet report much more. The differences are indeed statistically significant, which, in itself, is not surprising. The corpora are quite raw, however, so that the differences we are finding to-date are dominated by hesitation noises and errors in tagging. The promise is in the technique. If we have succeeded in developing an automated measure of syntactic difference, we have opportunities for application to a host of further questions about syntactic differences, e.g., about where these differences are detectable, and where not; about the time course of contamination effects (do second-language learners keep improving, or is there a ceiling effect?); and about the role of the source language in the degree of contamination. Some crucial computational questions would remain, however, concerning detecting the source of contamination. 7
Other areas
As noted in the introduction, this brief survey has tried to develop a few ideas in order to convince you that there are promising lines of inquiry for computationalists who would seek to contribute to a broader range of linguistic subfields. We suspect that there are many other areas, as well. We have deliberately omitted grammatical theory from the list of potential near-term adopters of computational techniques. There are two reasons for eschewing a sub-focus on grammar here, the first being the fact that the potential relevance of computational work to grammatical theory has been recognized for a long time, as grammar has been cited since the earliest days of CL as a likely beneficiary of closer engagement (Kay 2002). But second, even as computational grammar studies uncover new means of contributing to the study of pure grammar (van Noord 2004), it seems to be a minority of (non-computational) grammarians who recognize the value of computational work. Many researchers have explored this avenue, but the situation has stabilized to one in which computational work is pursued vigorously by small specialized groups (Head-Driven Phrase Structure Grammar and Lexical-Functional Grammar), and largely ignored by most non-mainstream grammarians. We deplore this situation as do others (Pollard 1993), but it unfortunately appears to be quite stable. In addition to the areas discussed above, it is easily imaginable that CL techniques could play an interesting role in a number of other linguistic subareas. As databases of linguistic typology become more detailed and more comprehensive, they should become attractive targets for data-mining techniques Psycholinguistic studies of processing are promising because they provide a good deal of empirical data. We shall be content with a single example. Moscoso del Prado Mart´ın (2003) reviews a large number of studies relating the difficulty of processing complex word forms, i.e., those involv-
12
JOHN NERBONNE
ing inflectional and/or derivational structure to the “family size” of a word form, i.e. how many other word forms are related to it. He is able to show that a simple characterization of family size and frequency due to information theory correlates highly with processing difficulty. 8
Conclusions
We have urged computational linguists to consider how much they might contribute to curiosity-driven research into language, i.e., linguistic theory, focusing on examples in dialectology, diachronic linguistics, language acquisition and language contact. We have suggested that there are many avenues to pursue for those with a broader interest in language, and also that the tools and training one receives in developing language technology will be of direct use. We have not suggested that contributions in pure science are any easier or harder to make, and the experience has been general that the dynamics involved in pursuing non-applied goals are every bit as demanding, and every bit at provocative: a successful effort invariably suggests new questions and new avenues to explore. We have been careful to avoid deprecating application-research and, at the risk of repetition, restate that the development of useful applications is a most valuable aspect of current CL. We encourage colleagues to think of both channels of activity rather than to force a choice of one over the other. If we are right that most of the interesting techniques for exploring issues in non-computational linguistics have arisen through the development of techniques for engineering activities, then we may have another case where applied science furthers the progress of pure science (Burke 1985). In making this remark, we are reneging on the promise in Section 2 not to concern ourselves with whether a particular technique originated in theoretical vs. applied CL, but given the preponderance of applied work in CL, it would be surprising if it were not true in many instance that techniques from engineering were being conscripted for work in theory. The use of a stemmer to extract lexical differences from lists of word forms in dialectology (Nerbonne & Kleiweg 2003) is an example of the sort of contribution where a technique developed only for application purposes could be put to a purely scientific use, that of detecting lexical overlap across a dialect continuum. The Porter stemmer which was used for this purpose is not to be confused with a genuine lemmatizer, which is interesting both linguistically and practically. But it usually reduces word forms to the same stem when they in fact are elements of the same inflectional paradigm. It was developed for use in information retrieval (Porter 1980), not for the
LINGUISTIC CHALLENGES
13
purpose of exploring linguistic structure or its processing, but its use in dialectology has no ambitions toward practical application. This would appear to be a genuine case of an engineering technique serving a purpose in curiosity-driven research. To the extent CL is involved in other pure science (beyond CL proper), this sort of cross-fertilization must be standard. Only time will tell whether it will remain true of future computational forays into pure linguistics. REFERENCES Aarts, Jan & Sylviane Granger. 1998. “Tag Sequences in Learner Corpora: A Key to Interlanguage Grammar and Discourse”. Learner English on Computer ed. by Sylviane Granger, 132-141. London: Longman. Albright, Adam & Bruce Hayes. 2003. “Rules vs. Analogy in English Past Tenses: A Computational/Experimental Study”. Cognition 90:119-161. Benedetto, Dario, Emanuele Caglioti & Vittorio Loreto. 2002. “Language Trees and Zipping”. Physical Review Letters 88:4.048702. Brants, Thorsten. 2000. “TnT — A Statistical Part of Speech Tagger”. 6th Applied Natural Language Processing Conference, 224-231. Seattle, Washington. Brent, Michael, ed. 1997. Computational Approaches to Language Acquisition. Cambridge, Mass.: MIT Press. Brent, Michael. 1999. “An Efficient, Probabilistically Sound Algorithm for Segmentation and Word Discovery”. Machine Learning Journal 34:71-106. Brent, Michael. 1999. “Speech Segmentation and Word Discovery: A Computational Perspective”. Trends in Cognitive Science 3:294-301. Burke, James. 1985. The Day the Universe Changed. Boston, Mass.: Little, Brown & Co. Chambers, J.K. & Peter Trudgill. 1998. Dialectology. Cambridge, U.K.: Cambridge University Press. Cole, Ronald A., Joseph Mariani, Hans Uszkoreit, Annie Zaenen & Victor Zue. 1996. Survey of the State of the Art in Human Language Technology. National Science Foundation (NSF) and European Commission (EC), www.cse.ogi.edu/CSLU/HLTsurvey/ Dunn, Michael, Angela Terrill, Geert Reesink, Robert A. Foley & Stephen C. Levinson. 2005. “Structural Phylogenetics and the Reconstruction of Ancient Language History”. Science 309:5743.2072-2075. Eska, Joseph F. & Don Ringe. 2004. “Recent Work in Computational Linguistic Phylogeny”. Language 80:3.569-582. Gildea, Daniel & Daniel Jurafsky. 1996. “Learning Bias and Phonological Rule Induction”. Computational Linguistics 22:4.497-530.
14
JOHN NERBONNE
Goldsmith, John. 2001. “Unsupervised Learning of the Morphology of a Natural Language”. Computational Linguistics 27:2.153-198. Goodman, Joshua. 2002. “Extended comment on ‘Language trees and zipping’ ”. Condensed Matter Archive, February 21, 2002. arXiv:cond-mat/0202383. Gooskens, Charlotte & Wilbert Heeringa. 2004. “The Position of Frisian in the Germanic Language Area”. On the Boundaries of Phonology and Phonetics, ed. by Dicky Gilbers, Maartje Schreuder & Nienke Knevel, 61-88. Groningen: CLCG. Gooskens, Charlotte & Ren´ee van Bezooijen. 2006. “Mutual Comprehensibility of Written Afrikaans and Dutch: Symmetrical or Asymmetrical?” Literary and Linguistic Computing 21:4:543-557. Gooskens, Charlotte. 2004. “Norwegian Dialect Distances Geographically Explained”. Language Variation in Europe. Papers from the Second International Conference on Language Variation in Europe (ICLAVE 2 ), ed. by B.-L. Gunnarson et al. 195-206. Uppsala, Sweden: Uppsala University. Gray, Russell D. & Quentin D. Atkinson. 2003. “Language-Tree Divergence Times Support the Anatolian Theory of Indo-European Origin”. Nature 426:435-439. Heeringa, Wilbert & John Nerbonne. 2002. “Dialect Areas and Dialect Continua”. Language Variation and Change 13:375-400. Heeringa, Wilbert. 2004. Measuring Dialect Pronunciation Differences Using Levenshtein Distance. Ph.D. dissertation, Rijksuniversiteit Groningen, Groningen. Heeringa, Wilbert, John Nerbonne, Hermann Niebaum & Rogier Nieuweboer. 2000. “Measuring Dutch-German Contact in and around Bentheim”. Languages in Contact, ed. by Dicky Gilbers, John Nerbonne & Jos Schaeken, 145-156. Amsterdam-Atlanta: Rodopi. Joshi, Aravind K. 1999. “Computational Linguistics”. The MIT Encyclopedia of the Cognitive Sciences, ed. by Robert A. Wilson & Frank C. Keil, 162-164. Cambridge, Mass.: MIT Press. Kay, Martin. 2002. “Introduction”. Handbook of Computational Linguistics, ed. by R.Mitkov, xvii–xx. Oxford, U.K.: Oxford University Press. Kay, Martin. 2005. “A Life of Language”. Computational Linguistics 31:4.425438. Kessler, Brett. 2001. The Significance of Word Lists. Stanford, Calif.: CSLI Press. Kondrak, Grzegorz. 2002. Algorithms for Language Reconstruction. Ph.D. dissertation, Univ. of Toronto, Canada. Manning, Chris & Hinrich Sch¨ utze. 1999. Foundations of Statistical Natural Language Processing. Cambridge, Mass.: MIT Press.
LINGUISTIC CHALLENGES
15
Moscoso del Prado Mart´ın, Fermin. 2003. Paradigmatic Structures in Morphological Processing: Computational and Cross-Linguistic Experiemental Studies. Ph.D. dissertation, Radboud Univ. Nijmegen, The Netherlands. Mufwene, Salikoko. 2001. The Ecology of Language Evolution. Cambridge, U.K.: Cambridge University Press. Nerbonne, John & Peter Kleiweg. “Lexical variation in LAMSAS”. Computers and the Humanities (= Special Issue on Computational Methods in Dialectometry) ed. by John Nerbonne & William Kretzschmar, Jr., 37:3.339-357. Nerbonne, John & Stasinos Konstantopoulos. 2004. “Phonotactics in Inductive Logic Programming”. Intelligent Information Processing and Web Mining (IIPWM’04 ) ed. by Mieczyslaw A. Klopotek, Slawomir T. Wierzchon & Krzysztof Trojanowski, 493-502, Springer: Berlin. Nerbonne, John & William Kretzschmar, eds. 2003. Computational Methods in Dialectometry (= Special Issue of Computers and the Humanities, 37:3). Nerbonne, John & William Kretzschmar, eds. 2006. Progress in Dialectometry (= Special Issue of Literary and Linguistic Computing, 21). Nerbonne, John & Ivilin Stoianov. 2004. “Learning Phonotactics with Simple Processors”. On the Boundaries of Phonology and Phonetics (CLCG) ed. by Dicky Gilbers, Maartje Schreuder & Nienke Knevel, 89-121. Groningen. Nerbonne, John. 2003. “Linguistic Variation and Computation”. 10th Meeting of the European Chapter of the Association for Computational Linguistics, vol. 10, 3-10. Nerbonne, John, Wilbert Heeringa & Peter Kleiweg. 1999. “Edit Distance and Dialect Proximity”. Time Warps, String Edits and Macromolecules: The Theory and Practice of Sequence Comparison ed. by David Sankoff & Joseph Kruskal, v-xv. Stanford, Calif.: CSLI. Niyogi, Partha & Robert C. Berwick. 1996. “A Language Learning Model for Finite Parmeter Spaces”. Cognition 61:161-193. Osenova, Petya, Wilbert Heeringa & John Nerbonne. Forthcoming. “A Quantitative Analysis of Bulgarian Dialect Pronunciation”. To appear in Zeitschrift f¨ ur Slavische Philologie. Pinker, Steven. 1994. The Language Instinct. New York: W. Morrow & Co. Pollard, Carl. 1993. “On Formal Grammars and Empirical Linguistics”. ESCOL’93: 10th Eastern States Conference on Linguistics ed. by Andreas Kathol & M. Bernstein. Columbus, Ohio: Ohio State University. Porter, Martin F. 1980. “An Algorithm for Suffix Stripping”. Program 14:3.130137. Saffran, Jenny R., E.K. Johnson, Richard N. Aslin & Elissa L. Newport. “Statistical Learning of Tonal Sequences by Human Infants and Adults”. Cognition 70:27-52. Thomason, Sarah & Terrence Kaufmann. 1988. Language Contact, Creolization, and Genetic Linguistics. Berkeley, Calif.: University of California Press.
16
JOHN NERBONNE
van Coetsem, Frans. 1988. Loan Phonology and the Two Transfer Types in Language Contact. Dordrecht: Foris Publications. van Noord, Gertjan. 2004. “Error Mining for Wide-Coverage Grammar Engineering”. 42nd Meeting of the Association for Computational Linguistics, 446-453. Barcelona, Spain. Watson, Greg. 1996. “The Finnish-Australian English Corpus”. ICAME Journal: Computers in English Linguistics 20:41-70. Yang, Charles. 2004. “Universal Grammar, Statistics or Both?”. Trends in Cognitive Science 8:10.451-456.
NLP: An Information Extraction Perspective Ralph Grishman New York University Abstract This chapter examines some current issues in natural language processing from the vantage point of information extraction (ie), and so gives some perspective on what is needed to make ie more successful. By ie we mean the identification of important types of relations and events in unstructured text. Ie provides a nice reference point because it can benefit from a wide range of technologies, including techniques for analyzing sentence structure, for identifying syntactic and semantic paraphrases, and for discovering the information structure of domains.
1
The challenges of information extraction
Information Extraction (ie) provides an interesting perspective from which to view many of the methods and challenges of natural language processing (nlp). On the one hand, some extraction tasks can be successfully addressed using simple nlp tools, such as regular expressions and basic probabilistic sequential models. On the other hand, ie in its full generality involves understanding the structure of the information conveyed by a collection of documents, and then distilling this information from the documents. This will require a deep analysis of the language using a rich palette of nlp tools, including some still to be developed. We will use this survey to present a small portion of this palette, including tools developed by our group and others. Ie is a domain-specific task; the important types of objects and events for one domain (e.g., people, companies, products, money and securities in the financial news) can be quite different from those for another domain (e.g., genes and proteins in genomics). Developing ie for a new domain can be broken down into three tasks: 1. determining what the important types of facts are for the domain; 2. for each type of fact, determining the various ways in which it is expressed linguistically; 3. identifying instances of these expressions in text. While there is a fuzzy boundary between these tasks, this division will provide a basis for organizing this overview. Steps 1 and 2 are done in advance, and often by hand, while step 3 is part of every ie system, so we will start with this last step and then work our way backwards.
18 2
RALPH GRISHMAN
Identifying instances of a linguistic expression
In simplest terms, ie can be realized as a pattern matching process. We are looking for linguistic expressions which convey a particular relationship or fact involving one or more arguments and modifiers. We find such expressions by creating patterns with constants and constrained variables (positions in the pattern which match the arguments). If the constants and variables match individual words in sequence, we will end up with patterns which are too specific; for example, extracting reports of deaths by looking for “Mr.” followed by a capitalized token followed by “died”. To achieve any degree of generality (and hence reasonable coverage), matching must occur at a structural level; for example, that a noun phrase referring to a person appears as the subject of some inflected form of “die”. So the crucial problem here, at the heart of many nlp applications, is the accurate identification of the structure of sentences and entire discourses. 2.1
Stages of analysis and joint inference
This structure exists on many levels: the structure of names; the grammatical structure of sentences; and coreference structure across a discourse (and even across multiple discourses). Each of these is important to ie – to figuring out the participants in an event. And each of these has been studied separately and quite intensively over the past decade. Annotated corpora have been prepared for each of these levels of structure, and a wide range of models and machine learning methods have been applied to construct analyzers (particularly for name and grammatical structure). Except for coreference analysis, the result of these efforts have in general been quite satisfactory levels of performance – on the order of 90% accuracy for names and for grammatical constituents.1 In a typical system, these analyzers are applied sequentially to preprocess a text for extraction. Unfortunately, the analysis errors of the individual stages not only add up, they compound: an error in an early stage will often lead to further errors as analysis progresses. The net result is that overall analysis performance, and hence extraction performance, is still not very good. For the muc [Message Understanding Conference] evaluations in the 1990’s, recall on the event task rarely broke the 60% ‘ceiling’ (Hirschman 1998), and it’s not clear if we are doing much better today. One limitation is the reliance on relatively local features in the early stages of analysis. Most ne (named entity) analyzers, for example, are based on simple models that look only one or two tokens ahead and behind. 1
When tested on texts similar in genre and time period to those on which the analyzer was trained.
NLP: AN INFORMATION EXTRACTION PERSPECTIVE
19
This fails to capture such basic tendencies as the increased likelihood of a name that was mentioned once in a document being mentioned again. To account for this, some systems employ a name cache or, more elaborately, features based on the context of other instances of the same string (Chieu & Ng 2002) — in effect, trying to do simple coreference within the name tagger. However, preferences which depend on more complex syntactic structures — for instance, that names appearing as the subjects of selected verbs are likely to be person names — remain difficult to capture because the structures are simply not available at this stage of analysis. A more general approach harnesses the richer representations of the later stages to aid the performance of earlier ones. Such an approach may generate multiple hypotheses in one analysis stage and then rescore them using information from subsequent stages. In general, we rely on the idea that the discourse is coherent — that in a properly-analyzed discourse, there will be many connections between entities. For example, we expect that a correct name tagging will license more coreference relations as well as more semantic relations (such as ‘X is located in Y’, ‘X works for Y’, etc.). By evaluating the result of these later stages of analysis for each hypothesized set of name tags, for example, a system can use these later stages to improve name tagging. Ji and Grishman (2005) generated N-best NE hypotheses and rescored them after coreference and semantic relation identification; they obtained a significant improvement in Chinese NE performance. Roth and Yi (2004) built separate models for name classification and for semantic relation identification, and then used a linear programming model to capture the interactions between names and relations and to maximize the total probability (the product of name and relation probabilities).2 They obtained significant improvements in both name classification and relation detection. We can expect this ‘global optimization’ approach will be extended in the future to integrate a wider range of analysis levels and provide further performance improvements, possibly even incorporating cross-document information. 2.2
Stages of analysis: Alternative evidence
While such approaches should reduce analysis error, we need to consider how to deal with the error that remains. ‘Deeper’ representations can in principle do a better job in supporting ie (by identifying the common features of variant syntactic forms), but they will generally involve greater error. This is a dilemma which has faced ie developers for a decade. It has led many groups to rely on partial parsing which, while less informative, 2
They assumed the extent of the names in the text was given.
20
RALPH GRISHMAN
is more accurate than full parsing.3 However, machine learning methods which can handle large numbers of features have allowed recent systems to integrate information from multiple levels of representation in predicting the existence of ie relations and events (Kambhatla 2004). Zhao et al. (2004), Zhao & Grishman (2005) have shown how using kernel methods to combine information from n-grams, chunks, and grammatical relations can improve extraction performance over using a single level of representation. In some cases where there is an error in the deep analysis, a correct extraction decision will still be made based on the shallow features. 3
Finding linguistic expressions of an event or relation
The methods just described will give us a better chance of identifying instances of a particular linguistic expression, but we are still faced with the problem of finding the myriad linguistic expressions of an event — all (or most of) the paraphrases of a given expression of an event. A direct approach is to annotate all the examples of an event in a large corpus, and then collect and distill them either by hand or using some linguistic representation and machine learning method. However, good coverage may require a really large corpus, which can be quite expensive. Could we do better? 3.1
Syntactic paraphrase
We need first of all to differentiate syntactic and semantic paraphrase. Syntactic paraphrases are applicable over broad (grammatical) classes of words — relations between active, passive, and relative clauses, for example, as well as complement alternations and nominalizations. Many of these can be addressed by using a deeper syntactic representation that captures the commonality among such different expressions. In particular, a predicateargument representation, such as is being encoded for English in PropBank (Kingsbury & Palmer 2002) and NomBank (Meyers et al. 2004), would collapse many of these syntactic paraphrases. 3.2
Semantic paraphrase
What remains are the much more varied and numerous semantic paraphrases. There are dozens of ways of saying that a company hired someone, or that two people met. Lexical-semantic resources (such as WordNet) provide some assistance (Stevenson & Greenwood 2005), but they are largely 3
Many groups made this choice prior to the recent improvements in treebank-trained parsers, but the choice is still not clear-cut.
NLP: AN INFORMATION EXTRACTION PERSPECTIVE
21
limited to single-word paraphrases and so cover only a portion of the myriad expressions required for an ie task. To complement these manuallyprepared resources, efforts have been underway for the past few years to learn paraphrase relations from corpora. The basic idea is to identify pairs of expressions A R B and A S B which involve the same arguments (A, B) and most likely convey the same information; then R and S stand a good chance of being paraphrases. One source of such pairs are two translations of the same text (Barzilay & McKeown 2001). If we can sentence-align the texts, the corresponding sentences are likely to carry the same information. Another source are comparable news articles — articles from the same day about the same news topic (Shinyama et al. 2002). The opening sentences of such articles, in particular, are likely to contain phrases which convey the same information. The likelihood is even greater if we focus on phrases which are both relevant to the same topic (see the next section). Finally, frequency can build confidence: if we have several pairs of individuals A1 , B1 ; A2 , B2 ; ... which appear in both context R (A1 R B1 ; A2 R B2 ) and context S (A1 S B1 ; A2 S B2 ), then R and S stand a good chance of being paraphrases. This general approach has been used to find paraphrases for individual relations (Brin 1998, Agichtein & Gravano 2000, Lin & Pantel 2001) and to collect the primary paraphrase relations of a domain (Sekine 2005). 4
Discovering what’s important
Finally, there may be situations where we don’t have specific event or relation types in mind – where we simply want to identify and extract the ‘important’ events and relations for a particular domain or topic. Riloff (1996) introduced the basic idea of dividing a document collection into relevant (on topic) and irrelevant (off topic) documents, and selecting constructs which occur much more frequently in the relevant documents. Her approach relied on a relevance-tagged corpus. This idea was extended by (Yangarber et al. 2000) to bootstrap the discovery process from a small ‘seed’ set of patterns which define a topic. Sudo generalized the form of the discovered patterns (Sudo et al. 2003) and created a system which started from a narrative description of a topic and used this description to retrieve relevant documents (Sudo et al. 2001). These methods have been used to collect the linguistic expressions for a specific set of event types, and they are effective when these events form a coherent ‘topic’ – when they co-occur in documents. Because these methods are based on the distribution of constructs in documents, they may gather together related but non-synonymous forms like ‘hire’, ‘fire’, and ‘resign’, or ‘buy’ and ‘sell’. However, by coupling these methods with paraphrase
22
RALPH GRISHMAN
discovery, it should be possible to both gather relevant expressions and group those representing the same event types (Shinyama et al. 2002). By doing so, it should be possible in the future to automatically identify the key semantic structures of a topic or domain, and to create an ie system to extract this information (Sekine 2006). Acknowledgements. This research was supported by the Defense Advanced Research Projects Agency under Grant N66001-04-1-8920 and Contract HR001106-C-0023, and by the National Science Foundation under Grant 03-25657. This paper does not necessarily reflect the position or the policy of the U.S. Government. REFERENCES Agichtein, Eugene & Luis Gravano. 2000. “Snowball: Extracting Relations from Large Plain-Text Collections”. 5th ACM Int. Conf. on Digital Libraries, 85-94. San Antonio, Texas. Barzilay, Regina & Kathleen R. McKeown. 2001. “Extracting Paraphrases from a Parallel Corpus”. ACL/EACL 2001, 50-57. Toulouse, France. Brin, Sergei. 1998. “Extracting Patterns and Relations from the World Wide Web”. World Wide Web and Databases International Workshop (= LNCS 1590), 172-183. Heidelberg: Springer. Chieu, Hai Leong & Hwee Tou Ng. 2002. “Named Entity Recognition: A Maximum Entropy Approach Using Global Information”. 19th Int. Conf. on Computational Linguistics (COLING-2002 ), 190-196. Taipei. Hirschman, Lynette. 1998. “Language Understanding Evaluations: Lessons Learned from MUC and ATIS”. 1st Int. Conf. on Language Resources and Evaluation (LREC-1998 ), 117-122. Granada, Spain. Ji, Heng & Ralph Grishman. 2005. “Improving Name Tagging by Reference Resolution and Relation Detection”. 43rd Annual Meeting of the Association for Computational Linguistics (ACL’2005 ), 411-418. Ann Arbor, Michigan. Kambhatla, Nanda. 2004. “Combining Lexical, Syntactic and Semantic Features with Maximum Entropy Models for Extracting Relations”. Companion Volume to the Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL’2004 ), 178-181. Barcelona, Spain. Kingsbury, Paul & Martha Palmer. 2002. “From TreeBank to PropBank”. 3rd Int. Conf. on Language Resources (LREC-2002 ), 1989-1993. Las Palmas, Spain. Lin, Dekang & Patrick Pantel. 2001. “Discovery of Inference Rules for Question Answering”. Natural Language Engineering 7:4.343-360.
NLP: AN INFORMATION EXTRACTION PERSPECTIVE
23
Meyers, Adam, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young & Ralph Grishman. 2004. “Annotating Noun Argument Structure for Nombank”. 4th Int. Conf. on Language Resources and Evaluation (LREC-2004 ), 803-806. Lisbon, Portugal. Riloff, Ellen. 1996. “Automatically Generating Extraction Patterns from Untagged Text”. 13th National Conference on Artificial Intelligence, 1044-1049. Portland, Oregon. Roth, Dan & Wen-tau Yih. 2004. “A Linear Programming Formulation for Global Inference in Natural Language Tasks”. Conf. on Computational Natural Language Learning (CoNLL’2004 ), 1-8. Boston, Mass. Sekine, Satoshi. 2005. “Automatic Paraphrase Discovery Based on Context and Keywords between NE Pairs”. 3rd Int. Workshop on Paraphrasing, Jeju Island, South Korea. Sekine, Satoshi. 2006. “On-demand Information Extraction”. 21st Int. Conf. on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics (COLING-ACL’2006 ), Sydney, Australia. Shinyama, Yusuke, Satoshi Sekine, Kiyoshi Sudo & Ralph Grishman. 2002. “Automatic Paraphrase Acquisition from News Articles”. Human Language Technology Conference (HLT-2002 ), San Diego, Calif. Stevenson, Mark & Mark Greenwood. 2005. “A Semantic Approach to IE Pattern Induction”. 43rd Annual Meeting of Association for Computational Linguistics, 379-386. Ann Arbor, Michigan. Sudo, Kiyoshi, Satoshi Sekine & Ralph Grishman. 2001. “Automatic Pattern Acquisition for Japanese Information Extraction”. Human Language Technology Conference (HLT-2001 ), 51-58. San Diego, Calif. Sudo, Kiyoshi , Satoshi Sekine & Ralph Grishman. 2003. “An Improved Extraction Pattern Representation Model for Automatic IE Pattern Acquisition”. 41st Annual Meeting of the Association for Computational Linguistics (ACL’2003 ), 224-231. Sapporo, Japan. Yangarber, Roman, Ralph Grishman, Pasi Tapanainen & Silja Huttunen. 2000. “Automatic Acquisition of Domain Knowledge for Information Extraction”. 18th Int. Conf. on Computational Linguistics (COLING-2000 ), 940-946. Saarbr¨ ucken, Germany. Zhao, Shubin, Adam Meyers & Ralph Grishman. 2004. “Discriminative Slot Detection Using Kernel Methods”. 20th Int. Conf. on Computational Linguistics (COLING-2004 ), 757-763. Geneva. Zhao, Shubin & Ralph Grishman. 2005. “Extracting Relations with Integrated Information Using Kernel Methods”. 43rd Annual Meeting of the Association for Computational Linguistics, 419-428. Ann Arbor, Michigan.
Semantic Indexing using Minimum Redundancy Cut in Ontologies Florian Seydoux & Jean-C´ edric Chappelier Ecole Polytechnique F´ed´erale de Lausanne (EPFL) Abstract This paper presents a new method that improves semantic indexing while reducing the number of indexing terms. Indexing terms are determined using a minimum redundancy cut in a hierarchy of conceptual hypernyms provided by an ontology (e.g., WordNet, EDR). The results of some information retrieval experiments carried out on several standard document collections using the WordNet and EDR ontologies are presented, illustrating the benefit of the method.
1
Introduction
Three fields are mainly reported in the literature about the use of semantic knowledge for Information Retrieval: query expansion (Voorhees 1994, Moldovan & Mihalcea 2000), Word Sense Disambiguation (Ide & V´eronis 1998, Wilks & Stevenson 1998, Besan¸con et al. 2001) and semantic indexing. This contribution relates to the latest, the main idea of which is to use word senses rather than, or in addition to, the words (usually lemmas or stems) for indexing document, in order to improve both recall (by handling synonymy) and precision (by handling homonymy and polysemy). However, the experiments reported in the litterature lead to contradicting results: some claim that it degrades the performance (Salton 1968, Harman 1988, Voorhees 1993, Voorhees 1998); whereas for others the gain seems significant (Richardson & Smeaton 1995, Smeaton & Quigley 1996, Gonzalo et al. 1998a & 1998b, Mihalcea & Moldovan 2000). Although it seems desirable for IR systems to take a maximum of semantic information into account, the resulting expansion of the data processed may not develop its full potential. Indeed, the growth of the number of index terms not only increases the processing time but could also reduce the precision, as discriminating documents by using a very large number of index terms is a hard task. This problem is not new, and various techniques aiming at reducing the size of the indexing set already exist: filtering by stoplist, part of speech tags, frequencies, or through statistical techniques as in LSI (Deerwester et al. 1990) or PLSI (Hofmann 1999). However, most of these techniques are not adapted to the case where an explicit semantic information is available,
26
F. SEYDOUX & J.-C. CHAPPELIER
(a) (b) (c)
r
wi,j,k
c1 ... c 2
c3
c3 MRC
c2 c1
wi wj
wk
Fig. 1: Several indexing scheme: (a) usual indexing with words, stems or lemmas; (b) synset (or hypernyms synsets) indexing: each indexing term is replaced by its (hypernyms) synset; this, in principle, reduces the size of the indexing set since all the indexing terms that are shared by the same hypernym are regrouped in one single indexing feature; (c) Minimum Redundancy Cut (MRC) indexing: each indexing term is replaced by its dominating concept chosen with MRC. This furthermore reduces the size of the indexing set since all the indexing terms that are subsumed by the same concept in the MRC are regrouped in one single indexing feature. for example in the form of a thesaurus or an ontology (i.e., with some underlying formal – not statistical – structure). The focus of the work presented here is to use external1 structured semantic resources such as an ontology in order to limit the semantic indexing set. This work, which is a continuation of (Seydoux & Chappelier 2005), relates, but from a different point of view, with experiments described in (Gonzalo et al. 1998b, Whaley 1999, Mihalcea & Moldovan 2000), which uses the synsets (or hypernyms synsets (Mihalcea & Moldovan 2000)) of WordNet as indexing terms. We follow the onto-matching technique described in (Kiryakov & Simov 1999), but here selecting the indexing terms using the “Minimum Redundancy Cut” (MRC) (MRC, see Figure 1), an information theory based criterion applied to the inclusive ’is-a’ relation (hypernyms) provided by the WordNet (Fellbaum 1998, Miller 1995) and EDR taxonomies (Miyoshi et al. 1996). 2
Minimum redundancy cut in an ontology
The choice of the appropriate hypernym (a “concept” in the ontology) to be used for representing a word is not easy: be it too general, the performance of the system will degrade (lack of precision); be it too specific, the indexing set will not reduce enough, preserving some distinction between words with 1
By “external ”, we mean “not directly related to the document collection itself ”.
SEMANTIC INDEXING USING MRC
27
close senses (lack of recall). To select the appropriate level of conceptual indexing, we consider cuts in the ontology. Let N = {ni } represent the set of nodes and W the set of words in the ontology. A cut Γ in the directed acyclic graph (DAG) representing the ontology, is defined as a minimal subset2 of N which covers W. Each node in the cut then represents every leaf node it dominates. A probabilized cut M = (Γ, P ) is a couple consisting of a cut Γ and a probability distribution P on Γ. The main problem is to find a computable strategy to select an optimal cut. We propose to use an information theory based criterion, that selects a cut for which the redundancy is minimal. From now on, let’s consider the probabilized cut M = (Γ, Pf ), where Pf is defined using the relative frequencies of the words in the collection: Pf (ni ) = f (ni )/|D|, f (ni ) being the number of occurrences of the node ni in a document collection D of size |D|. To compute f (ni ), we consider that an occurrence of ni happens when any of the hyponym words of ni occurs. The redundancy R(M) of a probabilized cut M = (Γ, P ) is defined (Shannon 1948) as R(M) = 1 − H(M)/ log |M|, P where |M| denotes the number of nodes in the cut Γ and H(M) = − n∈Γ P (n) · log P (n). Minimizing the redundancy is thus equivalent to maximizing the ratio between the entropy H(M) of the cut and its maximum possible value (log |M|), i.e., balancing as much as possible the probabilities of the nodes in the cut. Notice that R does not necessarily have a unique minimum, but the ontology may rather have several equally minimal cuts. In practice, this can easily be overcome, considering for instance any of the minimal cuts, or those having a minimal number of nodes, or the minimum average depth of the nodes, etc. In order to identify global MRC, the whole set of possible cuts has to be considered. We thus decided to give up global optimality for the sake of tractability and focussed rather on more efficient heuristics. The proposed algorithm consists, starting from the leaves, in iteratively modifying a given cut by systematically replacing a node by its parent or its children that minimizes the redundancy. For each node ni in the current cut, we consider on one hand ni ↓ the (set of) children of ni , and on the other hand ni ↑ the (set of) parents of ni . Due to the DAG structure, this replacement can involve other nodes in the cut. In fact, when replacing ni by ni ↓ , those nodes which are already covered by other nodes in the cut must be excluded, i.e., we must consider ni ↓ \ (Γ \ {ni })⇓ instead of ni ↓ 2
“minimal subset” means that no node can be removed from the set without decreasing it’s coverage.
28
F. SEYDOUX & J.-C. CHAPPELIER
Γ
ni
ni
Γ"
Γ" Γ
ni
m
(a)
∋
m
(Γ \ {ni })
(ni )
(b)
Fig. 2: (a) Lower search: node ni is replaced by ni ↓ without the nodes already covered by other nodes in Γ (e.g., m), i.e., (Γ \ {ni })⇓ . (b) Upper search: node ni is replaced by ni ↑ , and all nodes covered by ni ↑ ⇓ (i.e., (ni ↑ ) ) are removed from Γ. (see fig. 2a), where n⇓ stands for the transitive closure of n↓ ; and similarly for ni ↑ (see fig. 2b). Then, the cut with minimal redundancy among these new considered cuts and the current one is kept, and the search continue as long as better cuts are found. The full algorithm3 is given hereafter (Algorithm 1). This algorithm converges towards a local minimum redundancy cut close to the leaves. Note that this algorithm can be stopped any time, if required, since it always works on a complete cut. 3
Experiments
We carried out several experiments with standard English document collections of the Smart system,and ontologies generated from the MySQL port of WordNet 4 and the English part of EDR Electronic Dictionary. WordNet gathers information about approximatively 200, 000 “words” (≈ 150, 000 different lexical strings including compounds) of type noun, verb, adjective and adverb; organized into ≈ 115, 000 synsets, with ≈ 100, 000 hypernyms relations between them. EDR gathers information about approximatively 420, 000 “words” (≈ 240, 000 different lexical strings, including compounds and idiomatic expressions) of differents type (whithout restriction on POS); organized into ≈ 490, 000 concepts, with ≈ 500, 000 super/sub relations between them; we gather here the two differents ontologies provided by EDR (a very large scale general ontology and a smallest one specialized on information science). All the experiments were acheived following the same procedure: 3
4
In practice, several optimizations can be made, which do not conceptually change the algorithm and are thus not presented here for the sake of clarity. WordNet v.2.0, by Android Technologies.
SEMANTIC INDEXING USING MRC
29
Algorithm 1 MRC local search algorithm Requires: a hierarchy N (the leaves of which are W) Ensures: a cut Γ with (local) minimal redundancy Γ ←W {current cut, start from the leaves} repeat Γ′ ← ∅ {best new cut} Γ′′ ← ∅ {tested candidate} continue ← false {search-loop control flag} for all ni ∈ Γ do {Evaluate the children’s cut:} ′′ ↓ Γ ← (Γ \ {ni }) ∪ ni \ (Γ \ {ni })⇓ Γ′ ← Argmin R(Γ′ ), R(Γ′′ ) {Evaluate each parent’s cut:} for all nj ∈ ni ↑ do Γ′′ ← (Γ ∪ {nj}) \ nj ⇓
Γ′ ← Argmin R(Γ′ ), R(Γ′′ ) if R(Γ′ ) < R(Γ) then Γ ← Γ′ {keep the best cut} continue ← true {the search goes on} {some watchdog or timer can be put here} until continue is false return Γ with R(∅) = R({c}) = 1, by convention.
1. First, textual information from documents and queries are tokenized and lemmatized;tokens are then filtered, according to their POS tag (only nouns, verbs, adverbs and adjectives are kept). 2. Then, for every tokens in a document, corresponding entries (leaves) in the ontology are looked for, first using the lexical string, then with the lemma if necessary. Tokens with no correspondence in the considered ontology are indexed in the usual way. The coverage rate of the collections by the ontology was about 90% for both WordNet and EDR. 3. Then the hierarchy of concepts related to the tokens found in the ontology is expanded by selecting all possible senses for WordNet (relying on the mutual reinforcement induced by collocations to have a
30
F. SEYDOUX & J.-C. CHAPPELIER ...
Γ
A
B
g
C
f
documents (and queries)
e
d
c
b
a
queries only
Fig. 3: Possibles configuration when indexing the queries’ terms. Word ‘a’ (not in the ontology) is always indexed by himself, while ‘d’ is always indexed by ‘C’ and ‘e’, ‘f ’ and ‘g’ are always indexed by ‘B’. In the 0up strategy, ‘b’ and ‘c’ are not indexed; in the 1up strategy, ‘c’ is indexed by ‘C’, and in the 2up strategy, ‘b’ is indexed as well by ‘B’ and ‘C’. sort of disambiguation), and only the most frequent sense for EDR 5 . 4. A MRC cut is then computed with the algorithm previously presented. 5. The index of documents and queries are then computed; each token is substituted by the concepts from the cut which subordinate it. As the cut was computed only with nodes covering words contained in the documents (not the queries), tokens covered by the ontology but not by concepts in the selected cut can occur in the queries. We thus evaluated the three following strategies (see Figure 3): “0up” the first strategy simply consists in ignoring these tokens (in Figure 3: ’b’ and ’c’ are not indexed); “1up” in a more sophisticated strategy, we look if the related concepts (or synsets) subordinate a part of the cut; in this case, the term is indexed by the subordinate part of the cut, otherwise it is ignored (in Figure 3: ’b’ is ignored but ’c’ is indexed by ’C’); “2up” the most sophisticated strategy evaluated here consists in also ↑ looking for the hypernyms of the related concepts, i.e., (t↑ ) (in Figure 3: ’c’ is always indexed by ’C’, but ’b’ is indexed as well, by ’B’ and ’C’). Tokens not covered by the ontology are indexed in the standard way (in the figure, the token ’a’ is always indexed by itself).
5
This technique gives better results rather than keeping all possible senses in EDR (Seydoux & Chappelier 2005); this kind of WSD was not possible for WordNet, because the most frequent sense information was not present in the WordNet version used.
SEMANTIC INDEXING USING MRC
31
6. Finally, search and evaluation are performed, using the vector-space Smart information retrieval system (Salton 1971). Table 1 gathers the 11-pt precision and the 30-doc recall for the experiments carried out. 4
Discussion and conclusion
Three main conclusions can be drawn out of these experiments: 1. Using adapted additional semantic information can enhance the indexing of documents, and thus the performance of a IR system. The results of semantic (ontology-based) indexing (columns (c)) are better than the baseline system for three of the collections, but clearly worse on the MED collection. This can be explained by the specificity of the vocabulary of these collections, and their adequacy with the semantic resource. ADI and CACM have an important technical vocabulary, but well covered by the EDR ontology6 . CACM documents are extremely small, often restricted to a single title with few words. Conversely, the vocabulary of TIME is very general and the length of each document is large. The CISI collection present documents of average size, but with a significant number of dates, proper names, etc., for which the POS filtering seems to have annoying consequences (the very low performance is clearly due to an initial loss of information). Finally, the MED collection has an extremely specific vocabulary, for which the used ontologies were not adapted. 2. Excepted for the TIME collection, the results on EDR are globally better than those obtained on WordNet. This is probably caused by the (kind of) “WSD” technique used with EDR. Keeping all possible senses with EDR gives almost the same results as with WordNet, excepted for the TIME collection, for which the WordNet ontology seems really better. A semantic disambiguation, even rudimentary, appears thus to be necessary. The expected mutual reinforcement of collocations as a kind of “natural” disambiguation does actually not occur. The reason is probably that the hypernyms relations used do not constitute thematic links (for example, the thematically related terms ‘doctor’, ‘drug’, ‘hospital’, ‘nurse’, etc. are not linked together with the used ontologies). Anyway, this result enfavours the use of a proper WSD procedure for further improving the results. 3. The different strategies for dealing with tokens’ terms not covered by the indexing set (the cut) lead to different outcomes, depending on the 6
In this special case however, the kind of WSD used here with EDR is not adapted (especially for ADI). Better results were obtained with EDR without WSD – see (Seydoux & Chappelier 2005).
32
F. SEYDOUX & J.-C. CHAPPELIER
(a) (b) (c)-0up (c)-1up ADI collection (82 documents, 35 queries) [Documents from Information Science] EDR, most frequent precision 0.2497 0.2939 0.2924 0.2933 recall 0.5996 0.7141 0.6901 0.6996 concept, tf EDR, most frequent precision 0.3578 0.4274 0.4266 0.4238 recall 0.6984 0.7217 0.7081 0.7176 concept, tf.idf precision 0.2497 0.2671 0.2593 0.2562 WordNet, all synsets, tf recall 0.5996 0.6361 0.6146 0.6064 WordNet, all synsets, precision 0.3578 0.3564 0.3547 0.3450 recall 0.6984 0.6649 0.6767 0.6494 tf.idf TIME collection (423 documents, 83 queries) [General world news articles from the magazine Time (1963)] EDR, most frequent precision 0.3288 0.3692 0.3964 0.3908 recall 0.7755 0.7590 0.8124 0.8124 concept, tf EDR, most frequent precision 0.5496 0.5565 0.5602 0.5543 recall 0.8901 0.9053 0.8968 0.8975 concept, tf.idf precision 0.3288 0.2579 0.2348 0.3449 WordNet, all synsets, tf recall 0.7755 0.6263 0.6122 0.7760 WordNet, all synsets, precision 0.5496 0.5059 0.5565 0.5668 recall 0.8901 0.8421 0.9196 0.9161 tf.idf MED collection (1033 documents, 30 queries) [Collection of abstract from a medical journal] EDR, most frequent precision 0.3623 0.3229 0.3251 0.3253 recall 0.4574 0.4230 0.4214 0.4238 concept, tf EDR, most frequent precision 0.4607 0.4518 0.4394 0.4380 recall 0.5547 0.5404 0.5344 0.5386 concept, tf.idf precision 0.3623 0.2750 0.1914 0.3174 WordNet, all synsets, tf recall 0.4574 0.3803 0.2894 0.4216 WordNet, all synsets, precision 0.4607 0.4216 0.3329 0.4390 recall 0.5547 0.5120 0.4267 0.5263 tf.idf CISI collection (1460 documents, 112 queries) [Articles from Information science (Library science)] EDR, most frequent precision 0.0687 0.0805 0.0817 0.0817 recall 0.1239 0.1300 0.1243 0.1243 concept, tf EDR, most frequent precision 0.1733 0.1825 0.1785 0.1672 recall 0.2318 0.2313 0.2243 0.2243 concept, tf.idf precision 0.0687 0.0588 0.0449 0.0738 WordNet, all synsets, tf recall 0.1239 0.0926 0.0745 0.1226 WordNet, all synsets, precision 0.1733 0.1336 0.0875 0.1653 recall 0.2318 0.1979 0.1364 0.2204 tf.idf CACM collection (3204 documents, 64 queries) [Collection of titles and abstracts from a Computer science journal] EDR, most frequent precision 0.1555 0.1472 0.1525 0.1493 recall 0.3082 0.2926 0.2996 0.2982 concept, tf EDR, most frequent precision 0.2865 0.2804 0.2964 0.2796 recall 0.4534 0.4567 0.4514 0.4489 concept, tf.idf precision 0.1555 0.1628 0.1101 0.1637 WordNet, all synsets, tf recall 0.3082 0.2736 0.2019 0.2993 WordNet, all synsets, precision 0.2865 0.2390 0.2024 0.2621 recall 0.4534 0.3758 0.3366 0.4390 tf.idf
(c)-2up
0.2844 0.6806 0.3700 0.7007 0.2567 0.6064 0.3353 0.6422
0.3908 0.8124 0.5544 0.8975 0.3451 0.7880 0.5695 0.9281
0.3158 0.4143 0.3989 0.4862 0.3212 0.4241 0.4369 0.5261
0.0814 0.1257 0.1621 0.2211 0.0739 0.1224 0.1644 0.2192
0.1482 0.2982 0.2724 0.4363 0.1625 0.2940 0.2562 0.4354
Table 1: Evaluation of several indexing scheme (see Figures 1 and 3) on several collections: (a): words only; (b): direct hypernyms; (c)-0up, (c)-1up and (c)-2up : hypernyms from the MRC.
SEMANTIC INDEXING USING MRC
33
used ontology. As with EDR, the simple ’0up’ strategy gives better results, this strategy is clearly the worst with WordNet, while the ’2up’ seems to give the best results. This can perhaps be explained by the structure of the ontologies: all the synsets of WordNet directly cover several leaves, while for EDR, many of the concepts are just “pure concept”, only related to others concepts. As future work, it is forseen to confront the results presented here with similar experiments using better WSD procedure (especially for WordNet), and also using different weighting scheme for the evaluation (as for example the “lnu-weighting”, more reliable for handling noisy data and dealing with long documents). It would be also interesting to generalize this technique in order to take in account all relationships between terms given by the ontology, in addition to the thesaurus of hypernyms. Finally, we want to emphasize that the technique presented here does not limit to IR but could also be apply to other NLP domains, such as document clustering (Hotho et al. 2003) and text summarization. REFERENCES Besan¸con, R., J.-C. Chappelier, M. Rajman & A. Rozenknop. 2001. “Improving Text Representations through Probabilistic Integration of Synonymy Relations”. Xth Int. Symp. on Applied Stochastic Models and Data Analysis (ASMDA’2001 ), vol. 1, 200-205. Compi`egne, France. Deerwester, S.C., S.T. Dumais, T.K. Landauer, G.W. Furnas & R.A. Harshman. 1990. “Indexing by Latent Semantic Analysis”. Journal of the American Society of Information Science 41:6.391-407. Fellbaum, Christiane, ed. 1998. WordNet, An Electronic Lexical Database. Cambridge, Mass.: MIT Press. Gonzalo, J., F. Verdejo, I. Chugur & J. Cigarran. 1998. “Indexing with WordNet Synsets Can Improve Text Retrieval”. COLING/ACL Workshop on Usage of WordNet for Natural Language Processing, 38-44. Montreal, Canada. Gonzalo, J., F. Verdejo, C. Peters & N. Calzolari. 1998. “Applying EuroWordNet to Multilingual Text Retrieval”. Journal of Computers and the Humanities 32:2-3.185-207. Harman, D. 1988. “Towards Interactive Query Expansion”. 11th Conf. on Research and Development in Information Retrieval, 321-331. Grenoble. Hofmann, T. 1999. “Probabilistic Latent Semantic Indexing”. 22nd Int. Conf. on Research and Development in Information Retrieval, 50-57. Berkeley. Hotho, A., S. Staab & G. Stumme. 2003. “WordNet Improves Text Document Clustering”. SIGIR 2003 Semantic Web Workshop. Toronto, Canada. Ide, Nancy & J. V´eronis. 1998. “Word Sense Disambiguation: The State of the Art”. Computational Linguistics 24:1.1-40.
34
F. SEYDOUX & J.-C. CHAPPELIER
Kiryakov, A.K. & K.Iv. Simov. 1999. “Ontologically Supported Semantic Matching”. Nordic Conf. on Computational Linguistics. Trondheim, Norway. Mihalcea, R. & D. Moldovan. 2000. “Semantic Indexing Using WordNet Senses”. ACL Workshop on IR & NLP. Hong Kong. Miller, G.A. 1995. “Wordnet: A Lexical Database for English”. Communications of the ACM 38:11.39-41. Miyoshi, H., K. Sugiyama, M. Kobayashi & T. Ogino. 1996. “An Overview of the EDR Electronic Dictionary and the Current Status of Its Utilization”. 16th Int. Conf. on Computational Linguistics (COLING-1996 ), 1090-1093. Copenhagen. Moldovan, D.I. & R. Mihalcea. 2000. “Using WordNet and Lexical Operators to Improve Internet Searches”. IEEE Internet Computing 4:1.34-43. Richardson, R. & A.F. Smeaton. 1995. “Using WordNet in a Knowledge-based Approach to Information Retrieval”. Technical Report CA-0395. Dublin, Ireland: Dublin City University. Salton, G. 1968. Automatic Information Organization and Retrieval. New York: McGraw-Hill Book Company. Salton, G. 1971. The SMART Retrieval System - Experiments in Automatic Document Processing. Englewood Cliffs, NJ: Prentice Hall Inc. Seydoux, F. & J.-C. Chappelier. 2005. “Hypernyms Ontologies for Semantic Indexing”. Methodologies and Evaluation of Lexical Cohesion Techniques in Real-world Applications (electra’05), 49-55. Salvador, Brazil. Shannon, C.E. 1948. “A Mathematical Theory of Communication”. The Bell System Technical Journal 27.379-423. Smeaton, A.F. & I. Quigley. 1996. “Experiments on Using Semantic Distances between Words in Image Caption Retrieval”. 19th Int. ACM-SIGIR Conf. on Research and Development in Information Retrieval, 174-180. Zurich. Voorhees, E.M. 1993. “Using WordNet to Disambiguate Word Senses for Text Retrieval”. 16th Annual Int. ACM-SIGIR Conf. on Research and Development in Information Retrieval, 171-80. Pittsburgh, Penn. Voorhees, E.M. 1994. “Query Expansion Using Lexical-Semantic Relations”. 17th Int. ACM-SIGIR Conf. on Research and Development in Information Retrieval, 61-69. Dublin, Ireland. Voorhees, E.M. 1998. “Using WordNet for Text Retrieval”. WordNet, An Electronic Lexical Database 285-303. Cambridge, Mass.: MIT Press. Whaley, J.M. 1999. “An Application of Word Sense Disambiguation to Information Retrieval”. Technical Report PCS-TR99-352. Hanover, New Haven: Dartmouth College, Computer Science Dept. Wilks, Yorick & Mark Stevenson. 1998. “Word Sense Disambiguation Using Optimised Combinations of Knowledge Sources”. 17th Int. Conf. on Computational Linguistics (COLING-1998 ), 1398-1402. Montreal, Canada.
Indexing and Querying Linguistic Metadata and Document Content Niraj Aswani, Valentin Tablan, Kalina Bontcheva & Hamish Cunningham University of Sheffield Abstract The need for efficient corpus indexing and querying arises frequently both in machine learning-based and human-engineered natural language processing systems. This paper presents the annic system, which can index documents not only by content, but also by their linguististic annotations and features. It also enables users to formulate versatile queries mixing keywords and linguistic information. The result consists of the matching texts in the corpus, displayed within the context of linguistic annotations (not just text, as is customary for kwic systems). The data is displayed in a graphical user interface, which facilitates its exploration and the discovery of new patterns, which can in turn be tested by launching new annic queries.1 .
1
Introduction
The need for efficient corpus indexing and querying arises frequently both in machine learning-based and human-engineered natural language processing systems. A number of query systems have been proposed already and Mason (1998:20-27), Bird (2000b:1699-1706) and Gaizauskas (2003: 227-236) are amongst the most recent ones. In this paper, we present a fullfeatured annotation indexing and retrieval search engine, called annic (ANNotations-In-Context), which has been developed as part of gate (General Architecture for Text Engineering) (Cunningham 2002:168175). Whilst systems such as McKelvie & Mikheev (1998), Gaizauskas (2003: 227-236) and Cassidy (2002) are targeted towards specific types of documents, Bird (2000b:1699-1706) and Mason (1998:20-27) are general purpose systems. Annic falls in between these two types, because it can index documents in any format supported by the gate system. These existing systems were taken as a starting point, but annic goes beyond their capabilities in a number of important ways. New features address issues such as extensive indexing of linguistic information associated with document content, 1
This work was partially supported by an ahrb grant etcsl and an eu grant sekt
36
ASWANI, TABLAN, BONTCHEVA & CUNNINGHAM
independent of document format. It also allows indexing and extraction of information from overlapping annotations and features. Its advanced graphical user interface provides a graphical view of annotation mark-ups over the text along with an ability to build new queries interactively. In addition, annic can be used as a first step in rule development for nlp systems as it enables the discovery and testing of patterns in corpora. Section 2 introduces the gate text processing platform which is the basis of this work. Following this, we provide details of the annic implementation and the changes made in Lucene. 2
GATE
Gate is a large-scale infrastructure for natural language processing applications (Cunningham 2002:168-175). Lingustic data associated with language resources such as documents and corpora is encoded in the form of annotations. Gate supports a variety of formats including xml, rtf, html, sgml, email and plain text. In all cases, when a document is created/opened in gate, the format is analysed and converted into a single unified model of annotation. The annotation format is a modified form of the tipster format (Grisham 1997) which has been made largely compatible with the Atlas format (Bird 2000a:807-814). The annotations associated with each document are a structure central to gate, because they encode the language data read and produced by each processing module. Each annotation has a start and an end offset and a set of features associated with it. Each feature has a “name” and a “value”, which holds the descriptive or analytical information such as part-of-speech tags, syntactic analysis, co-reference information etc. Jape, Java Annotation Patterns Engine, is part of the gate system. It is an engine based on regular expression pattern/action rules over annotations. Jape is a version of cpsl (Common Pattern Specification Language). This engine executes the jape grammar phases - each phase consists of a set of pattern/action rules. The left-hand-side (lhs) of the rule represents an annotation pattern and the right-hand-side (rhs) describes the action to be taken when pattern found in the document. Jape executes these rules in a sequential manner and applies the rhs action to generate new annotations over the matched regular expression pattern. Rule prioritisation (if activated) prevents multiple assignments of annotations to the same text string. This paper demonstrates how annic indexes gate processed documents with their annotations and features and enables users to formulate versatile queries using jape patterns.
INDEXING & QUERYING METADATA & CONTENT
3
37
ANNIC
Annic is built on top of the Apache Lucene2 a high performance fullfeatured search engine implemented in Java which supports indexing and search of large document collections. Our choice of ir engine is due to the customisability of Lucene. After few changes in the behaviour of the key components of Lucene, we were able to make Lucene adaptable to our requirements. 3.1 Lucene token generation For each word in a document, Lucene creates a separate token. Every Lucene token has its own position in the token stream. This position remains relative to its previous token and is stored as a position increment factor. Since Lucene only indexes the text attribute of a Lucene token, to meet our requirements, i.e., to index the linguistic information and metadata, Lucene was modified to index also the type attribute. Type attribute holds a string assigned by lexical analyzer that defines the lexical or syntactic class of the Lucene token. Every annotation in gate has a corresponding features associated with it. We create a separate Lucene token for every feature in the document. Consider the following example: The word ‘‘Bill’’ is annotated as: AnnotationType:Token Features=POS:NNP,String:Bill AnnotationType:Person Sr. No. 1 2 3 4
Token Text Token NNP Bill Person
Token Type * Token.pos Token.string *
Pos. Incr. 1 0 0 0
Description Annotation Type “Token” “pos” feature with value “NNP” “string” feature with value “Bill” Annotation Type “Person”
Table 1: Token stream for the word “Bill” Table 3.1 explains the token stream that contains tokens for the above annotations. The annotation type itself is stored as a separate Lucene token with its attribute type “*” and text as the value of “annotation type”. This allows users to search for a particular annotation type. In order not to confuse features of one annotation with others, feature names are qualified with their respective annotation type names. Where there exist multiple annotations over the same piece of text, only the position of the very first feature of the very first annotation is set to 1 and it is set to 0 for the rest of the annotations and their features. This enables users to query over overlapping annotations and features. 2
http://lucene.apache.org
38
ASWANI, TABLAN, BONTCHEVA & CUNNINGHAM
It is possible for two annotations to share the same offsets. If two annotations share both offsets, such annotations are kept on the same position in the token stream, and otherwise one after another. But, what if annotations overlap each other (i.e., they share only one of the start and end offsets)? In this case, though annotations do not appear one after another, they are stored one after another. This may lead to incorrect results being returned and therefore the results are verified in order to filter out invalid overlapping patterns. Before indexing gate documents with Lucene, we convert them into the Lucene format and refer to them as gate Lucene documents. For each document, the token stream is stored in a separate file as a Java serializable object in the index directory. Later, it is retrieved in order to fetch left and right contexts of the found pattern. 3.2
Gate query parser
Jape patterns support various query formats. Below we give few examples of jape patterns. Actual patterns can also be a combination of one or more of the following pattern clauses: 1. 2. 3. 4. 5. 6.
“String” {AnnotationType} {AnnotationType == “String”} {AnnotationType.feature == “feature value”} {AnnotationType1, AnnotationType2.feature == “featureValue”} {AnnotationType1.feature == “featureValue”, AnnotationType2.feature == “featureValue”}
Order of the annotations specified in annic query is very important. In Lucene, document must contain the specified keywords, no matter in which order they exist. Order is important only for the phrase queries. Since the default implementation of Lucene indexer indexes only the text attribute of Lucene Token, it does not allow searching over the type attribute. Lucene query parser does not support position increments in queries. For example if one wants to search for annotations of type Location and Person referring to the same piece of text, Lucene does not support this. On the other hand, the respective JAPE pattern would be {Location, Person}. JAPE Pattern Token “String” {annotType} {annotType.featureType == “value”}
Term Text String annotType value
Query Term Term Type Token.string “*” annotType.featureType
Table 2: JAPE pattern tokens and their respective query terms We introduce our own query parser (annic query parser) which accepts jape queries and converts them into Lucene queries. Lucene query parser,
INDEXING & QUERYING METADATA & CONTENT
39
before accessing index, converts each keyword into an instance of Term class and compares them with the terms in index. Table 3.2 demonstrates how jape pattern tokens are converted into query terms. In order to use predefined Lucene queries (i.e., Boolean and Phrase queries), jape patterns with “OR” operator are normalized into the “AND normalized form” and all such patterns are ORed together to form a Boolean query. Lucene Phrase query considers its each token as a separate term and sets its position to the previous terms position + 1. This behaviour leads to a problem in the context of jape queries. For example, user issues the following query: {Location, Person.gender = ‘‘male’’}
This should search for the text that is annotated as Location and Person, where the Person annotation must contain a feature called “gender” with value “male”. In this case, the annic query parser creates two separate terms (Location and Person.gender = “male”). In order to make both terms referring to the same location, positions of these terms must remain same. If the position of first term is n, Lucenes phrase query implementation makes the position of second term to n+1. This results into a pattern where the first annotation is Location and is followed by the annotation Person.gender = “male”. To overcome this problem, one solution is to pass customized term positions along with terms to the phrase query. Given a term and its position respective to its previous term, Lucene searches within its index to find the term only at the given position. Thus, instead of searching the second term at the n+1 position, Lucene seeks a term that occurs at n position. This disables automatic increment in term’s position and also allows searching for the overlapping annotation. But even after this arrangment, there exists one major overlapping problem. For example for the text “Mr. Tim-Berners Lee told ...”, where the text “Mr.” is annotated as “Title”, “Tim-Berners” as “FirstName”, “Lee” as “Surname”, “Mr. Tim-Berners Lee” as “Person” and finally “told” as “Token” with the part-of-speech tag “verb”. For these annotations, the tokens “Title” and “Person” will be placed at the same position in the token stream, while “FirstName”, “Surname” and “Verb” will be placed one after another after the “Title” and the “Person” annotations. This results into incorrect results when the query is : {Person} {Token.string == “told”}. When searching this pattern in the token stream, “Person” is not followed by the Token string “told”, instead “Person” is followed by the annotation “FirstName”, which is followed the annotation “Surname” and which is followed by the “told”. To solve this problem, after converting the jape query into the Lucene query terms, we issue the query that contains only the initial terms which refer to the same location. For example, instead of querying with {Person}{Token.string == “told”} , we query index with {Person} . As a result
40
ASWANI, TABLAN, BONTCHEVA & CUNNINGHAM
this query returns all positions from the token stream where the annotation is “Person”. We compare the rest terms (i.e., “Token.string == “told” ) by fetching terms after the “Person” annotation and by comparing query terms with them. Klene operators: Annic supports two operators, + and *, to specify the number of times a particular annotation or a sub pattern should appear in the main query pattern. Here, ({A})+n means one and up to n occurrences of annotation {A} and ({A})*n means zero or up to n occurrences of annotation {A}. 4
ANNIC user interface
Annic provides an advanced user interface at the presentation layer that allows users to index a large collection of documents (i.e., corpus), search indices and analyze the found patterns along with their left and right contexts concordances. At indexing time, the user can specify the corpus to be indexed, the annotation type that acts as document tokens, annotation set which contains the annotations to index, features and annotation types not to include in index and finally the location of index on the local or network file system. At search time, the user specifies the maximum number of patterns to retrieve as results, number of tokens to show in the left and right contexts and finally the jape pattern query. 4.1
ANNIC viewer
Fig. 1: annic Viewer Figure 1 gives a snapshot of the annic search window. The bottom section in the window contains the patterns along with their left and right context
INDEXING & QUERYING METADATA & CONTENT
41
concordances and the section at top shows graphical visualization of annotations. Annic shows each pattern in a separate row and provides tool tip that shows the query that the selected pattern refers to. Along with its left and right context texts, it also lists the name of documents that the patterns come from. When the focus changes from one pattern to another, graphical visualization of annotations (gva, above the pattern table) changes its current focus to the selected pattern. Here, users have an option of visualising annotations and their features for the selected pattern. The figure shows the highlighted spans of annotations for the selected pattern. Annotation types and features can also be selected from the drop-down combo box and their spans can also be highlighted into the gva. When users choose to highlight the features of annotations (e.g., Token.cat), gva shows the highlighted spans containing values of those features. Whereas when users choose to highlight the annotation with feature “all”, annic adds a blank span in gva and shows all its features in a popup window when mouse enters the span region. A new query can also be generated and executed from the annic gui. When clicked on any of the highlighted spans of the annotations, the respective query clause is placed in the New Query text box. Clicking on Execute issues a new query and refreshes the gui output. Annic also provides an option to export results in xml or html files with options of all patterns and selected patterns. QP Patterns 1 {Token.string==“Microsoft”} | “Microsoft Corp” 2 {Person} {Token.cat==“IN”} {Token.cat==“DT”})*1 {Organization} 3 ({Token.kind==number})+4 ({Token.string =“/”} | {Token.string==“-”}) ({Token.kind==number})+2 ({Token.string==“/”} | {Token.string==“-”}) ({Token.kind==number})+2 4 {Title} ({Token.orth==upperInitial} | {Token.orth==allCaps}) ({FirstPerson})*1 5 ({Token.cat==“DT”})*1 {Location} {Token.cat==“CC”} ({Token.cat==“DT”})*1 {Location} {Organization}{Token.cat==“IN”} ({Token.cat==“DT”})*1 {Location} QP=Query Pattern
Table 3: ANNIC queries
5
Applications of ANNIC
Annic is used as a tool aiding the development of jape rules. Language engineers use their intuition when writing jape rules trying to strike the ideal balance between specificity and coverage. This requires them to make a series of informed guesses which are then validated by testing the resulting ruleset over a corpus. Annic can replace the guesswork in this process with
42
ASWANI, TABLAN, BONTCHEVA & CUNNINGHAM
actual live analysys of the corpus. Each pattern intended as part of a jape rule can be easily tested directly on the corpus and have its specificity and coverage assesed almost instantaneously. Annic can be used also for corpus analysys. It allows querying the information contained in a corpus in more flexible ways than simple full-text search. Consider a corpus containing news stories that has been processed with a standard named entity recognition system like annie3 . A query like {Organization} ({Token})*3 ({Token.string==’up’}|{Token.string==’down’}) ({Money} | {Percent})
would return mentions of share movements like “BT shared ended up 36p” or “Marconi was down 15%”. Locating this type of useful text snippets would be very difficult and time consuming if the only tool available were text search. Annic can also be useful in helping scholars to analyse linguistic corpora. Sumerologists, for instance, could use it to find all places in the etcsl corpus 4 where a particular pair of lemmas occur in sequence. 6
Performance results
In order to evaluate the performance of annic, we experimented on three different corpora (large, medium, and small), processed with gate: 10% of the bnc (British National Corpus)(374 documents,1443.84MB), hse (Health and Security Experiments)(192 documents,896MB), and finally the news corpus (446 documents, 39.4MB). We tested the performance with several types of queries: string only queries, combinations of strings and linguistic data, and patterns with quantified Klene operators. Table 3 lists some of the different types of queries which were issued over the indexed corpuses. Table 4 gives the statistics of output of these queries. It provides different statistics including the time taken by annic to retrieve the results and the number of patterns retrieved. BNC 10% HSE NEWS QP ST P ST P ST P 1 11.276 112 0.5 0 1.252 3 2 5.23 6 7.0 6 0.432 2 3 50.139 264 110.738 39 6.652 36 4 39.029 238 120.054 180 12.37 1038 5 62.971 81 126.823 124 5.508 281 6 6.191 10 11.875 5 0.692 11 QP=Query Pattern,ST=Search Time,P=Patterns
Table 4: ANNIC query results 3 4
Gate is distributed with an ie system called annie, A Nearly-New IE system. http://www-etcsl.orient.ox.ac.uk/
INDEXING & QUERYING METADATA & CONTENT
7
43
Related work
McKelvie & Mikheev (1998) describe a suite of programs, lt index, that supports indexing of large sgml documents. It indexes elements by their position in the document structure and by their textual content. Annic is more generic, because it can cope with a wider range of formats, while covering the same functionality. Cue (Corpus Universal Examiner) system (Mason 1998:20-27) splits the corpus data into different data streams (e.g., actual words, pos information), which are stored along with their positioning information in the index. Unlike cue, annic maintains a fixed structured data format (Term string, Term type, position) within indices and converts all annotations and their features into this consistent format. Gaizauskas (2003:227-236) describe a system, xara that indexes any well-formed xml document. It combines an indexer, a server and a windows client. Indexer requires information like how element content is to be tokenized and how tokens are to be mapped to index terms etc. Annic supports not only xml but many other types of documents supported by gate. Similar to xara, in annic as well, the decision of how documents be tokenized is left on a user (e.g., gate supplies tokenisers for several languages). In order to investigate new models for semi structured data that are appropriate to xml, Buneman (1998) describes a query language that is beyond any xml query languages. They describe extraction rules that consist of expressions along the tree and are expressed using the html Extraction Languages (hel). Their query language comes with navigation operators, regular expressions and conditions to retrieve information even from the nested structures. Annic query parser works on top of the gate annotations and features and supports search over overlapping annotations and features. Its advanced user interface allows users to visualize the nested structure of the annotations with their features highlighted. Kazai (2004:72-79) discuss the overlapping problem in content-oriented xml retrieval. They discuss the INitiative for the Evaluation of xml Retrieval (inex) system, which discusses the matrices to evaluate the xml retrieval results. Their argument is that if in an xml document, a sub element satisfies a content-oriented query, parent element would also satisfies the same query. Thus, instead of including only a subcomponent in the result, inex also includes the parent component. In annic, the overlapping problem, as discussed in Kazai (2004:72-79), does not exist due to two reasons. 1) Annotations in gate documents are stored as an annotation graph. Thus comparing the structure of xml documents where elements contain texts, in gate documents annotations are created over the text.
44
ASWANI, TABLAN, BONTCHEVA & CUNNINGHAM
2) Annic queries are very specific about the annotation types, i.e., query itself describes the annotation type in which the string should be searched. If user does not specify annotation type, annic does it automatically to search strings with the gate token annotation type. REFERENCES Bird, S., P. Buneman & W. Tan. 2000. “Towards a Query Language for Annotation Graphs”. Proceedings of the Second International Conference on Language Resources and Evaluation, 807-814. Athens. Bird, S., D. Day, J. Garofolo, J. Henderson, C. Laprun & M. Liberman. 2000b. “ATLAS: A Flexible and Extensible Architecture for Linguistic Annotation”. Proceedings of the Second International Conference on Language Resources and Evaluation, 1699-1706, Athens. Buneman, P., A. Deutsch, W. Fan, H. Liefke, A. Sahuguet & W.C. Tan. 1998. “Beyond XML Query Languages”. Proceedings of the Query Language Workshop (QL 1998 ). http://www.w3.org/TandS/QL/QL98/pp/penn.ps [Source checked in April 2006] Cassidy, S. 2002. “Xquery as an Annotation Query Language: A Use Case Analysis”. Proceedings of 3rd Language Resources and Evaluation Conference (LREC’2002 ), Gran Canaria, Spain. Cunningham, H., D. Maynard, K. Bontcheva & V. Tablan. 2002. “GATE: A Framework and Graphical Development Environment for Robust NLP Tools and Applications”. Proceedings of the 40th Anniversary Meeting of the Association for Computational Linguistics (ACL 2002 ), 168-175. Philadelphia. Gaizauskas, R., L. Burnard, P. Clough & S. Piao. 2003. “Using the XARA XMLAware Corpus Query Tool to Investigate the METER Corpus”. Proceedings of the Corpus Linguistics 2003 Conference, 227-236. Lancaster, U.K. Grishman, Ralph. 1997. “TIPSTER Architecture Design Document Version 2.3”. Technical report, The Defense Advanced Research Projects Agency (DARPA). http://www.itl.nist.gov/div894/894.02/related projects/tipster/ [Source checked in April 2006] Kazai, G., M. Lalmas & A. Vries. 2004. “The Overlapping Problem in ContentOriented XML Retrieval Evaluation”. Proceedings of the 27th International conference on Research and Development in Information Retrieval, 72-79. Sheffield, U.K. Mason, O. 1998. “The CUE Corpus Access Tool”. Workshop on Distributing and Accessing Linguistic Resources, 20-27, Granada, Spain. http://www.dcs.shef.ac.uk/~hamish/dalr/ [Source checked in February 2006]. McKelvie, D. & A. Mikheev. 1998. “Indexing SGML files using LT NSL”. LT Index documentation http://www.ltg.ed.ac.uk/ [Source checked in April 2006]
Term Representation with Generalized Latent Semantic Analysis Irina Matveeva∗ , Gina-Anne Levow∗ , Ayman Farahat∗∗ & Christiaan Royer∗∗ ∗
University of Chicago,
∗∗
Palo Alto Research Center
Abstract Representation of term-document relations is very important for document indexing and retrieval. In this paper, we present Generalized Latent Semantic Analysis as a framework for computing semantically motivated term and document vectors. Our focus on term vectors is motivated by the recent success of co-occurrence based measures of semantic similarity obtained from very large corpora. Our experiments demonstrate that glsa term vectors efficiently capture semantic relations between terms and outperform related approaches on the synonymy test.
1
Introduction
Document indexing and representation of term-document relations are crucial for document classification, clustering and retrieval (Deerwester et al. 1990, Ponte & Croft 1998, Salton & McGill 1983). Since many classification and categorization algorithms require a vector space representation for the data, it is often important to have a document representation within the vector space model approach (Salton & McGill 1983). In the traditional bag-of-words document representation (Salton & McGill 1983), words represent orthogonal dimensions which makes an unrealistic assumption about their independence within documents. Since the document vectors are constructed in a very high dimensional vocabulary space, there has been a considerable interest in low dimensional document representations. Latent Semantic Analysis (lsa) (Deerwester et al. 1990) is one of the best known dimensionality reduction algorithms that deals with synonymy and polysemy. The dimensions of the resulting vector space are interpreted as latent semantic concepts. In this paper, we introduce Generalized Latent Semantic Analysis (glsa) as a framework for computing semantically motivated term and document vectors. Glsa focuses on computing term vectors in the space of semantic concepts; document vectors are computed as linear combinations of term vectors. Thus, unlike lsa and other related approaches glsa is not based on bag-of-words document vectors. Instead, we begin with semantically motivated pair-wise term similarities to compute a representation for terms.
46
MATVEEVA, LEVOW, FARAHAT & ROYER
This shift from the dual document-term representation to term representation has the following motivation: • Terms offer a greater flexibility in exploring similarity relations than documents. The availability of large document collections such as the Web offers a great resource for statistical approaches. Co-occurrence based measures of semantic similarity between terms improve performance on semantic proximity tests, e.g., (Terra & Clarke 2003). • Content bearing words are often combined into semantic classes which contain synonyms and semantically related words. It seems natural to represent terms as low dimensional vectors in the space of semantic concepts. In this paper, we use a large document collection to extract point-wise mutual information, and the singular value decomposition as a dimensionality reduction method and compute term vectors. Our experiments show that the glsa term representation outperforms related approaches on term-based tasks such as the synonymy test. The rest of the paper is organized as follows. Section 2 contains the outline of the glsa algorithm. Section 4 presents our experiments, followed by conclusion in section 5. 2 2.1
Generalized Latent Semantic Analysis GLSA framework
The glsa algorithm has the following setup. We assume that we have a document collection C with vocabulary V and a large corpus W . 1. Construct the weighted term-document matrix D based on C; 2. For the vocabulary words in V , obtain a matrix of pair-wise similarities, S, using the large corpus W ; 3. Obtain the matrix U T of a low dimensional vector space representation of terms that preserves the similarities in S, U T ∈ Rk×|V | ; 4. Compute document vectors by taking linear combinations of term ˆ = U T D. vectors D Cosine similarity between term and document vectors is often used as a measure of semantic association. Therefore, we would like to obtain term vectors so that their pair-wise cosine similarities preserve the semantic similarity between the corresponding vocabulary terms. The glsa approach can combine any kind of similarity measure on the space of terms with any suitable method of dimensionality reduction. The traditional termdocument matrix is used in the last step to provide the weights in the linear combination of term vectors.
TERM REPRESENTATION WITH GLSA
2.2
47
Low-dimensional representation
2.2.1 Singular value decomposition The singular value decomposition (svd) is applied to the matrix S containing pair-wise similarities between the vocabulary terms. svd of S is S = UΣV T , where U and V are column orthogonal matrices containing the left and right singular vectors of S, respectively. Σ is a diagonal matrix with the singular values sorted in decreasing order. Eckart and Young, see Golub & Reinsch (1971), have shown that the matrix Sk = Uk Σk VkT obtained by setting all but the first k diagonal elements in Σ to zero is the best element-wise approximation of S: Sk = argminX ||S − X||2F , where X is a matrix of rank k. The P minimum is taken with respect to the Frobenius norm, where ||A||2F = ij A2ij . Since S is a real symmetric matrix, V = U, so that S = UΣU T . If all singular values are non-negative, S can be represented as a product of two matrices, S = Uˆ UˆT , with Uˆ = UΣ1/2 . This means that the entries of S, which in the glsa case represent pair-wise term similarities, are best preserved by inner products between the appropriately scaled left singular vectors of S. Lsa is a special case within the glsa framework. It uses svd to preserve the pair-wise term similarities computed as inner products between the term vectors in the space of bag-of-words documents, see Bartell et al. (1992) for details. 2.2.2 PMI as measure of semantic association The matrix of semantic associations between the pairs of vocabulary terms can be obtained using different collection-based term association measures, such as point-wise mutual information, likelihood ratio, χ2 test etc. (Manning & Schutze 1999). In this paper we use point-wise mutual information (pmi) because it has been successfully applied to collocation discovery and semantic proximity tests such as the synonymy test and taxonomy induction (Manning & Schutze 1999, Terra & Clarke 2003, Widdows 2003). Pmi between the random variables representing two words, w1 and w2 , is computed as pmi(w1 , w2 ) = log
P (W1 = 1, W2 = 1) . P (W1 = 1)P (W2 = 1)
The similarity matrix S with pair-wise pmi scores may not be positive semidefinite. Since such matrices work well in practice (Cox & Cox 2001) one
48
MATVEEVA, LEVOW, FARAHAT & ROYER
Fig. 1: Precision with GLSA, PMI and count over different window sizes, for the TOEFL (left), TS1 (middle) and TS2 (right) tests common approach is to use only the eigenvectors corresponding to the positive eigenvalues (Cox & Cox 2001). This is the approach which we use in our experiments. 3
Related approaches
Iterative Residual Rescaling (Ando 2000) tries to put more weight on documents from underrepresented clusters of documents to improve the performance of lsa on heterogeneous collections. Locality Preserving Indexing (lpi) (He et al. 2004) is a graph-based dimensionality reduction algorithm which preserves the similarities only locally. The Word Space Model (ws) for word sense disambiguation developed by Schutze (1998) and Latent Relational Analysis (lra) (Turney 2001) are another special cases of glsa which compute term vectors directly. These approaches are limited to only one measure of semantic association. They use weighted co-occurrence statistics in the document space (lsa, lpi), in the space of the most frequent informative terms (ws) and in the space of context patterns (lra) to compute input similarities. 4
Experiments
The goal of the experimental evaluation of the glsa term vectors was to demonstrate that the glsa vector space representation for terms captures their semantic relations. We used the synonymy and term pairs tests for the evaluation. To collect the co-occurrence information for the matrix of pair-wise term similarities S, we used a subset of the English Gigaword collection (ldc), containing New York Times articles labeled as “story”. We had 1,119,364
TERM REPRESENTATION WITH GLSA
49
documents with 771,451 terms. We used the Lemur toolkit1 to tokenize and index all document collections used in our experiments; we used stemming and a list of stop words. The similarities matrix S was constructed using the pmi scores. The results with likelihood ratio and χ2 were below those for pmi and we omit them here. We used the pmi matrix S in combination with svd to compute glsa term vectors. Unless stated otherwise, for the glsa method we report the best performance over different numbers of embedding dimensions. We used the plapack package2 to perform the svd. 4.1
Synonymy test
The synonymy test represents a list of words and for each of them, there are 4 candidate words. The task is to determine which of these candidate words is a synonym to the word in question. This test was first used to demonstrate the effectiveness of lsa term vectors (Landauer & Dumais 1997). More recently, the pmi-ir approach developed by Turney was shown to outperform lsa on this task (Terra & Clarke 2003, Turney 2001). We evaluated the glsa term vectors for the synonymy test and compared the results to Terra & Clarke (2003). We used the toefl test containing 80 synonymy questions. We also used the preparation tests called TS1 and TS2. Since glsa in its present formulation cannot handle multi-word expressions, we had to modify the TS1 and TS2 tests slightly. We removed all test questions that contained multi-word expressions. From 50 TS1 questions we used 46 and from 60 TS2 questions we used 49. Thus, we would like to stress that the comparison of our results on TS1 and TS2 to the results reported in Terra & Clarke (2003) is only suggestive. We used the TS1 and TS2 tests without context. The only difference in the experimental setting for the toefl test between our experiments and the experiments in Terra & Clarke (2003) is in the document collections that were used to obtain the co-occurrence information. 4.1.1 GLSA setting To have a richer vocabulary space, we added 2000 most frequent words from the English Gigaword collection to the vocabularies of the toefl, TS1 and TS2 tests and computed glsa term vectors for the extended vocabularies. We selected the term t∗ whose term vector had the highest cosine similarity to the question term vector t~q as the synonym. We computed precision scores as the ratio of correctly guessed synonyms. 1 2
http://www.lemurproject.org/ http://www.cs.utexas.edu/users/plapack/
50
MATVEEVA, LEVOW, FARAHAT & ROYER
The co-occurrence counts can be obtained using either term co-occurrence within the same document or within a sliding window of a fixed size. In our experiments we used the window-based approach which was shown to give better results (Schutze 1998, Terra & Clarke 2003). Since the performance of co-occurrence based measures is sensitive to the window size, we report the results for different window sizes.
Fig. 2: Precision at different numbers of GLSA dimensions with the best window size
4.1.2 Results on the synonymy test Figure 1 shows the precision using different window sizes. The baselines are to choose the candidate with the highest co-occurrence count or pmi score. For all three data sets, glsa significantly outperforms pmi scores computed on the same collection. The results that we obtained using just the pmi score are below those reported in Terra & Clarke (2003). One explanation for this discrepancy is the size and the composition of the document collections used for the co-occurrence statistics. The English Gigaword collection that we used is smaller and, more importantly, less heterogeneous than the web based collection in Terra & Clarke (2003). Nonetheless, on the toefl data set glsa achieves the best precision of 0.86, which is much better than our pmi baseline as well as the highest precision of 0.81 reported in Terra & Clarke (2003). GLSA achieves the same maximum precision as in Terra & Clarke (2003) for TS1 (0.73) and a much higher precision on TS2 (0.82 vs. 0.75 in Terra & Clarke (2003)). Figure 2 shows the precision for the glsa terms only, using different number of dimensions. svd-based approaches usually perform best with 300-400 resulting dimensions (Deerwester et al. 1990). The variation of precision at different numbers of glsa embedding dimensions within the 100-600 range is somewhat high for TS1 but much smoother for the toefl and TS2 tests.
51
TERM REPRESENTATION WITH GLSA
k 1 5 10 50 100
top pmi 0.0 0.0 0.0 0.32 0.24
100 glsa 1.0 1.0 1.0 0.88 0.76
top pmi 0.0 0.0 0.0 0.12 0.1
1000 glsa 1.0 1.0 0.8 0.8 0.8
Table 1: Precision for the term pairs test at the top k most similar pairs 4.2
k 1 5 10 50 100
top pmi 0.27 0.40 0.35 0.14 0.08
100 glsa 0.67 0.48 0.37 0.13 0.08
top pmi 0.08 0.8 0.1 0.16 0.16
1000 glsa 0.43 0.40 0.37 0.20 0.18
Table 2: Average precision for the term pairs test at the top k nearest words
Term pairs test
Some of the terms on the synonymy test are infrequent (e.g., “wig”) or not informative (e.g., “unlikely”). We used the following test to evaluate how the cosine similarity between glsa vectors captures similarity between content words. We computed glsa term vectors for the vocabulary of the 20 news groups document collection. Using the Rainbow software3 we obtained the top N words with the highest mutual information with the class label. We also obtained the probabilities that each of these words has with respect to each of the news groups. We assigned the group in which the word has the highest probability as the word’s label. Some of the top words and their labels can be seen in Table 3. Although the way we assigned labels may not strictly correspond to the semantic relations between words, this table shows that for this particular collection and for informative words (e.g., “bike”, “team”) they do make sense. We computed pair-wise similarities between the top N words using the cosine between the glsa vectors representing these words and also used just the pmi scores. Then we looked at the pairs of terms with the highest similarities. Since for this test we selected content bearing words, the intuition is that most similar words should be semantically related and are likely to appear in documents belonging to the same news group. Therefore, they should have the same label. In the synonymy task the comparisons are made between the pmi scores of a few carefully selected terms that are synonymy candidates for the same word. While pmi-ir performs quite well on the synonymy task, it is in general difficult to compare pmi scores across different pairs of words. Our experiments illustrate that glsa significantly outperforms the pmi scores on this test. We used N = {100, 1000} top words by the MI with the class 3
http://www-2.cs.cmu.edu/ mccallum/bow/rainbow/
52
MATVEEVA, LEVOW, FARAHAT & ROYER
word god (18) bike (15) team (17) car (7) windows (1) dod (15) article (15)
nn=1 jesus (18) motorcycle (15) coach (17) driver (1) microsoft (1) agency (10) publish (13)
nn=2 bible (18) rider (15) league (20) auto (7) os (3) military (13) fax (4)
nn=3 heaven (18) biker (15) game (17) ford (7) nt (1) nsa (10) contact (5)
Prec 1 1 0.6 0.6 0.4 0 0
Table 3: Precision at the 5 nearest terms for some of the top 100 words by mutual information with the class label. The table also shows the first 3 nearest neighbors. The word’s label is given in the brackets (1=os.windows; 3=hardware; 4=graphics; 5=forsale; 7=autos; 10=crypt; 13=middle-east;15=motorcycles; 17=hokey; 18=religion-christian; 20=baseball) label. The top 100 are highly discriminative with respect to the news group label whereas the top 1000 words contain many frequent words. Our results show that glsa is much less sensitive to this than pmi. First, we sorted all pairs of words by similarity and computed precision at the k most similar pairs as the ratio of word pairs that have the same label. Table 1 shows that glsa significantly outperforms the pmi score. pmi has very poor performance, since here the comparison is done across different pairs of words. The second set of scores was computed for each word as precision at the top k nearest terms, similar to precision at the first k retrieved documents used in information retrieval. We report the average precision values for different values of k in Table 2. Glsa achieves higher precision than pmi. Glsa performance has a smooth shape peaking at around 200-300 dimension which is in line with results for other svd-based approaches. The dependency on the number of dimensions was the same for the top 1000 words. In Table 3 we show results for some of the words. Glsa representation achieves very good results for terms that are good indicators of particular news groups, such as “god” or “bike”. For frequent words, and words which have multiple senses, such as “windows” or “article”, the precision is lower. The pair “car”, “driver” is semantically related for one sense of the word “driver”, but the word “driver” is assigned to the group “windows-os” with a different sense.
TERM REPRESENTATION WITH GLSA
5
53
Conclusion and future work
Our experiments have shown that the cosine similarity between the glsa term vectors corresponds well to the semantic similarity between pairs of terms. Interesting questions for future work are connected to the computational issues. As other methods based on a matrix decomposition, glsa is limited in the size of vocabulary that it can handle efficiently. Since terms can be divided into content-bearing and function words, glsa computations only have to include content-bearing words. Since the glsa document vectors are constructed as linear combinations of term vectors, the inner products between the term vectors are implicitly used when the similarity between the document vectors is computed. Another interesting extension is therefore to incorporate the inner products between glsa term vectors into the language modelling framework and evaluate the impact of the glsa representation on the information retrieval task. We have presented the glsa framework for computing semantically motivated term and document vectors. This framework allows us to take advantage of the availability of large document collections and recent research of corpus-based term similarity measures and combine them with dimensionality reduction algorithms. Using the combination of point-wise mutual information and singular value decomposition we have obtained term vectors that outperform the state-of-the-art approaches on the synonymy test and show a clear advantage over the pmi-ir approach on the term pairs test. Acknowledgements. We are very grateful to Paolo Bientinesi for his extensive help with adopting the plapack package to our problem. The toefl questions were kindly provided by Thomas K. Landauer, Department of Psychology, University of Colorado. This research has been funded in part by contract #MDA904-03-C-0404 to Stuart K. Card and Peter Pirolli from the Advanced Research and Development Activity, Novel Intelligence from Massive Data program. REFERENCES Ando, R.K. 2000. “Latent Semantic Space: Iterative Scaling Improves Precision of Inter-Document Similarity Measurement”. Proceedings of the 23th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR’00 ), 216-223. Athens, Greece. Bartell, B.T., G.W. Cottrell, & R.K. Belew. 1992. “Latent semantic indexing is an optimal special case of multidimensional scaling”. Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR’92 ), 161-167. Copenhagen, Denmark.
54
MATVEEVA, LEVOW, FARAHAT & ROYER
Cox, T.F. & M.A. Cox. 2001. Multidimensional Scaling. London: CRC/Chapman and Hall. Deerwester, S.C., S.T. Dumais, T.K. Landauer, G.W. Furnas & R.A. Harshman. 1990. “Indexing by Latent Semantic Analysis”. Journal of the American Society of Information Science 41:6.391-407. Golub, G. & C. Reinsch. 1971. Handbook for Matrix Computation II, Linear Algebra. New York: Springer-Verlag He, X., D. Cai, H. Liu & W. Ma. 2004. “Locality preserving indexing for document representation”. Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’04 ), 96-103. Sheffield, U.K. Landauer, T.K., S.T. Dumais. 1997. “A Solution to Platos Problem: The Latent Semantic Analysis Theory of the Acquisition, Induction, and Representation of Knowledge”. Psychological Review 104:2.211-240. Manning, C. & H. Schutze. 1999. Foundations of Statistical Natural Language Processing. Cambridge, Mass.: MIT Press. Ponte, J.M. & W.B. Croft. 1998. “A language modeling approach to information retrieval”. Proceedings of the 21th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’98 ), 275-281. Melbourne, Australia. Salton, G. & M.J. McGill. 1983. Introduction to Modern Information Retrieval. New York: McGraw-Hill. Schutze, H. 1998. “Automatic Word Sense Discrimination”. Computational Linguistics 24:21.97-124 Terra, E.L. & C.L. Clarke. 2003. “Frequency Estimates for Statistical Word Similarity Measures”. Proceedings of the Joint Human Language Technology Conference and the Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL’2003 ), vol. 1, 165-172. Edmonton, Canada. Turney, Peter D. 2001. “Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL”. Machine Learning: ECML 2001 — Proceedings of the 12th European Conference on Machine Learning, Freiburg, Germany ed. by Luc de Raedt & Peter Flach (= Lecture Notes in Artificial Intelligence, 2167), 491-502. Heidelberg: Spinger. Widdows, D. 2003. “A Mathematical Model for Context and Word-Meaning”. Proceedings of the 4th International and Interdisciplinary Conference on Modeling and Using Context, 23-25. Stanford, Calif.
Multilingual Dependency Parsing: A Pipeline Approach Ming-Wei Chang, Quang Do & Dan Roth University of Illinois at Urbana-Champaign Abstract This paper develops a general framework for machine learning based dependency parsing based on a pipeline approach, where a task is decomposed into several sequential stages. To overcome the error accumulation problem of pipeline models, we propose two natural principles for pipeline frameworks: (i) make local decisions as reliable as possible, and (ii) reduce the number of sequential decisions made. We develop an algorithm that provably satisfies these principles and show that the proposed principles support several algorithmic choices that improve the dependency parsing accuracy significantly. We present state of the art experimental results for English and several other languages.1
1
Introduction
A pipeline process over the decisions of learned classifiers is a common computational strategy in natural language processing. In this model a task is decomposed into several stages that are solved sequentially, where the computation in the ith stage typically depends on the outcome of computations done in previous stages. For example, a semantic role labeling program (Punyakanok et al. 2005) may start by using a part-of-speech tagger, then apply a shallow parser to chunk the sentence into phrases, identify predicates and arguments and then classify them to types. In fact, any left to right processing of an English sentence may be viewed as a pipeline computation as it processes a token and, potentially, makes use of this result when processing the token to the right. The pipeline model is a standard model of computation in natural language processing for good reasons. It is based on the assumption that some decisions may be easier to make and that the outcome of earlier decision may sometimes be useful when making further decisions. Nevertheless, it is clear that it results in error accumulation and suffers from its inability to correct mistakes made in previous stages. Researchers have recently started to address some of the disadvantages of this model. Roth & Yih (2004) suggests a model in which global constraints are taken into account in a 1
This paper extends and unifies our previous works (Chang et al. 2006a, 2006b).
56
MING-WEI CHANG, QUANG DO & DAN ROTH
later stage to fix mistakes due to the pipeline. Punyakanok et al. (2005) and Marciniak & Strube (2005) also address some aspects of this problem. However, these solutions rely on the fact that all decisions are made with respect to the same input; specifically, all classifiers considered use the same examples as their input. In addition, the pipelines they study are shallow. This paper develops a general framework for decisions in pipeline models which addresses these difficulties. Specifically, we are interested in deep pipelines—a large number of predictions that are being chained. A pipeline process is one in which decisions made in the ith stage (1) depend on earlier decisions and (2) feed on input that depends on earlier decisions. The latter issue is especially important at evaluation time since, at training time, a gold standard data set might be used to avoid this issue. We develop and study the framework in the context of a bottom up approach to dependency parsing. We suggest that two principles should guide the pipeline algorithm development: (i) Make local decisions as reliable as possible. (ii) Reduce the number of decisions made. Using these as guidelines we devise an algorithm for dependency parsing, prove that it satisfies these principles, and show experimentally that this improves the accuracy of the resulting tree. Specifically, our approach is based on a shift-reduced parsing as in (Yamada & Matsumoto 2003). Our general framework provides insights that allow us to improve their algorithm, and to principally justify some of the algorithmic decisions. Specifically, the first principle suggests to improve the reliability of the local predictions, which we do by improving the set of actions taken by the parsing algorithm, and by using a look-ahead search. The second principle is used to justify the control policy of the parsing algorithm, namely, which edges to consider at different stages in the algorithm. We prove that our control policy is optimal in some sense and, overall, that the decisions we made guided by these principles lead to a significant improvement in the accuracy of the resulting parse tree. 1.1 Dependency parsing and pipeline models Dependency trees provide a syntactic representation that encodes functional relationships between words; it is relatively independent of the grammar theory and can be used to represent the structure of sentences in different languages. Dependency structures are more efficient to parse (Eisner 1996) and are believed to be easier to learn, yet they still capture much of the predicate-argument information needed in applications (Haghighi et al. 2005), which is one reason for the recent interest in learning these structures (Lin 1994, Eisner 1996, Yamada & Matsumoto 2003, Nivre & Scholz 2004, Mcdonald et al. 2005).
MULTILINGUAL DEPENDENCY PARSING
57
Eisner’s work—O(n3 ) parsing time generative algorithm—embarked the interest in this area. His model, however, seems to be limited when dealing with complex and long sentences. Mcdonald et al. (2005) build on this work, and use a global discriminative training approach to improve the edges’ scores, along with Eisner’s algorithm, to yield an expected improvement. A different approach was studied by Yamada & Matsumoto (2003), that develop a bottom-up approach and learn the parsing decisions between consecutive words in the sentence. Local actions are used to generate a dependency tree using a shift-reduce parsing approach (Aho et al. 1986). This is a true pipeline approach in that the classifiers are trained on individual decisions rather than on the overall quality of the parser, and chained to yield the global structure. Conceptually similar work was done in other successful parsers, e.g., (Ratnaparkhi 1997). Clearly, this approach suffers from the limitations of pipeline processing, such as accumulation of errors, but nevertheless, yields very competitive parsing results. A somewhat similar approach was used in (Nivre & Scholz 2004) to develop a hybrid bottom-up/top-down approach; there, the edges are also labeled with semantic types, yielding lower accuracy than the works mentioned above. The overall goal of dependency parsing (DP) learning is to infer a tree structure. An example of a dependency tree is shown in Figure 1.
John
gives
me
a
beautiful
puppy .
Fig. 1: An example of a dependency tree A common way to do that is to predict with respect to each potential edge (i, j) in the tree, and then choose a global structure that (1) is a tree and that (2) maximizes some score. The first provides a set of structural constraints on the resulting structure, and the second provides a principled way to choose among (possibly many) resulting trees. In the context of DPs, this “edge based factorization method” was proposed by (Eisner 1996). In other contexts, this is similar to the approach of (Roth & Yih 2004) in that scoring each edge depends only on the raw data observed and not on the classifications of other edges, and that global considerations can be used to overwrite the local (edge-based) decisions. The key in a pipeline model is that making a decision with respect to the edge (i, j) may gain from taking into account decisions already made with respect to neighboring edges. However, given that these decisions are noisy, there is a need to devise policies for reducing the number of predictions in order to make the parser more robust. This is exemplified in (Yamada &
58
MING-WEI CHANG, QUANG DO & DAN ROTH
Matsumoto 2003)—a bottom-up approach, that is most related to the work presented here. Their model is a “traditional” pipeline model—a classifier suggests a decision that, once taken, determines the next action to be taken (as well as the input the next action observes). For many languages such as English, Chinese and Japanese (with a few exceptions), DPs without edge crossings, called projective dependency trees, are sufficient to analyze most sentences. Even for non-projective languages, usually the proportion of non-projective edges is low (Buchholz & Marsi 2006). As is common in most earlier work on DP, the work described here is also concerned mainly with projective trees. However, we also discuss an extension that deals with non-projective trees by converting them into projective trees (Nivre & Nilsson 2005). In the rest of this paper, we propose and justify a framework for improving pipeline processing based on the principles mentioned above: (i) make local decisions as reliably as possible, and (ii) reduce the number of decisions made. We use the proposed principles to examine the (Yamada & Matsumoto 2003) parsing algorithm and show that the framework suggests modifying some of the decisions made there and, consequently, results in better overall dependency trees. After introducing the task and our general approach, Section 2 formally defines the model we consider, the pipeline dependency parsing approach and its properties. Our experimental results on English are presented in Section 3. In Section 4 we discuss a few extensions to the model, and experimental results on other languages. Section 5 concludes and discusses some future directions. 2
Dependency parsing as a pipeline model
This section describes our dependency parsing algorithm and justifies its advantages by viewing and analyzing it as a pipeline model. Definition 1 For words x, y in a sentence T , we introduce the following notation: x → y : x is the direct parent of y. x →∗ y: x is an ancestor of y. x ↔ y : x → y or y → x. x < y : x is to the left of y in T . We now introduce the concept of a projective language (Nivre 2003): Definition 2 (Projective language) ∀a, b, c ∈ T, a ↔ b and a < c < b imply that a →∗ c or b →∗ c.
MULTILINGUAL DEPENDENCY PARSING
2.1
59
A pipeline dependency parsing algorithm
Our parsing algorithm is a modified shift-reduce parser (Aho et al. 1986) that applies a set of actions (described below) in a left to right manner on consecutive pairs of words (a, b) (a < b) in a sentence. A machine learning algorithm is used to determine what actions (namely, parsing decisions) to take between two candidate words in the sentence, and a tree is thus constructed in a bottom up manner. The basic actions used in this model, as in (Yamada & Matsumoto 2003), are: Shift: there is no relation between a and b, or the action is deferred because the child word of a and b still has some unlinked children. For example, if a is the parent of b and b has a child to the right of b, we must defer the action. Right: b is the parent of a (and a is thus eliminated from further consideration). Left: a is the parent of b (and b is thus eliminated from further consideration).
a
S X
R
L
b
a
b a
b
Fig. 2: This figure demonstrates the basic parsing actions: Left, Right and Shift These three actions are illustrated in Figure 2. It is important to note that one word (the child word) will be eliminated from T after performing Left or Right. We remove the child word because we already found its parent. This is a true pipeline approach in that (1) the classifiers are trained on individual decisions rather than on the overall quality of the parse tree, and are chained to yield a global structure; and consequently, decisions made with respect to a pair of words affect what pair of words is considered next by the algorithm. In order to complete the description of the algorithm we need to describe which edge to consider once an action is taken. We describe it via the notion of the focus point: Definition 3 (Focus point) When the algorithm considers the pair (a, b), a < b, we call the word a the current ‘focus point’. Next we describe several policies for determining the focus point of the algorithm following an action. We note that, with a few exceptions, determining
60
MING-WEI CHANG, QUANG DO & DAN ROTH
the focus point does not affect the correctness of the algorithm. It is easy to show that for (almost) any focus point chosen, if the correct action is selected for the corresponding word pair, the algorithm will eventually yield the correct tree (but may require multiple cycles through the sentence). In practice, the actions selected are noisy, and a wasteful focus point policy will result in a large number of actions, and thus in error accumulation. To minimize the number of actions taken, we want to find a good focus point placement policy. We always let the focus point move one word to the right after S. After L or R there are several natural placement policies to consider for the focus point: Start Over: Move focus point to the first word in T . Stay: Move focus point to be the next word to the right. That is, for T = (a, b, c), and the focus point is a, an L action results in a as the focus point (and the pair considers (a, c), while R action results in the focus being b (and the pair considered (b, c)). Step Back: The focus point moves to the previous word (on the left). That is, for T = (a, b, c), and the focus point is b, in both cases, a will be the focus point (and the pair considered are (a, c) in case of a R action, and (a, b) in case of a L). In practice, different placement policies have a significant effect on the number of pairs considered by the algorithm and, therefore, on the final accuracy2 . The following analysis justifies the Step Back policy. We claim that if Step Back is used, the algorithm will not waste any action. Thus, it achieves the goal of minimizing the number of actions taken in a pipeline algorithm. Notice that using this policy, when L is taken, the pair (a, b) is reconsidered, but with new information, since now it is known that c is the child of b. Although this seems wasteful, we will show in Lemma 1 that this movement is necessary in order to reduce the number of actions. As mentioned above, each of these policies yields the correct tree. Table 1 provides an experimental comparison of the three policies in terms of the number of actions required to build a tree. As a function of the focus point placement policy, and using the correct action for each pair considered, it shows the number of word pairs considered in the process of generating all the trees for section 23 of Penn Treebank. It is clear from Table 1 that the policies result in very different number of actions and that Step Back is the best choice. Note that since the actions are the gold-standard actions, 2
Note that Yamada & Matsumoto 2003) mention that they move the focus point back after R, but do not state what they do after executing L actions, and why. Yamada (p.c. 2006) indicates that they also moved the focus point back after L.
MULTILINGUAL DEPENDENCY PARSING
Policy Start over Stay Step back
#Shift #Left 156545 26351 117819 26351 43374 26351
61
#Right 27918 27918 27918
Table 1: The number of actions required to build all the trees for the sentences in section 23 of Penn Treebank (Marcus et al. 1993 ) as a function of the focus point placement policy. The statistics are taken with the correct (gold-standard ) actions the policy affects only the number of S actions used, and not the L and R actions, which are a direct function of the correct tree. The number of required actions in the test stage (when the actions taken are based on a learned classifier) shows the same trend and consequently the Step Back also gives the best dependency accuracy. The parsing algorithm with this policy is given Algorithm 2. Note that a good policy should always try to consider a child before considering its parent to save the number of decisions. As we will show later, the Step Back has several nice properties to be considered as a good policy. 2.2
Correctness and pipeline properties
We can prove several properties of our algorithm. First we show that the algorithm builds the dependency tree in only one pass over the sentence. Then, we show that the algorithm does not waste actions in the sense that it never considers a word pair twice in the same situation. Consequently, this shows that under the assumption of a perfect action predictor, our algorithm makes the smallest possible number of actions, among all algorithms that build a tree sequentially in one pass. Finally, we show that the algorithm only requires linear number of actions. Note that this may not be true if the action classifier is not perfect, and one can contrive examples in which an algorithm that makes several passes on a sentence can actually make fewer actions than a single pass algorithm. In practice, however, as our experimental data shows, this is unlikely. The following analysis is made under the assumption that the action classifiers are perfect. While this assumption does not hold in practice, it makes the following analysis tractable, and still gives a very good insight into the practical properties of our algorithm, given that, as we show the action classifier is quite accurate, even if not perfect. For example, in Lemma 1, we show that the algorithm requires only one round to parse a sentence under this assumption. In practice, in the evaluation stage, we allow the parsing algorithm to run multiple rounds. That is, when the focus point reaches the last word in T , we will set the
62
MING-WEI CHANG, QUANG DO & DAN ROTH
Algorithm 2 Dependency parsing with Step Back policy. getFeatures extracts the features describing the word pair currently considered; getAction determines the appropriate action for the pair; assignParent assigns a parent for the child word based on the action; and deleteWord deletes the child word in T at the focus once the action is taken. Note that if other policies are used, the algorithm needs to set f ocus = 0 after f ocus = T if the tree is not completed. t: a word token focus: index into the sentence T = {t1 , t2 , . . . , t|T | }: sentence f ocus = 1 while f ocus < |T | do ~v = getF eatures(tf ocus , tf ocus+1) α = getAction(tf ocus , tf ocus+1, ~v ) if α = L or α = R then assignP arent(tf ocus , tf ocus+1, α) deleteW ord(T, f ocus, α) // performing Step Back here if f ocus 6= 1 then f ocus = f ocus − 1 else if f ocus 6= 1 then f ocus = f ocus − 1 focus point at the beginning of the sentence if the tree is not complete. We found that over 99% of the sentences can be parsed in a single round and that, the average number of rounds needed to parse a sentence (average taken over all sentences in the test corpus) is 1.01. Therefore, we believe that the following analysis, done, under the assumption of golden actions can still be quite useful. Lemma 1 For projective languages, the dependency parsing algorithm that uses the Step Back policy completes the tree when it reaches the end of the sentence for the first time. That is, when the focus point is at the last word of T , the algorithm completes the tree. Before providing our argument, it is important to recall that one child word will be eliminated from T after performing Left or Right. The Step Back policy and this elimination procedure are the key points in this lemma. Note that in Algorithm 2, we use the focus point to choose the word pair. Since we move the focus point in the word vector T , we can only consider “consecutive” pairs in T . Note that the elimination after Left or Right will generate new “consecutive” word pairs in T .
MULTILINGUAL DEPENDENCY PARSING
63
In order to prove the lemma, we need the following definition. We call a pair of words (a, b) a free pair if and only if there is a relation between a and b and the algorithm can perform either a L or a R actions on that pair when it is considered. There are two requirement so that we can say that the algorithm can perform either L or R. First, the word pair must be consecutive in the currently considered T . Second, the action can be perform right now (we do not need to defer it). Formally, Definition 4 (Free pair) A pair (a, b) considered by the algorithm is a free pair, if it satisfies all the following conditions: 1. a ↔ b 2. a, b are consecutive in T (but not necessarily in the original sentence). 3. If a → b, no other word in T is the child of b. If b → a, no other word in T is the child of a. Proof: It is easy to see that there is at least one free pair in T , with |T | > 2. The reason is that if no such pair exists, there must be three words {a, b, c} s.t. a ↔ b, a < c < b and ¬(a → c ∨ b → c). However, this violates the properties of a projective language. Now assume {c, a, b} are three consecutive words in T . We claim that when using Step Back, the focus point is always to the left of all free pairs in T . This is clearly true when the algorithm starts. Assume that (a, b) is the first free pair in T and let c be just to the left of a and b. Then, the algorithm will not make a L or R action before the focus point meets (a, b), and will make one of these actions then. It’s possible that (c, a ∨ b) becomes a free pair after removing a or b in T so we need to move the focus point back. However, we also know that there is no free pair to the left of c. Therefore, during the algorithm, the focus point will always remain to the left of all free pairs. So, when we reach the end of the sentence, every free pair in the sentence has been taken care of, and the sentence has been completely parsed. 2 Lemma 2 All actions made by a dependency parsing algorithm that uses the Step Back policy are necessary. Specifically, a pair (a, b) will never be considered again unless there is an additional child linked to a or b. Proof: Note that if R or L is taken, either a or b will become a child word and be eliminated from further considerations by the algorithm. Therefore, if the action taken on (a, b) is R or L, this pair of words will never be considered again. If S is taken, it is possible to consider the word pair (a, b) again. From Lemma 1, the algorithm will complete the tree in one round and a pair (a, b)
64
MING-WEI CHANG, QUANG DO & DAN ROTH
will be considered again only if the focus point “steps back”. Therefore, we consider two cases here. First, if the next action followed S is L, b will get a new child and the algorithm will step back and consider (a, b) again. Since b gets new information, it is reasonable to check (a, b) again to see if the algorithm can recover the deferred action or not. Second, if the next action followed S is R, b will be eliminated from T and (a, b) will not be considered again. 2 The next lemma claims a stronger property: the algorithm requires only O(n) actions, where n is the length of the input sentence. Lemma 3 For a sentence T of size n, the number of actions used by a dependency parsing algorithm with the Step Back policy is bounded by 3n. Proof: In order to calculate the number of actions, we consider the number of word pairs processed by the algorithms, taking into account word pairs that may be processed multiple times. At the beginning of the algorithm, the algorithm only has n − 1 word pairs since it can only consider two consecutive words. Every time the algorithm performs an action other than S, some words will be eliminated and new word pairs will be generated. Furthermore, since a word gets new information (a new child), some word pairs may be considered again. From Lemma 2, we know that a word pair will be considered again only if one of the words has a new child connected to it. Therefore, we only need to calculate the number of newly generated word pairs and the number of word pairs needed to be reconsidered. Assume, w.l.o.g, that there are four words (a, b, c, d) and the focus point is at b, that is, the algorithm is considering (b, c). If the algorithm performs R, b becomes c’s child and (a, c) becomes a new word pair. We also need to consider (c, d) (again, if we considered it before) since now c gets more information. Similarly, if L is performed, we need to reconsider (a, b) for the new information and examine the new word pair (b, d). In conclusion, after performing every L or R action we may need to (re)consider two additional word pairs. Let m be the total number of word pairs which are needed to be considered. The previous discussion shows that we have: m ≤ n − 1 + 2l + 2r where l and r are the number of L and R actions taken, respectively. Note that l + r ≤ n − 1 since the sentence has only n words. Therefore, m ≤ 3n and the total number of actions is bounded by 3n. 2
MULTILINGUAL DEPENDENCY PARSING
2.3
65
Improving the parsing action set
So far we considered the “standard” set of three actions. While the L and R actions have a clear meaning, the S action covers several cases. When we use machine learning techniques to learn when to perform each action, this may result in an inaccurate decision with respect to this action. In order to improve the accuracy of the action predictors, we suggest a new (hierarchical) set of actions: Shift, Left, Right, WaitLeft, WaitRight. We believe that predicting these is easier due to finer granularity – the S action is broken into sub-actions in a natural way. The new actions have the following semantics: WaitLeft: a < b. a is the parent of b, but it’s possible that b is a parent of other nodes. Action is deferred. Notice that if we mistakenly perform Left instead, the child of b can not find its parents later. WaitRight: a < b. b is the parent of a, but it’s possible that a is a parent of other nodes. Similar to WL, action is deferred. Consequently, we also change the meaning of the action S to take effect only if there is no relationship between a and b3 . The new set of actions is shown to better support our parsing algorithm, when tested on different placement policies. When WaitLeft or WaitRight is performed, the focus will move to the next word, as in S. It is interesting to note that WaitRight is not needed in projective languages if Step Back is used. This gives another strong justification to use Step Back, since the action classification becomes more accurate—a more natural class of actions, with a smaller number of candidate actions. We show this property formally below. Lemma 4 If we apply the algorithm on projective languages, the WaitRight action is not necessary if the algorithm uses Step Back policy. Proof: Assume w.l.o.g. that the target sentence is (a, b, c) with b as the focus word, and that the algorithm needs to perform WaitRight. That it, c is the parent of b and b is the parent of other words (for example, a). This implies that there is a free pair to the left of the focus point, contradicting Lemma 1. Note that Lemma 1 still holds since WaitRight and WaitLeft behave as Shift. Therefore, the algorithm never needs to use the WaitRight action. 2 2.4
Improving the actions’ accuracy: A pipeline model with look ahead
Once the parsing algorithm, along with the focus point policy, is determined, it only remains to determine a way to choose an action. We do this using 3
Interestingly, Yamada & Matsumoto (2003) mention the possibility of an additional single Wait action, but it is not added it to the model considered there.
66
MING-WEI CHANG, QUANG DO & DAN ROTH
a machine learning algorithm. Given an annotated corpus we use of the parsing algorithm to determine the action needed for each consecutive pair; this is used to train a classifier to predict one of the possible actions. The details of the classifier and the feature used are given in Section 3. When the learned model is evaluated on new data, the sentence is processed left to right and the parsing algorithm, along with the action classifier, are used to produce the dependency tree. The parsing process is somewhat more involved, since the action classifier is not used as is, but rather via a look ahead inference step described next. The advantage of a pipeline model is that it can use more information, based on the outcomes of previous predictions. As discussed earlier, this may result in error accumulation. The importance of having a reliable action predictor in a pipeline model motivates the following approach. We devise a look ahead algorithm and use this policy as a way to determine the predicted action more accurately. This approach can be used in any pipeline model but we illustrate it below in the context of our dependency parser. The following example illustrates a situation in which an early mistake in predicting an action causes a chain reaction and results in further mistakes. This stresses the importance of correct early decisions, and motivates our look ahead policy.
W
W
X
Y
Y
Z
Z
X
Fig. 3: Top: correct dependency relations between w, x, y and z. Bottom: if the algorithm mistakenly decides that x is a child of w before deciding that y and z are x’s children, we cannot find the correct parent for y and z Let (w, x, y, z) be a sentence of four words, and assume that the correct dependency relations are as shown in the top part of Figure 3. If the system mistakenly predicts that x is a child of w before y and z becomes x’s children, we can only consider the relationship between w and y in the next stage. Consequently, we will never find the correct parent for y and z. The previous prediction error propagates and impacts future predictions. On the other hand, if the algorithm makes a correct prediction, in the next
MULTILINGUAL DEPENDENCY PARSING
67
Algorithm 3 Look ahead algorithm. y represents an action sequence. The function search considers all possible action sequences with |depth| actions and returns the sequence with the highest score. Algo predictAction(Model, Depth, State) x = getNextFeature(State) y = search(x, Depth, Model, State) lab = y[1] State = update(State, lab) return lab Algo search(x, Depth, Model, State) maxScore = −∞ F = {y | kyk = Depth} for y in F do s = 0, TmpState = State for i = 1 . . . Depth do x = getNextFeature(TmpState) s = s+ score(y[i], x, Model ) TmpState = update(TmpState, y[i]) if s > maxScore then ˆ=y y maxScore = s ˆ return y stage, we do not need to consider w and y. As shown, getting useful rather than misleading information in a pipeline model, requires correct early predictions. Therefore, it is advised to utilize some inference framework that may help minimizing the error accumulation problem. In order to improve the accuracy of the action prediction, we might want to examine all possible combinations of action sequences and choose the one that maximizes some score. It is clearly intractable to find the global optimal prediction sequences in a pipeline model of the depth we consider. Therefore, we use a look ahead strategy, implemented via a local search framework, which uses additional information but is still tractable. The look-ahead search algorithm is presented in Algorithm 3. The algorithm accepts three parameters, Model, Depth and State: Model : is our learning model—the classifier that is used to predict the action. We assume a classifier that can give a confidence in its prediction (see Section 3). Depth: The parameter Depth determines the depth of the search procedure.
68
MING-WEI CHANG, QUANG DO & DAN ROTH
State: encodes the configuration of the environment (in the context of the dependency parsing this includes the sentence, the focus point and the current parent and children for each word). Note that State changes when a prediction is made and that the features extracted for the action classifier also depend on State. The search algorithm will perform a search of length Depth. Additive scoring is used to score the sequence, and the first action in this sequence is selected and performed. Then, the State is updated, the new features for the action classifiers are computed and search is called again. One interesting property of this framework is that it allows the use of future information in addition to past information. The pipeline model naturally allows access to all the past information. Since the algorithm uses a look ahead policy, it also uses future predictions. The significance of this becomes clear in Section 3. There are several parameters, in addition to Depth that can be used to improve the efficiency of the framework. For example, given that the action predictor is a multi-class classifier, we do not need to consider all future possibilities in order to determine the current action. In our experiments, we only consider two actions with highest score at each level (which was shown to produce almost the same accuracy as considering all four actions). Note that our look ahead search is a local search so the depth is usually small. Furthermore, we can keep a cache of the search paths of the look-ahead search for the i-th action. These paths can be reused in the look-ahead search for the (i + 1)-th action. Therefore, our look ahead search is not very expensive. 3
Experimental study
In this section, we describe a set of experiments done to investigate the properties of our algorithm and evaluate it. All the experiments here were done on English. Results on other languages are described in Section 4. We use the standard corpus for this task, the Penn Treebank (Marcus et al. 1993). The training set consists of sections 02 to 21 and the test set is section 23. The pos tags for the evaluation data sets were provided by the tagger of (Toutanova et al. 2003) (which has an accuracy of 97.2% on section 23 of the Penn Treebank). 3.1
Learning algorithm
As our learning algorithm we use a regularized variation of the perceptron update rule, as incorporated in SNoW (Roth 1998, Carlson et al. 1999)4, a multi-class classifier that is tailored for large scale learning tasks and has 4
The package can be downloaded from http://L2R.cs.uiuc.edu/∼cogcomp/software.php
MULTILINGUAL DEPENDENCY PARSING
69
been used successfully in a large number of NLP tasks, e.g., (Punyakanok et al. 2005). SNoW uses softmax over the raw activation values as its confidence measure, which can be shown to produce a reliable approximation of the labels’ conditional probabilities. The softmax score can be written as follows: eactlab score(x, lab) = pi = P acti 1≤i≤n e
where acti means the normalized activation value for i-th class of feature vector x. The activation value is directly come from our machine learning model. 3.2
Features
For each word pair (w1 , w2 ) we use the words, their pos tags and also these features of the children of w1 and w2 . We also include the lexicon and pos tags of 2 words before w1 and 4 words after w2 (as in (Yamada & Matsumoto 2003)). The key additional feature we use, relative to (Yamada & Matsumoto 2003), is that we include the previous predicted action as a feature. We also add conjunctions of above features to ensure expressiveness of the model. Yamada & Matsumoto (2003) make use of polynomial kernels of degree 2 which is equivalent to using even more conjunctive features (Cumby & Roth 2003). Overall, the average number of active features in an example is about 50. 3.3
Evaluation methodology
We use the same evaluation metrics as in (Mcdonald et al. 2005). Dependency accuracy is the proportion of non-root words that are assigned the correct head. Sentence accuracy indicates the fraction of sentences that have a complete correct analysis. We also measure the root accuracy and leaf accuracy, as in (Yamada & Matsumoto 2003). When evaluating the result, we exclude the punctuation marks as done in (Mcdonald et al. 2005) and (Yamada & Matsumoto 2003). The punctuation include comma, colon, dot, quotation, and double quotation. All other symbols are scoring tokens. 3.4
Results
We first present the results of several of the experiments that were intended to help us analyze and understand some of the design decisions in our pipeline algorithm. To see the effect of the additional action, we present in Table 2 a comparison between a system that does not have the WaitLeft action (similar to the Yamada & Matsumoto’s (2003) approach) with one that does. For
70
MING-WEI CHANG, QUANG DO & DAN ROTH
Evaluation metric Dependencies Root Sentence Leaf
Action set w/o WaitLeft w/ WaitLeft 90.27 90.53 90.73 90.76 39.28 39.74 93.87 93.94
Table 2: The significance of the action WaitLeft a fair comparison, in both cases, we do not use the look ahead procedure. Note that, as stated above, the action WaitRight is never needed for our parsing algorithm. It is clear that adding WaitLeft increases the accuracy significantly. Evaluation metric Dependencies Root Sentence Leaf
Depth=1 90.53 90.76 39.74 93.94
Depth=2 90.67 91.51 40.23 93.96
Depth=3 90.69 92.05 40.52 93.94
Depth=4 90.79 92.26 40.68 93.95
Table 3: The effect of different Depth settings. Increasing the Depth usually improves the accuracy Table 3 investigates the effect of the look ahead, and presents results with different Depth parameters (Depth= 1 means “no search”), showing a consistent trend of improvement. Table 4 breaks down the results as a function of the sentence length; it is especially noticeable that the system also performs very well for long sentences, another indication for its global performance robustness. Evaluation metric Dependencies Root Sentence Leaf
<11 93.4 96.7 85.2 94.6
Sentence length 11-20 21-30 31-40 92.4 90.4 90.4 93.7 91.8 89.8 56.1 32.5 16.8 94.7 93.4 94.0
> 40 89.7 87.9 8.7 93.3
Table 4: The effect of sentence length (Depth = 4). Note that our algorithm performs very well on long sentences Table 5 shows the results with three settings of the pos tagger. The best result is, naturally, when we use the gold standard also in testing. However, it is worthwhile noticing that it is better to train with the same pos tagger available in testing, even if its performance is somewhat lower. Finally, Table 6 compares the performances of several of the state of the art dependency parsing systems with ours. When comparing with other dependency parsing systems it is especially worth noticing that our system gives significantly better accuracy on completely parsed sentences.
MULTILINGUAL DEPENDENCY PARSING
Evaluation metric Dependencies Root Sentence Leaf
71
Sources of tags (train-test) gold−pos pos−pos gold−gold 90.7 90.8 92.0 92.0 92.3 93.9 40.8 40.7 43.6 93.8 94.0 95.0
Table 5: Comparing different sources of POS tagging in a pipeline model. We use “gold” if true tags are applied and we use “pos” if tags are generated by a trained tagger (Depth= 4 for all experiments in this table). It is better to use the POS used in the evaluation also when training Evaluation metric Dependencies Root Sentence Leaf
Dependency parsing systems Y&M03 N&S04 M&C&P05 Current 90.3 87.3 90.9 90.8 91.6 84.3 94.2 92.3 38.4 30.4 37.5 40.7 93.5 n/a n/a 94.0
Table 6: Comparison between the current work and other dependency parsers. Our system performs especially well at the sentence level 4
Extensions: Non-projective trees and edge labels
Some languages such as Czech are very different from English. In this section, we address two major difficulties one needs to deal with in order to extend the approach we described here to other languages. First, the dependency graph in some languages may have multiple roots. In this case, we need to refer to the dependency edges of a sentence as a dependency graph. Second, some languages are non-projective . We discuss below how to overcome these two difficulties within the approach described in this work. Fortunately, the multi-root problem is not difficult as it might seem. In fact, the algorithm described earlier can handle multiple roots problem if the edges are projective. Assume the dependency graph of a sentence has multiple roots and that the language is projective. Then, we can view the sentence as a union of several small sentences with no dependencies among them, where each of them has a single-rooted dependency tree. The algorithm can generate multiple trees by performing Shift between root words. In order to deal with crossing edges we can use one of two possible approaches. The first one is to introduce some additional actions to handle the long distance relationships. The new actions, along with S, R and L, could handle most of the non-projective trees efficiently (Attardi 2006). Another approach is to convert non-projective trees into projective ones (Nivre & Nilsson 2005). In this article, we adopt the second method, as
72
MING-WEI CHANG, QUANG DO & DAN ROTH
described below. Any projective dependency tree can be mapped into a projective tree by using the Lift operation, which is defined as follows: Lift(wj → wk ) = parent(wj ) → wk , where a → b means that a is the parent of b, and parent is a function which returns the parent word of the given word. The procedure is as follows. First, the mapping algorithm examines if there is a crossing edge in the current tree. If there is a crossing edge, it will perform repeated Lift operations and replace this edge until the tree becomes projective. Let E be the dependency graph of a sentence. The algorithm for converting E into a projective graph is as follows (Nivre & Nilsson 2005): 1. If E is projective, exit. Otherwise, 2. Choose the smallest non-projective edge e, and 3. Let e′ be the new edge after performing Lift on e. 4. Let E ← E − {e} ∪ {e′ }. Goto 1. After we convert every training non-projective sentence in the training data, we train the model as before, on the modified data. In the evaluation phase, we directly apply the trained model to predict the dependency, resulting in projective graphs. In this paper, we do not try to recover the information lost during the mapping procedure. 4.1
Dealing with dependency labels
It is often desirable to predict also the dependency label (type) corresponding to the edges in the parse tree. Dependency labels can be very informative and include, in English, labels such as the “object” of a verb or the “subject” of a verb. In this paper, we evaluated a simple two-stage framework to predict the dependency labels. We view labeling the type of the dependencies as a post-task after the phase of predicting the head for each token in the sentence. Thus, it is a multi-class classification task. The number of the dependency types for each language can be found in (Buchholz & Marsi 2006). In the phase of learning dependency types, the parent of each token, which was labeled in the first phase, will be used among the features. The predicted actions can therefore help us to make accurate predictions of the dependency types. 4.2
Experimenting with multilingual dependency parsing
In this section, we report the result of applying our system, along with the simple extensions described above in conll-x shared task (Buchholz & Marsi 2006). The target dataset consists of data in 12 languages (English is not included), listed in Table 7. The resources provided for the 12 languages are described in (Hajiˇc et al. 2004, Chen et al. 2003, B¨ohmov´a et al. 2003,
73
MULTILINGUAL DEPENDENCY PARSING
Kromann 2003, van der Beek et al. 2002, Brants et al. 2002, Kawata & Bartels 2000, Afonso et al. 2002, Dˇzeroski et al. 2006, Civit Torruella & Mart´ı Anton´ın 2002, Nilsson et al. 2005, Oflazer et al. 2003, Atalay et al. 2003). We apply the techniques earlier in this section to extend our algorithm to on non-projective languages and to the prediction of both head words and dependency labels. Language Arabic Chinese Czech Danish Dutch German Japanese Portuguese Slovene Spanish Swedish Turkish
Ours 76.09 89.60 81.78 86.85 76.25 86.90 90.77 88.60 80.32 83.09 89.05 73.15
UAS AV 73.48 84.85 77.01 84.52 75.07 82.60 89.05 86.46 76.53 77.76 84.21 69.35
SD 4.94 5.99 6.70 8.97 5.78 6.73 5.20 4.17 4.67 7.81 5.45 5.51
Ours 60.92 85.05 72.88 80.60 72.91 84.17 89.07 83.99 69.52 79.72 82.31 60.51
LAS AV 59.94 78.32 67.17 78.31 70.73 78.58 85.86 80.63 65.16 73.52 76.44 55.95
SD 6.53 8.82 8.93 11.34 6.66 7.51 7.09 5.83 6.78 8.41 6.46 7.71
Table 7: Our results are compared with the average scores. UAS=Unlabeled Attachment Score, LAS=Labeled Attachment Score, AV=Average score, and SD=standard deviation We emphasize that no special language enhancement has been applied for any language in this experiment. We use the same features and the same parameters for all languages. In the dataset used, some languages have additional morphological information. The information is stored in the column feat. We incorporated the information of feat for the languages when it is available. The evaluation presented here is done using the script provided by conll-x and is thus somewhat different from the evaluation used in Section 3. Two kinds of evaluation metrics are used here: LAS =
number of tokens with correct head and correct labels number of tokens
and
number of tokens with correct head number of tokens Note that uas is the proportion of tokens with correct head. The only difference between uas and dependency accuracy is that uas does consider root words. UAS =
74 4.3
MING-WEI CHANG, QUANG DO & DAN ROTH
Multilingual results
Table 7 shows our results on uas and las. Our results are compared with the average scores (av) and the standard deviations (sd) of all the systems that took part in the shared task of conll-x. Our average uas for 12 languages is 83.54% with the standard deviation 6.01; and 76.80% with the standard deviation 9.43 for average las. Comparing our results to other systems that participated in the tasks (Buchholz & Marsi 2006), our system performs better than the average of systems participated in conll-x for every language. Perhaps surprising given that no tuning to any specific langauge was made. Method w/o WaitLeft w/ WaitLeft w/ WaitLeft + search
UAS 87.95 88.47 89.07
LAS 81.48 81.82 82.31
Table 8: An algorithmic analysis on a Swedish dataset (Depth = 3) To provide some more understanding in the context of other languages, we show next some specific cases. Table 8 shows the results on the Swedish dataset. We compare three systems. The first system used only the three standard actions, without look-ahead search. The second system used the improved action set. The best system used improved action set and lookahead search. Thus, we observe the same trends as we observed when running these experiments on the English dataset. 5
Conclusions and further work
We have addressed the problem of using learned classifiers in a pipeline fashion, where a task is decomposed into several stages and stage classifiers are used sequentially, with each stage using the outcome of previous stages as its input. This is a common computational strategy in natural language processing and is known to suffer from error accumulation and an inability to correct mistakes in previous stages. We abstracted two natural principles, one which calls for making the local classifiers used in the computation more reliable and a second, which suggests to devise the pipeline algorithm in such a way that minimizes the number of decisions (actions) made. In this work we studied this framework in the context of designing a bottom up dependency parsing. Not only we manage to use this framework to justify several design decisions, but we also showed experimentally that following these principles results in improving the accuracy of the inferred trees relative to existing models.
MULTILINGUAL DEPENDENCY PARSING
75
Interestingly, we can show that the trees produced by our algorithm are relatively good even for long sentences, and that our algorithm is doing especially well when evaluated globally, at a sentence level, where our results are significantly better than those of existing approaches—perhaps showing that the design goals were achieved. We also present the primarily results for the multilingual datasets provided in the conll-x shared task. This shows that dependency parsers based on a pipeline approach can handle many different languages. Recently, (McDonald & Pereira 2006) proposed using “second-order” feature information when applying the Einser’s algorithm and a global reranking algorithm. The proposed method outperform (Mcdonald et al. 2005) and the approach proposed in this paper. Although there is more room for comparison, we believe the the key advantage there is simply using a better feature set. Overall, the bottom line from this work, as well as from the conll-x shared task is that a pipeline approach is competitive with a global reranking approach (Buchholz & Marsi 2006). Beyond enhancing our current results by introducing more and better features, we intend to consider incorporating more information sources (e.g., shallow parsing results) into the pipeline process, to investigate different search algorithms in the pipeline framework and to study approaches to predicts dependencies and their type as a single task. Acknowledgements. We are grateful to Ryan McDonald and the organizers of conll-x for providing the annotated data set and to Vasin Punyakanok for useful comments and suggestions. This research was supported by the Advanced Research and Development Activity (arda)’s Advanced Question Answering for Intelligence (aquaint) Program and a doi grant under the Reflex program. REFERENCES Abeill´e, A. ed. 2003. Treebanks: Building and Using Parsed Corpora. (= Text, Speech and Language Technology, 20). Dordrecht: Kluwer Academic. Afonso, S., E. Bick, R. Haber & D. Santos. 2002. “Floresta sint´ a(c)tica: A treebank for Portuguese”. 3rd Int. Conf. on Language Resources and Evaluation (LREC ), 1698-1703. Las Palmas, Canary Islands, Spain. Atalay, N. B., K. Oflazer & B.Say. 2003. “The Annotation Process in the Turkish Treebank”. 4th Int. Workshop on Linguistically Interpreteted Corpora (LINC ). Budapest, Hungary. Aho, A.V., R. Sethi & J. D. Ullman. 1986. Compilers: Principles, Techniques, and Tools. Reading, Mass.: Addison-Wesley. Attardi, Giuseppe. 2006. “Experiments with a Multilanguage Non-projective Dependency Parser”. 10th Conf. on Computational Natural Language Learning (CoNLL-X ), 166-170. New York.
76
MING-WEI CHANG, QUANG DO & DAN ROTH
Brants, S., S. Dipper, S. Hansen, W. Lezius & G. Smith. 2002. “The TIGER Treebank”. 1st Workshop on Treebanks and Linguistic Theories (TLT ). Sozopol, Bulgaria. B¨ohmov´ a, A., J. Hajiˇc, E. Hajiˇcov´ a & B. Hladk´ a. “The PDT: A 3-level Annotation Scenario”. In (Abeill´e 2003), ch. 7. Buchholz, Sabine & Erwin Marsi. 2006. “Conll-X Shared Task on Multilingual Dependency Parsing”. 10th Conference on Computational Natural Language Learning (CoNLL-X ), 149-164. New York. Carlson, A., C. Cumby, J. Rosen & D. Roth. 1999. “The SNoW learning architecture”. Technical Report UIUCDCS-R-99-2101. Urbana, Illinois: University of Illinois at Urbana-Champaign (UIUC) Computer Science Department. Chang, M.-W., Q. Do & D. Roth. 2006a. “A Pipeline Framework for Dependency Parsing”. Annual Meeting of the Association for Computational Linguistics (ACL), 65-72. Sydney, Australia. Chang, M.-W., Q. Do & D. Roth. 2006b. “A Pipeline Model for Bottom-up Dependency Parsing”. 10th Conf. on Computational Natural Language Learning (CoNLL-X ), 186-190. New York City. Chen, K., C. Luo, M. Chang, F. Chen, C. Chen, C. Huang & Z. Gao. 2003. “Sinica Treebank: Design Criteria, Representational Issues and Implementation”. In (Abeill´e 2003), ch. 13, 231-248. M. Civit Torruella & Ma A. Mart´ı Anton´ın. 2002. “Design Principles for a Spanish Treebank”. 1st Workshop on Treebanks and Linguistic Theories (TLT ). Sozopol, Bulgaria. Cumby, C. & D.Roth. 2003. “On Kernel Methods for Relational Learning”. Int. Conf. on Machine Learning (ICML), 107-114. Washington, DC, USA.. ˇ ˇ Dˇzeroski, S., T. Erjavec, N. Ledinek, P. Pajas, Z. Zabokrtsky & A. Zele. 2006. “Towards a Slovene Dependency Treebank”. 5th Int. Conf. on Language Resources and Evaluation (LREC ).Geneva, Switzerland. Eisner, J. 1996. “Three New Probabilistic Models for Dependency Parsing: An Exploration”. Int. Conf. on Computational Linguistics (COLING), 340-345. Copenhagen, Denmark. Haghighi, A., A. Ng & C. Manning. 2006. “Robust Textual Inference via Graph Matching”. Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, 387-394. Vancouver, Canada. ˇ Hajiˇc, J., O. Smrˇz, P. Zem´ anek, J. Snaidauf & E. Beˇska. 2004. “Prague Arabic Dependency Treebank: Development in Data and Tools”. NEMLAR Int. Conf. on Arabic Language Resources and Tools, 110-117. Cairo, Egypt. Kawata, Y. & J. Bartels. 2000. “Stylebook for the Japanese Treebank in VERBMOBIL”. Verbmobil-Report 240. T¨ ubingen, Germany: Seminar f¨ ur Sprachwissenschaft, Universit¨ at T¨ ubingen.
MULTILINGUAL DEPENDENCY PARSING
77
Kromann, Matthias Trautner. 2003. “The Danish Dependency Treebank and the DTAG Treebank Tool”. 2nd Workshop on Treebanks and Linguistic Theories (TLT ) ed. by Joakim Nivre & Erhard Hinrichs, 217-220. V¨ axj¨ o, Sweden. Lin, D. 1994. “Principar—An Efficient, Broad-coverage, Principle-based Parser”. Int. Conf. on Computational Linguistics (COLING), 482-488. Kyoto, Japan. McDonald, R., K. Crammer & F. Pereira. 2005. “Online Large-margin Training of Dependency Parsers”. Annual Meeting of the Association for Computational Linguistics (ACL’05 ), 91-98. Ann Arbor, Michigan. McDonald, R. & F. Pereira. 2006. “Online Learning of Approximate Dependency Parsing Algorithms”. 11th Conf. of the European Chapter of the Association for Computational Linguistics (EACL’06 ), 81-88. Trento, Italy. Marciniak, T. & M. Strube. 2005. “Beyond the Pipeline: Discrete Optimization in NLP”. 9th Conference on Computational Natural Language Learning (CoNLL-2005 ), 136-143. Ann Arbor, Michigan. Marcus, M. P., B. Santorini & M. Marcinkiewicz. 1993. “Building a Large Annotated Corpus of English: The Penn Treebank”. Computational Linguistics 19:2.313-330. Nilsson, J., J. Hall & J. Nivre. 2005. “MAMBA Meets TIGER: Reconstructing a Swedish Treebank from Antiquity”. 15th Nordic Conference of Computational Linguistics (NODALIDA) Special Session on Treebanks. Copenhagen Studies in Language 32, 119-132. Copenhagen, Denmark. Nivre, Joakim. 2003. “An Efficient Algorithm for Projective Dependency Parsing”. Int. Workshop on Parsing Technology (IWPT ), 149-160. Nancy, France. Nivre, J. & J. Nilsson. 2005. “Pseudo-projective Dependency Parsing”. 43rd Annual Meeting of the Association for Computational Linguistics (ACL), 99-106. Ann Arbor, Michigan. Nivre, J. & M. Scholz. 2004. “Deterministic Dependency Parsing of English Text”. 20th Int. Conference on Computational Linguistics (COLING-04 ), 64-70. Geneva, Switzerland. Oflazer, K., B. Say, D. Zeynep Hakkani-T¨ ur & G. T¨ ur. 2003. “Building a Turkish Treebank”. In (Abeill´e 2003), ch. 15. Punyakanok, V., D. Roth & W. Yih. 2005. “The Necessity of Syntactic Parsing for Semantic Role Labeling”. Int. Joint Conference on Artificial Intelligence (IJCAI ), 1117-1123. Edinburgh, Scotland, UK. Ratnaparkhi, Adwait. 1997. “A Linear Observed Time Statistical Parser Based on Maximum Entropy Models”. 2nd Conf. on Empirical Methods in Natural Language Processing (EMNLP ), 1-10. Somerset, New Jersey. Roth, Dan. 1998. “Learning to Resolve Natural Language Ambiguities: A Unified Approach”. National Conference on Artificial Intelligence (AAAI ), 806813. Montr´eal, Qu´ebec, Canada.
78
MING-WEI CHANG, QUANG DO & DAN ROTH
Roth, D. & W. Yih. 2004. “A Linear Programming Formulation for Global Inference in Natural Language Tasks”. Annual Conf. on Computational Natural Language Learning (CoNLL) ed. by Hwee Tou Ng & Ellen Riloff, 1-8. Boston, Mass. Toutanova, K., D. Klein & C. Manning. 2003. “Feature-rich Part-of-speech Tagging with a Cyclic Dependency Network”. Human Language Technology Conference (HLT-NAACL), 173-180. Edmonton, Canada. van der Beek, L., G. Bouma, R. Malouf & G. van Noord. 2002. “The Alpino Dependency Treebank”. Computational Linguistics in the Netherlands (CLIN ). Groningen, The Netherlands. Yamada, H. & Y.Matsumoto. 2003. “Statistical Dependency Analysis with Support Vector Machines”. Int. Workshop on Parsing Technologies (IWPT ), 195-206. Nancy, France.
How Does Treebank Annotation Influence Parsing? Or How Not to Compare Apples and Oranges ¨bler Sandra Ku SfS-CL, University of T¨ ubingen Abstract In this paper, we will investigate the influence which different decisions in the annotation schemes of treebanks have on parsing. The investigation uses the comparison of similar treebanks of German, Negra and T¨ uBa-D/Z, which are subsequently modified to allow a comparison of the differences. The results show that some modification have a negative effect while others have a positive influence.
1
Introduction
In the last decade, the Penn treebank (Marcus et al. 1994) has become the standard data set for evaluating pcfg parsers. The fact that most parsers are solely evaluated on this specific data set leaves the question unanswered to what degree these results depend on the annotation scheme of the treebank. This point becomes more urgent in the light of more recent publications on parsing the Penn or the Negra treebank such as (Dubey & Keller 2003; Klein & Manning 2003), which show that parsing results can be improved if the parser is optimized for the input data. Klein & Manning (2003), e.g., gain more than 1 point in F-score when they mark each phrase that immediately dominates a verb, thus showing shortcomings in the annotation scheme. These findings raise the question whether such shortcomings in the annotation can be avoided during the design of the annotation scheme of a treebank. The question, however, can only be answered if we know which design decisions are favorable for pcfg parsing. In this chapter, we will investigate how different decisions in the annotation scheme influence parsing results. In order to answer this question, however, a method needs to be developed which allows the comparison of different annotation decisions without comparing unequal categories. For a comparison of different annotation schemes, one ideally needs one treebank with two different sets of (manual) annotations. An automatic conversion from one annotation scheme with flatter structures to another with deeper structures is impossible because of the high probability that systematic errors are introduced by the conversion. If no such doubly annotated text is available, two treebanks which are based on the same language and the same text genre and which are annotated with different annotation
¨ SANDRA KUBLER
80
schemes can be used. In the absence of more detailed methods of comparison, testing the effect of modifying individual annotation decisions gives insight into the factors that influence parsing results. However, the annotation schemes must be similar enough to enable a comparison. For English, no such treebanks based on the same text type are available. Only recently, a new pair of treebanks for German has become available, the Negra (Skut et al. 1998) and the T¨ uBa-D/Z (Telljohann et al. 2004) treebanks. Both treebanks are based on newspaper text, both use the stts pos tagset (Thielen & Schiller 1994), and both use an annotation scheme based on constituent structure augmented with grammatical functions. However, they differ in other respects, which makes them ideally suited for an investigation on how decisions in the design of an annotation scheme influence parsing accuracy. In section 2, we will describe the treebanks used in this investigation in more detail, section 3 describes the experimental setup and the method of comparison, and section 4 discusses the results of the comparison. S 505
HD
OC
MO
PP 500
AC
Im
NK
Rathaus−Foyer
0
APPRART
SB
NP 504
VP 503 MO
NN
PP 501 NK
AC
1
wird
2
VAFIN
neben APPR
3
dem ART
4
MO
NK
NK
MNR
HD
NK
Fund NN
PP 502 NK
AC
5
auch ADV
6
die
Forschungsgeschichte
ART
NN
7
8
zum
9
APPRART
Hochheimer ADJA
NK
10
Spiegel NN
11
präsentiert VVPP
12
.
13
$.
Fig. 1: A sample tree from the Negra treebank
2
The Negra and the T¨ uBa-D/Z treebanks
Both treebanks use German newspapers as their data source: the Frankfurter Rundschau newspaper for Negra and the ‘die tageszeitung’ (taz) newspaper for T¨ uBa-D/Z. Negra comprises 20 000 sentences, T¨ uBa-D/Z 15 000 sentences. Both treebanks use an annotation framework that is based on phrase structure grammar and that is enhanced by a level of predicateargument structure. Despite all these similarities, the treebank annotations differ in four important aspects: 1) Negra does not allow unary branching (nodes with one daughter) while T¨ uBa-D/Z does; 2) in Negra, phrases receive a flat annotation while T¨ uBa-D/Z uses phrase internal structure; 3) Negra uses crossing branches to represent long-distance relationships while T¨ uBa-D/Z uses a projective tree structure combined with functional labels; 4) Negra encodes grammatical functions in a combination of structural and functional labeling while T¨ uBa-D/Z uses a combination of topological fields (H¨ohle 1986) and functional labels, which results in a flatter structure on
81
HOW DOES TREEBANK ANNOTATION INFLUENCE PARSING? SIMPX 521
−
−
−
−
−
NF 520 OA−MOD
VF 518 ON
R−SIMPX 519 −
−
NX 516
HD
−
PX 509
−
NCX 501
Der
HD
0
ART
LK 510 HD
HD
NCX 500
−
Autokonvoi NN
−
1
−
MF 517
mit
2
APPR
den
VXFIN 502 HD
HD
3
ART
Probenbesuchern NN
4
fährt
5
VVFIN
MF 511 OA
NCX 503
−
eine ART
VC 504 VPT
HD
6
Straße NN
7
entlang PTKVZ
8
V−MOD
PRED
ADVX 513
EN−ADD 514 −
VC 515 HD
NCX 507 HD
VXFIN 508 HD
C 512 ON
−
NCX 505 HD
ADVX 506 HD
,
die
noch
$,
PRELS
ADV
9
10
HD
11
heute ADV
12
Lagerstraße NN
13
heißt
14
VVFIN
.
15
$.
Fig. 2: A sample tree from the T¨ uBa-D/Z treebank the clause level. The two treebanks also use different notions of grammatical functions: T¨ uBa-D/Z defines 36 grammatical functions covering head and non-head information, as well as subcategorization for complements and modifiers. Negra utilizes 48 grammatical functions. Apart from commonly accepted grammatical functions, such as SB (subject) or OA (accusative object), Negra grammatical functions also comprise a more extended notion, e.g. RE (repeated element) or RC (relative clause). Because of the differences in the definition of grammatical functions, a comparison of grammatical functions is only possible in a task-based evaluation within an application that uses these grammatical functions as input. Figure 1 shows a typical tree from the Negra treebank. The syntactic categories are shown in circular nodes, the grammatical functions as edge labels in square boxes. The PP “Im Rathaus-Foyer” (in the foyer of the town hall) and the NP “auch die Forschungsgeschichte zum Hochheimer Spiegel” (also the research history of the Hochheimer Spiegel) do not contain internal structure, the noun kernel elements are marked via the functional labels NK. The fronted PP is grouped under the VP, resulting in crossing branches. Figure 2 shows a typical example from T¨ uBa-D/Z. Here, the complex NP “Der Autokonvoi mit den Probenbesuchern” (the car convoy with the visitors of the rehearsal) contains an NP and the PP with internal NP, both NPs being explicitly annotated (as NX). The tree also contains several unary nodes, e.g. the verb phrases “f¨ahrt” (goes) and “heißt” (is called) or the street name “Lagerstraße”. The main ordering principle on the clause level are the topological fields, long-distance relationships such as the relation between the noun phrase “eine Straße” (a street) and the extraposed relative clause “die heute noch Lagerstraße heißt” (which is still called Lagerstraße) are marked via functional labeling; OA-MOD specifies that this clause modifies the accusative object OA.
82 3
¨ SANDRA KUBLER
Comparing treebanks for parsing
Both treebanks do not completely adhere to the requirements of a cfg: Apart from Negra’s crossing branches, which occur in 30% of the sentences, both treebanks contain sentences that consist of more than one tree. For all sentences, a virtual root node that groups all trees is inserted, and parenthetical trees are attached to the surrounding tree. In order to resolve Negra’s crossing branches, a script was used that is provided by Thorsten Brants. The script isolates crossing constituents and attaches the non-head constituents higher up in the trees. After the conversion, both PP modifiers of the VP in Figure 1 are reattached as daughters of S in order to resolve the crossing branches. Unfortunately, the modified tree does not contain any information on the scope of the modifiers, which has previously been shown by the low attachment in the VP. For the experiments, the statistical left-corner parser LoPar (Schmid 2000) was used. Since the experiments are designed to show differences in parsing quality depending on the annotation decisions, the parser was used without EM training or lexicalization of the grammar, and the parser was given the gold pos tags for the test sentences. For all the experiments reported here, only sentences with a length of maximally 40 words were used. These sentences were randomly split into 90% training data and 10% test data. This split was kept across experiments in order to enable error analysis. For each experiment, two different types of tests were performed: For one type, the data contained only syntactic constituents, i.e., the grammatical functions, which are shown as square boxes in the trees, were omitted. Thus, the rule describing the complex NP and its daughters in Figure 1 is represented as “NP → ADV ART NN PP”. These tests are reported below as “labeled precision” and “labeled recall”. In the second type of tests, the syntactic categories were augmented by their grammatical function. Thus, the same rule extracted from the tree in Figure 1 now contains the grammatical function for each node: “NP-SB → ADV-MO ART-NK NNNK PP-MNR”. These tests are reported below as “function labeled”. 3.1
First results and a methodology for comparing treebanks
The results of the experiments on the original treebanks after preprocessing are shown in Table 1. As reported above, Negra contains crossing branches in 30% of the sentences, which had to be resolved in preprocessing. The results show that the F-score for T¨ uBa-D/Z is significantly higher than for Negra trees. In contrast, the number of crossing brackets is lower for Negra. The Negra results raise the question whether the low crossing brackets rate
83
HOW DOES TREEBANK ANNOTATION INFLUENCE PARSING?
Negra 1.07 70.09% 67.82% 68.94 1.04 52.75% 51.85% 52.30
crossing brackets labeled recall labeled precision labeled F-score crossing brackets function labeled recall function labeled precision function labeled F-score
T¨ uBa-D/Z 2.27 84.15% 87.94% 86.00 1.93 73.65% 76.13% 74.87
Table 1: The results of comparing Negra and T¨ uBa-D/Z in Negra is only due to the low number of constituents in the trees. The percentage of nodes per words shows that while Negra trees contain on average 0.88 nodes per word, T¨ uBa-D/Z trees contain 2.38 nodes per word. These results raise the question whether the deeper structures in T¨ uBa-D/Z can be parsed reliably but may not be useful for further processing. SIMPX 507 −
−
−
MF 506 PRED
SB
S 501 MO
HD
MO
VF 503 ON
LK 504 HD
PP 500
NX 500
VXFIN 501 HD
AC
"
Ich
stehe
$(
PPER
VVFIN
0
1
2
ständig ADJD
3
unter
−
NK
4
APPR
Strom NN
5
HD
.
"
Das
$.
$(
ART
6
0
1
Ergebnis NN
2
ist
3
VAFIN
ADJX 505
−
HD
ADJX 502 HD
völlig
4
ADJD
offen
5
ADJD
Fig. 3: Negra (left) and T¨ uBa-D/Z (right) trees with one word phrases These inconclusive results necessitate a more detailed evaluation. A logical next step towards a more conclusive evaluation would be a comparison of single constituents and grammatical functions. Such a comparison, however, is only meaningful if both annotation schemes describe the same phenomena with the same categories. Unfortunately, for Negra and T¨ uBa-D/Z, this assumption often does not hold. The most obvious area in which the two treebanks differ is the treatment of unary nodes: while T¨ uBa-D/Z annotates such constituents, Negra does not allow unary branching. From the examples in Figure 3, it becomes clear that the differences are widespread and concern, amongst others, verb phrases, adverbial phrases, and prepositional phrases. Due to these significant differences, a comparison of single constituents is not meaningful since one would compare, for example, all NPs in T¨ uBa-D/Z to more complex NPs (with two words or more) in Negra.
.
6
$.
7
¨ SANDRA KUBLER
84 SIMPX 526
ON
HD
OA
VPT
OA−MOD
NX 522
−
HD
−
ON
PX 521
−
Der
0
ART
Autokonvoi NN
1
mit
−
2
APPR
den
3
Probenbesuchern NN
−
4
fährt
5
VVFIN
eine ART
MOD
HD
6
HD
MF 523
NX 525
HD
ART
R−SIMPX 524 −
Straße NN
7
entlang PTKVZ
8
,
die
noch
$,
PRELS
ADV
9
10
MOD
11
heute ADV
PRED
12
Lagerstraße NN
13
heißt
14
VVFIN
.
15
$.
Fig. 4: The sentence from Figure 2 flattened and without unary branches In the absence of more detailed methods of comparison, testing the effect of modifying individual annotation decisions gives insight into the factors that influence parsing results. As mentioned above, Negra and T¨ uBa-D/Z differ in three major points (the fourth difference, crossing branches in Negra, is already addressed in preprocessing): flatter phrases and no unary nodes in Negra, and flatter structures on the clause level in T¨ uBa-D/Z. In order to test the individual decisions, the opposite treebank is modified to also follow the respective decision. Consequently, the following modifications of the treebanks were executed: 1. To test the influence of not annotating unary nodes (such as in Negra), all nodes with only one daughter were removed from T¨ uBa-D/Z, preserving the grammatical functions. In the following section, this version is called T¨ u NU. 2. To test the influence of Negra’s flat phrase structure, phrases in T¨ uBaD/Z were flattened. This version is called T¨ u flat. 3. Finally, both modifications, removing unary nodes and flattening phrases, were applied to T¨ uBa-D/Z. The resulting tree for the sentence in Figure 2 is shown in Figure 4. This version is called T¨ u cmb. 4. In order to test the influence of the flatter T¨ uBa-D/Z structure on the clause level, topological fields were introduced into the Negra annotations. The topological fields were automatically extracted from the Negra corpus by the DFKI Saarbr¨ ucken. The conversion algorithm had to apply heuristics, which leads to a small number of errors in the field annotation. The original annotation of Negra had to be modified when the topological fields were introduced. In many cases, the topological fields cross phrasal boundaries: These phrasal nodes were removed. The resulting tree for the sentence in Figure 1 is shown in Figure 5. This version of Negra is called Ne field. For testing the effects of the modifications on parser performance, the same data split as in the baseline experiments was used. The results of these experiments are shown in Table 2.
85
HOW DOES TREEBANK ANNOTATION INFLUENCE PARSING? S 509
−−
−−
−−
−−
MF 508
MO
SB
NP 506
VF 505 MO
MO
PP 500
AC
Im
LK 501 HD
NK
Rathaus−Foyer
0
APPRART
NN
1
wird
2
VAFIN
PP 502 NK
AC
neben APPR
3
dem ART
4
NK
NK
NK
Fund NN
PP 503 NK
AC
5
auch ADV
6
die
Forschungsgeschichte
ART
NN
7
RK 507 OC
MNR
8
zum
9
APPRART
Hochheimer
VP 504 HD
NK
10
ADJA
Spiegel NN
11
präsentiert
12
VVPP
.
13
$.
Fig. 5: The sentence from Figure 1 with fields 4
Discussion of the results of the comparison
Table 2 gives the results for the evaluation of the two types of tests: the upper half of the table gives results for parsing with syntactic node labels only while the lower half of the table gives results for parsing syntactic categories and grammatical functions. The results show that all transformations approximate the other treebank in annotation as well as in parser performance. cross. br. lab. recall lab. precision lab. F-score % sent. n. parsed cross. br. func. lab. recall func. lab. prec. func. lab. F-score % sent. n. parsed nodes/words
Negra 1.07 70.09% 67.82% 68.94 0.55% 1.04 52.75% 51.85% 52.30 12.59% 0.88
Ne field 1.30 75.21% 77.17% 76.18 0.05% 1.21 69.85% 69.53% 69.19 2.17% 1.38
T¨ uBa 2.27 84.15% 87.94% 86.00 0.48% 1.93 73.65% 76.13% 74.87 1.03% 2.38
T¨ u NU 1.87 77.41% 81.52% 79.41 1.91% 2.17 62.11% 65.43% 63.73 9.98% 1.33
T¨ u flat 1.09 85.63% 86.24% 85.93 0.62% 1.07 73.80% 74.66% 74.23 3.55% 2.00
T¨ u cmb 1.15 77.43% 76.44% 76.93 2.26% 1.29 53.63% 58.87% 56.13 18.87% 1.06
Table 2: The results of comparing the modified versions of Negra and T¨ uBa-D/Z 4.1
Modification of Negra
The modification of Negra, which introduces topological fields to flatten the clause structure, leads to an improved F-score but also to more crossing brackets. A first hypothesis would be that the improvement is due to the reliable recognition of the new field nodes only. This hypothesis can be rejected by an evaluation of the parsing results for single syntactic categories. This evaluation shows that the introduction of topological fields gives high F-scores for the major fields, but it also improves both precision and recall for adverbial phrases, noun phrases, prepositional phrases, and almost all
86
¨ SANDRA KUBLER
types of coordinated phrases. For adjectival phrases, precision improves from 55.95% to 64.46% - but at the same time, recall degrades from 56.38% to 50.97%. In contrast, the F-score for verb phrases deteriorates. This is probably due to the fact that only verb phrases which do not cross field boundaries are left in the annotation. One reason for the improvement in the overall F-score is the change in the number of rules for a specific syntactic category. A look at the rules extracted from the training corpus shows a dramatic drop in numbers: for adjectival phrases, the number decreases from more than 3900 rules containing AP to approximately 3400 – even though new rules were added for the treatment of topological fields. 4.2
Modification of T¨ uBa-D/Z
Each modification of T¨ uBa-D/Z results in a loss in F-score, but also in an improvement concerning crossing brackets. While flattening phrase structure only leads to minor changes, deleting unary nodes has a detrimental effect: the F-score decreases from 86.00 to 79.41 when parsing syntactic constituents, and from 74.83 to 63.73 when parsing syntactic constituents including grammatical functions. Deleting unary nodes in the experiments with grammatical functions increased the number of sentences that could not be parsed by a factor of almost 10. These sentences would have required additional rules not present in the training sentences. This leads to the question whether the deterioration is only due to the high number of sentences which were assigned no parse. However, an evaluation of only those sentences that did receive a parse shows only slightly better results in recall (obviously, precision remains the same): 68.18% for parsed sentences as compared to 62.11% for all sentences. This result, however, may also be caused by missing rules, which is corroborated by a look at the rules extracted from the test sentences: Approximately 24.0% of the rules needed for correctly parsing the test sentences in the modification without unary nodes are not present in the training set, as compared to 18.2% in the original version of the T¨ uBa-D/Z treebank. A closer look at the different constituents shows that the syntactic categories that are affected most by the deletion of unary nodes are noun phrases, finite verb phrases, adjectival phrases, adverbial phrases, and infinitival verb phrases. All those categories suffer losses in the F-score between 1.81% (for infinitival verb phrases) and 57.28% (for adverbial phrases). Since both precision and recall are similarly affected, this means that the parser does not only annotate spurious phrases but also misses phrases which should be annotated.
HOW DOES TREEBANK ANNOTATION INFLUENCE PARSING?
87
Flattening phrases in T¨ uBa-D/Z has a negative effect on precision but it causes a slight increase in recall. The latter effect is a consequence of the bias of the pcfg parser, which prefers small trees. A comparison of the average number of nodes per word in a sentence shows that for all models, the parsed trees contain significantly fewer nodes than the gold standard trees. For the original T¨ uBa-D/Z grammar including grammatical functions, the parsed tree contains 54.6% of the nodes in the gold standard; in the flattened version, the ratio is 58.6% (and for Negra, it is 62.5%). The category that profits most from this modification is the category of named entities; other syntactic categories that profit from a flattening of the trees are prepositional phrases and relative clauses. All of them need information from deeper levels of the tree to be recognized correctly. The combination of both modifications in T¨ uBa-D/Z, flat phrase structure and deleted unary nodes, leads to a dramatic loss in the F-score for functional parsing as compared to the experiment in which only the unary nodes were deleted. A look at the unlabeled F-scores shows that this loss is not only due to incorrect labels for constituents, it also affects the recognition of phrase boundaries: the unlabeled F-score degrades from 91.34 for the original version of T¨ uba-D/Z, to 81.06 for the version without unary nodes, and to 71.65 for the combination of both modifications. 5
Conclusion and future work
We have presented a method for comparing different annotation schemes and their influence on pcfg parsing. It is impossible to compare the performance of a parser on single syntactic categories since even rather similar annotation schemes apply different definitions for different phrase types. As a consequence, the comparison must be based on modifications within one annotation scheme to make it more similar to the other. The experiments presented here show that annotating unary nodes and structured phrases improve parsing results. On the clause level, however, a flatter structure incorporating topological fields is helpful for German. The experiments presented here were conducted with a standard pcfg parser. The next logical step is to extend the comparison to different probabilistic parsers with different probability models and different biases. The parser by Klein & Manning (2003) uses extensions of the probability model as well as lexicalization, both of which were very successful for English. It is, however, unclear what the effect of these extensions is on German data. Especially with regard to lexicalization, the situation for German is unclear: While lexicalization improves results for English (cf. e.g. (Collins 1999)), experiments for German (Dubey & Keller 2003) show a detrimental effect of lexicalization for the Negra data. Thus, a comparison of treebank
88
¨ SANDRA KUBLER
annotation schemes based on lexicalization only makes sense if a method of lexicalization can be found for both annotation schemes that does not overly decrease performance. Acknowledgments. We are grateful to the DFKI Saarbr¨ ucken for providing the topological field annotation for Negra. We would like to thank Wolfgang Maier, Julia Trushkina, Holger Wunsch, and Tylman Ule for scripts to convert and evaluate the data, and Erhard Hinrichs, Tylman Ule, and Yannick Versley for fruitful discussions. REFERENCES Collins, Michael. 1999. Head-Driven Statistical Models for Natural Language Parsing, Ph.D. dissertation. University of Pennsylvania. Philadelphia. Dubey, Amit & Frank Keller. 2003. “Probabilistic Parsing for German using Sister-Head Dependencies”. 41st Annual Meeting of the Association for Computational Linguistics (ACL’2003 ), 96-103. Sapporo, Japan. H¨ohle, Tilman. 1986. “Der Begriff ‘Mittelfeld’, Anmerkungen u ¨ber die Theorie der topologischen Felder”. Akten des Siebten Internationalen Germanistenkongresses 1985, 329-340. G¨ ottingen, Germany. Klein, Dan & Christopher Manning. 2003. “Accurate Unlexicalized Parsing”. Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL’2003 ), 423-430. Sapporo, Japan. Marcus, Mitchell, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz & Britta Schasberger. 1994. “The Penn Treebank: Annotating Predicate Argument Structure”. Proceedings of the Human Language Technology Workshop (HLT’94 ), 114-119. Plainsboro, New Jersey. Schmid, Helmut. 2000. LoPar: Design and Implementation. Technical report. Universit¨ at Stuttgart, Germany. Skut, Wojciech, Thorsten Brants, Brigitte Krenn & Hans Uszkoreit. 1998. “A Linguistically Interpreted Corpus of German Newspaper Texts”. ESSLLI Workshop on Recent Advances in Corpus Annotation. Saarbr¨ ucken, Germany. Telljohann, Heike, Erhard Hinrichs & Sandra K¨ ubler. 2004. “The T¨ uBa-D/Z Treebank: Annotating German with a Context-Free Backbone”. Proceedings of of the Fourth International Conference on Language Resources and Evaluation (LREC’2004 ), 2229-2235. Lisbon, Portugal. Thielen, Christine & Anne Schiller. 1994. “Ein kleines und erweitertes Tagset f¨ urs Deutsche”. Lexikon & Text ed. by H. Feldweg & E. Hinrichs, 215-226. T¨ ubingen: Niemeyer.
The SenSem Project: Syntactico-Semantic Annotation of Sentences in Spanish ´ n∗∗∗ , Laura Alonso∗ , Joan Antoni Capilla∗∗ , Irene Castello ∗∗∗∗ ´ndez-Montraveta ´zquez∗∗ Ana Ferna & Gloria Va ∗
FaMAF, Univ. Nacional de C´ ordoba Dept. of English and Linguistics, Univ. de Lleida ∗∗∗ Dept. of General Linguistics, Univ. de Barcelona ∗∗∗∗ Dept. of English and German Philology, UAB ∗∗
Abstract This paper presents SenSem, a project that aims to systematize the behavior of verbs in Spanish at the lexical, syntactic and semantic level. As part of the project, two resources are being built: a corpus where sentences are associated to their syntactico-semantic interpretation and a lexicon where each verb sense is linked to the corresponding annotated examples in the corpus. We also discuss some tendencies that can be observed in the current state of development.
1
Introduction
The SenSem project aims to build a databank of Spanish verbs based on a lexicon that links each verb sense to a significant number of manually analyzed corpus examples. This databank will reflect the syntactic and semantic behavior of Spanish verbs in naturally occurring text. We analyze the 250 verbs that occur most frequently in Spanish. Annotation is carried out at three different levels: the verb as a lexical item, the constituents of the sentence and the sentence as a whole. The annotation process includes verb sense disambiguation, syntactic structure analysis (syntagmatic categories, including the annotation of the phrasal heads, and syntactic functions), interpretation of semantic roles and analysis of sentential semantics. It is precisely this last area of investigation which sets our project apart from others currently being carried out with Spanish (Subirats & Petruck 2003, Garc´ıa De Miguel & Comesa˜ na 2004). Abstracting from the analysis of a significant number of examples, the prototypical behavior of verb senses will be systematized and encoded in a lexicon. The description of verb senses will focus on their properties at the syntactico-semantic interface, and will include information like the list of frames in which a verb can possibly occur. In addition, selectional restrictions will be inferred from words marked as heads of the constituents. Finally, the usage of prepositions will be studied.
90
´ FERNANDEZ, ´ ´ ALONSO, CAPILLA, CASTELLON, VAZQUEZ
The conjunction of all this information will provide a very fine-grained description of the syntactico-semantic interface at sentence level, useful for applications that require an understanding of sentences beyond shallow parsing. In the fields of automatic understanding, semantic representation and automatic learning systems, a resource of this type is very valuable. In the rest of the paper we will describe the corpus annotation process in more detail and examples will be provided. Section 2 offers a general overview of other projects similar to SenSem. In section 3, the levels of annotation are discussed, and the process of annotation is described in section 4. We then proceed to present the results obtained to date and the current state of annotation, and we put forward some tentative conclusions obtained from the results of the annotation thus far. 2
Related work
As shown by Levin (1993) and others (Jones et al. 1994, Jones 1995, Kipper et al. 2000, Saint-Dizier 1999, V´azquez et al. 2000), syntax and semantics are highly interrelated. By describing the way linguistic layers inter-relate, we can provide better verb descriptions since generalizations from the lexicon that previously belonged to the grammar level of linguistic description can be established (lexicalist approach). Within the area of Computational Linguistics, it is common to deal with both fields independently (Grishman et al. 1994, Corley et al. 2001). In other cases, the relationship established between syntactic and semantic components is not fully exploited and only basic correlations are established (Dorr et al. 1998, McCarthy 2000). We believe this approach is interesting even though it does not take full advantage of the existing link between syntax and semantics. Furthermore, we think that in order to coherently characterize the syntactico-semantic interface, it is necessary to start by describing linguistic data from real language. Thus, a corpus annotated at syntactic and semantic levels plays a crucial role in acquiring this information appropriately. In recent years, a number of projects related to the syntactico-semantic annotation of corpora have been carried out. The length of the present paper does not allow us to consider them all here, but we will mention a few of the most significant ones. FrameNet (Johnson & Fillmore 2000) is a lexicographic resource that describes approximately 2,000 items, including verbs, nouns and adjectives that belong to diverse semantic domains (communication, cognition, perception, movement, space, time, transaction, etc.). Each lexical entry has examples from the British National Corpus, manually annotated with argument structure and, in some cases, also adjuncts.
THE SENSEM PROJECT
91
PropBank (Kingsbury & Palmer 2002, Kingsbury et al. 2002) is a project based on the manual semantic annotation of a subset of the Penn Treebank II (a corpus which is syntactically annotated). This project aims to identify predicate-argument relations. In contrast with FrameNet, the sentences to be annotated have not been pre-selected. Both FrameNet and Propbank work with the use of corpora, although their objectives are a bit different. In FrameNet, a corpus is used to find evidence about linguistic behavior and to associate examples to lexical entries, whereas in Propbank, the objective is to enrich a corpus that has been already annotated at a syntactic level so that it can be exploited in more ambitious nlp applications. For Spanish, only a few initiatives address the syntactico-semantic analysis of corpus. The DataBase ”Base de Datos Sint´acticos del Espa˜ nol Actual” (Mu˜ niz et al. 2003) provides the syntactic analysis of 160.000 sentences extracted from part of the arthus corpus of contemporary texts. Syntactic positions are currently being labeled with semantic roles (Garc´ıa de Miguel & Comesa˜ na 2004). FrameNet-Spanish (Subirats & Petruck 2003) is the application of the FrameNet methodology for Spanish. Its target is to develop semantic frames and lexical entries for this language. Each verb sense is associated to its possible combinations of participants, syntactic functions and phrase types, as attested in the corpus. The SenSem project provides a different approach to the description of verb behavior. In contrast with FrameNet, its aim is not to provide examples for a pre-existing lexicon, but to shape the lexicon with the corpus examples annotated. Another difference from the FrameNet approach is that the semantic roles we use are far more general, they are not related to syntactic functions, and are less class-dependent. Finally, to the best of our knowledge, no large-scale corpus annotation initiative associates semantics to sentence such as their aspectual interpretations or types of causativity. 3
Levels of annotation
As mentioned previously, we are describing verb behavior so only constituents directly related to the verb will be analyzed. Elements beyond the scope of the verb (i.e., extra-sentential elements such as logical linkers, some adverbs, etc.) are disregarded, as in the following example: (1) ... El presidente, que ayer inici´ o una visita oficial a la capital francesa, hizo estas declaraciones ... ... The president, who began an official visit to the French capital yesterday, stated ...
92
´ FERNANDEZ, ´ ´ ALONSO, CAPILLA, CASTELLON, VAZQUEZ
Were we annotating the verb iniciar –begin– we would ignore the participants of the main sentence and only take into account the elements within the clause. If we were annotating the verb hacer –make– we would annotate the subject to include the relative clause as a whole, without further analysis, and with the word “president” as the head of the whole structure. 3.1
Sentential semantics level
At this level, different aspects of sentential semantics are accounted for. With regard to aspectual information, a distinction is made among three types of meaning, event, process or state, as in the following examples: (2) event: ... El di´alogo acabar´ a hoy ... ... The conversations will finish today ... (3) process: ... cuando le preguntaron de qu´e hab´ıa vivido hasta aquel momento ... ... when he was asked what he had been living on until then ... (4) state: ... El gasto de personal se acerca a los 2.990 millones ... ... Personnel expenses come close to 2.990 million ... Apart from aspectual information, we also annotate sentential level meanings using labels like anticausative, antiagentive, impersonal, reflexive, reciprocal or habitual. This feature is useful to account for the variation in the syntactic realizations of the argument structures of each verb sense. For example, the following sentences with the verb abrir –open– present an agentive and antiagentive interpretation, respectively: (5) agentive: El alcalde de Calafell [...] abrir´a un expediente... The mayor of Calafell [...] will open administrative proceedings ... (6) antiagentive: el vertedero de Tivissa no se abrir´a sin consenso. Tivissa’s rubbish dump will not be opened without a consensus. 3.2
Lexical level
At the lexical level, each example of a verb is assigned a sense. We have developed a verb lexicon in which the possible senses for a verb are defined, together with its prototypical event structure and thematic grid, a list of synonyms and antonyms and related synsets in WordNet (Fellbaum 1998). Various lexicographic sources have been taken as references to build the inventory of senses for each verb, mainly the Diccionario de la Real Academia de la Lengua Espa˜ nola and the Diccionario Salamanca de la Lengua Espa˜ nola. Less frequent meanings are discarded, together with archaic and restricted uses. This inventory of senses for each verb is only preliminary, and can be modified whenever the examples found in the corpus indicate the existence of a distinct sense which has not been considered.
THE SENSEM PROJECT
3.3
93
Constituent level
Finally, at the constituent level, each participant in the clause is tagged with its constituent type (e.g., noun phrase, completive, prepositional phrase) and syntactic function (e.g., subject, direct object, prepositional object). Arguments and adjuncts are also distinguished. Arguments are defined as those participants that are part of the verb’s lexical semantics. Arguments are assigned a semantic role describing their relation with the verb (e.g., agent, theme, initiator, . . . ). The head of the phrase is also signaled in order to acquire selectional restrictions for that verb sense. Negative adverbs and pronouns are also marked. They have been considered relevant in that it may alter some other information in different levels.
4
Annotation process
The SenSem corpus will describe the 250 most frequent verbs Spanish. For each verb, 100 examples are extracted randomly from 13 million words of corpora obtained from the electronic version of the Spanish newspaper El Peri´ odico de Catalunya, disregarding uses of the verb as an auxiliary and idioms. The manual annotation of examples is carried out via a graphical interface, seen in Figure 1, where the three levels are clearly distinguished. The interface displays one sentence at a time. First, when a verb sense is selected from the list of possible senses, and its prototypical event structure and semantic roles are displayed. Then, constituents are identified and analyzed by selecting the words that belong to it; the head of the arguments and its possible metaphorical usage are also signaled. Finally, annotators specify any applicable semantics at clause level (e.g., anticausative, reflexive, stative, etc.). At the constituent level, the semantic role chosen for each phrase is often predictive of the other labels of that phrase: agents tend to be noun phrases with subject function, themes tend to be noun phrases with subject or object function (if they occur in passives, antiagentives, anticausatives or statives), etc. These associations are exploited to semi-automate the annotation process: once a role is selected, the category and function most frequently associated with it and its role as a verb argument are pre-selected so that the annotator only has to validate the information. The final corpus will be available to the linguistic community by means of a web-based interface (http://grial.uab.es/projectes/sensem).
94
´ FERNANDEZ, ´ ´ ALONSO, CAPILLA, CASTELLON, VAZQUEZ
Fig. 1: Screenshot of the annotation interface 5
Preliminary results of annotation
At this stage of the project1 , 77 verbs have already been annotated, which implies that the corpus at this moment is made up of 7,700 sentences (199,190 words). A total of 900 sentences out of these 7,700 have already been validated, which means that a corpus of approximately 25,000 words has already undergone the complete annotation process. 5.1
Data analysis
In this section we describe the information about verb behavior that can be extracted from the corpus in its present state. We have found that, out of the 199,190 words that have already been annotated, 182,303 are part of phrases which are an argument of the verb and 16,887 are adjuncts. With regard to aspectuality, there is a clear predominance of events 1
This paper was finished in the summer of 2005.
THE SENSEM PROJECT
95
(74.26% of the sentences) over processes (20.67%) and states (8.96%). This skewed distribution of clause types, with a clear predominance of events, may be exclusive to the journalistic genre. We have yet to investigate its distribution in other genres. As concerns syntactic functions, the most frequent category is direct object, covering 40% of the labels, with a significant difference in subjects (23%). This is not surprising if we take into account that Spanish is a prodrop language. Prepositional objects (13%) and indirect objects (2%) are less frequent. As for semantic roles, themes are predominant, as would be expected given that the most common syntactic function is that of direct object, and that there is a high presence of antiagentive, anticausative and passive constructions. Within the different types of the semantic role theme, unaffected themes (moved objects) appear most frequently. 5.2
Inter-annotator agreement
In order to measure inter-annotator agreement, four sentences of 59 verbs have been annotated by 4 different judges so that divergences in criteria could be found. These common sentences were used in the preliminary phase with the aim of both training the annotators and detecting points of disagreement among them. This comparison has helped us refine and settle the annotation guidelines and facilitate the revision of the corpus. In order to detect these problematic issues, we calculated inter-annotator agreement for all levels of annotation. An overview of the most representative values for annotator agreement can be seen in Table 1. We found pair wise proportions of overall agreement and the kappa coefficient (Cohen 1960), which gives an indication of stability and reproducibility of human judgments in corpus annotation. As a general remark, agreement is comparable to what is reported in similar projects. For example, Kingsbury et al. (2002) report agreement between 60% and 100% for predicate-argument tagging within Propbank, noting that agreement tends to increase as annotators are more trained. In SenSem, the level of annotation that is comparable to predicate-argument relations, semantic role annotation, is clearly within this 60%-100% range. It is noteworthy that the values obtained for the kappa coefficient are rather low. After a close inspection, we found that these low values of kappa are mainly due to the fact that the annotation guidelines were still not well-established, and that annotators were still under training. In contrast, the tagging of semantic roles presents perfect agreement for very infrequent roles (indirect cause, instrument, location). More frequent roles show a higher level of disagreement: initiators are significantly less
96
´ FERNANDEZ, ´ ´ ALONSO, CAPILLA, CASTELLON, VAZQUEZ
category agr. kappa category agr. eventual semantics argumentality event 66% .11 argument 82% state 90% .33 adjunct 64% process 76% .06 semantic role initiator 70% .37 agent 84% cause 91% .89 experiencer 97% theme 68% .43 affected theme 74% non-affected theme 70% .34 goal 79% syntactic function agentive complement 100% 1.00 subject 87% direct object 80% .63 indirect object 77% prepositional object 1 67% .65 prepositional object 2 66% prepositional object 3 78% .24 Circumstantial 62% Predicative 76% .16 syntactic category noun phrase 78% .67 prepositional phrase 72% adjectival phrase 88% .69 negative adverbial 100% adverbial phrase 77% .54 adverbial clause 68% gerund clause 72% .65 relative clause 82% completive clause 95% .93 direct speech 96% infinitive clause 94% .98 prep. completive clause 96% prep. infinitive clause 81% .57 personal pronoun 97% relative pronoun 98% .96 other pronouns 94%
kappa .54 .46 .81 .92 .55 .70 .83 .79 .28 .42
.53 1.00 .66 .16 .95 .44 .81 .82
Table 1: Inter-annotator agreement for a selection of annotated categories
clearly perceived than agents or causes (note differences in k agreement). It is also clear that fine-grained distinctions are more difficult to perceive than coarse-grained ones, as exemplified by low agreement between subdistinctions within the superclass of theme. Among syntactic functions, the agentive complement of passives presents perfect agreement. Agreement is also high for subjects, direct and indirect objects, but the distinction between different kinds of prepositional objects and circumstantial complements is not clearly perceived. Finally, agreement is rather high for some syntactic categories: pronouns, adverbs of negation, adjectival complements, completive clauses, infinitive clauses and direct speech present k > .7 and ratios of agreement over 90%. However, major categories present a rather high ratio of disagreement, as well as those categories that are mostly considered adjuncts. After the study of inter-annotator agreement, the guidelines for anno-
THE SENSEM PROJECT
97
tation have been settled (V´azquez et al. 2005). These guidelines serve as a reference for annotators, and we believe they will increase the overall consistency of the resulting corpus. 6
Conclusions and future work
The linguistic resource we have presented constitutes an important source of linguistic information useful in several natural language processing areas as well as in linguistic research. The fact that the corpus has been annotated at several levels increases its value and its versatility. The project is in its second year of development, with still a year and a half to go. During this time we intend to continue with the annotation process and to develop a lexical database that will reflect the information found in the corpus. We are aware that the guidelines established in the annotation process are going to bias, to a certain extent, the resulting resources, but nevertheless we believe that both corpus and lexicon are of interest for the nlp community. All tools developed in the project and the corpus and lexicon themselves will be available to the scientific community. REFERENCES Carletta, J., A.Isard, S.Isard, J.C.Kowtko, G.Doherty-Sneddon & A.H.Anderson. 1996. “HCRC Dialogue Structure Coding Manual”. HCRC Technical report HCRC/TR-82. Edinburgh, U.K.: Univ. of Edinburgh. Cohen, J. 1960. “A Coefficient of Agreement for Nominal Scales”. Educational & Psychological Measure, vol. 20, 37-46. Corley, S., M. Corley, F. Keller, M. W. Crocker, S. Trewin. 2001. “Finding Syntactic Structure in Unparsed Corpora” Computer and the Humanities vol. 35, 81-94. Dorr, B., M. A. Mart´ı, I. Castell´ on. 1998. “Spanish EuroWordNet and LCSBased Interlingual MT”. Proceeding of the ELRA Congress. Granada, Spain. http://citeseer.ist.psu.edu/dorr97spanish.html Fellbaum, C. 1998. “A Semantic Network of English Verbs”, WordNet: An Electronic Lexical Database ed. by C. Fellbaum, 69-104. Cambridge, Mass.: MIT Press. Garcia de Miguel, J.M. & S. Comesa˜ na. 2004. “Verbs of Cognition in Spanish: Constructional Schemas and Reference Points”, Linguagem, Cultura e Cogniao: Estudos de Lingu´ıstica Cognitiva ed. by A. Silva, A. Torres & M. Gon¸calves, 399-420. Coimbra: Almedina. Grishman, Ralph, C. Macleod & A. Meyers. 1994. “Comlex Syntax: Building a Computational Lexicon”. International Conference on Computational
98
´ FERNANDEZ, ´ ´ ALONSO, CAPILLA, CASTELLON, VAZQUEZ
Linguistics (COLING-1994 ), 268-272. Kyoto, Japan. Johnson, C. & C.J. Fillmore. 2000. “The FrameNet Tagset for Frame-Semantic and Syntactic Coding of Predicate-Argument Structure”. Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL 2000 ), 56-62. Seattle, Washington. Jones, D., ed. 1994. “Verb Classes and Alternations in Bangla, German, English and Korean”. Memo n. 1517. MIT, Artificial Intelligence Laboratory. Jones, D. 1995. “Predicting Semantics from Syntactic Cues—Evaluating Levin’s English Verb Classes and Alternations”. UMIACS TR-95-121, Univ. of Maryland. P. Kingsbury, P. & M. Palmer. 2002. “From Treebank to Propbank”. 3rd Conf. on Language Resources and Evaluation (LREC-2002 ). Las Palmas, Spain. http://citeseer.ist.psu.edu/kingsbury02from.html
Kingsbury, P., M. Palmer & M. Marcus. 2002. “Adding Semantic Annotation to the Penn TreeBank”. Human Language Technology Conference. San Diego, Calif. http://citeseer.ist.psu.edu/kingsbury02adding.html Kipper, K., H.T. Dang & M. Palmer. 2000. “Class-based Construction of a Verb Lexicon”. 17th National Conference on Artificial Intelligence (AAAI-2000 ), 691-696. Austin, Texas. Levin, B. 1993. English Verb Classes and Alternations: A Preliminary Investigation. Chicago, Illinois: The University of Chicago Press. McCarthy, D. 2000. “Using Semantic Preferences to Identify Verbal Participation in Role Switching Alternations”. Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL 2000 ), 256-263. Seattle, Washington. Mu˜ niz, E., M. Rebolledo, G. Rojo, M.P. Santalla & S. Sotelo. 2003. “Description and Exploitation of BDS: a Syntactic Database about Verb Government in Spanish”. Int. Conf. on Recent Advances in Natural Language Processing (RANLP-2003 ) ed. by G. Angelova, K. Bontcheva, R. Mitkov & N. Nicolov, 297-303. Borovets, Bulgaria. Saint-Dizier, P. 1999. “Alternations and Verb Semantic Classes for French: Analysis and Class formation”. Predicative Forms in Natural Languages and Lexical Knowledge Bases ed. by P. Saint-Dizier, 139-170. Dordrecht, The Netherlands: Kluwer. Subirats-R¨ uggeberg, C. & M.R.L. Petruck. 2003. “Surprise: Spanish FrameNet!”. Workshop on Frame Semantics, International Congress of Linguists. Prague. http://framenet.icsi.berkeley.edu/papers/SFNsurprise.pdf
V´azquez, G., A. Fern´ andez & M. A. Mart´ı. 2000. Clasificaci´ on verbal. Alternancias de di´ atesis. Lleida: Universitat de Lleida. V´azquez, G., A. Fern´ andez & L. Alonso. 2005. “Guidelines for the SyntacticoSemantic Annotation of a Corpus in Spanish”. Int. Conf. on Recent Advances in Natural Language Processing (RANLP-2005 ) ed. by G. Angelova, K. Bontcheva, R. Mitkov & N. Nicolov, 603-607. Borovets, Bulgaria.
Generating Referring Expressions: Past, Present and Future Robert Dale Macquarie University Abstract The generation of referring expressions—noun phrases which speakers use to identify intended referents for their hearers—is the most well-established and agreed-upon subtask in natural language generation. This paper surveys the developments in the field that have brought us to this point, and characterises the state of the art in the area; we also point to unsolved problems and new directions waiting to be pursued.
1
Introduction
Natural language generation (nlg) is concerned with getting machines to produce natural language output, either written or spoken, from some underlying non-linguistic input, which might be information contained in a database or a knowledge base of some kind.1 As a research area, nlg is important for both practical and theoretical reasons. • On the practical side, nlg is an important component of a number of potentially valuable applications. For example, we would like embodied conversational agents to use language appropriately on the basis of context, overcoming the limitations of canned output that we see in today’s dialogue applications; and there is great scope for information personalisation, where nlg permits the production of massive numbers of individually-tailored emails, documents or web pages that take into account the preferences or characteristics of the recipient. In areas like these, nlg offers the scope for solutons that would be infeasible if we had to rely on human authoring. • On the theoretical side, nlg permits—and even promotes—a focus on aspects of language that are generally ignored by work in natural language analysis: it allows us to explore the decision processes that lie behind subtle, and sometimes not so subtle, variations in output. So, for example, why does a speaker produce a passive sentence rather than the corresponding active declarative sentence, or some other syntactic structure that appears to convey the same information? What 1
In more recent years, there has been an increasing amount of work in what we might call text-to-text generation, where the input and output are both natural language texts; these approaches are not directly relevant to the topic discussed here.
100
ROBERT DALE
determines whether, in a conditional expression, we express the antecedent before the consequent or vice versa? Why give a five minute exposition on a topic rather than a thirty minute one, and how are the higher-level structures and the overall content of the possible variants determined? These are the kinds of questions that nlg attempts to answer, and which work in nlu tends to ignore: where nlu seeks to remove the subtle nuances and variations in language use in order to determine some core ‘canonical’ representation of meaning, nlg seeks to explain the variety in expression that is contingent on contextual factors. A component task that is implicated in almost any natural language generation system is that of referring expression generation. Any nlg system has to talk about things, and to do so it has to make decisions as to how to refer to those things. For many entities in the world—people, companies, cities, ships, newspapers, movies, and so on—this is a relatively straightforward process: these entities have proper names, and in such cases the requirement is for a decision-making process that can choose between the use of the entity’s full proper name, an abbreviated form if one is available, or a pronoun. But most entities in the world do not have proper names, and so if we want to refer to these entities, we have to do so in terms of properties they possess; here the range of possibilities is significantly greater, and the task correspondingly more complex. It is this process of determining how to identify an intended referent for a hearer that we call referring expression generation. The present paper aims to do three things. First, in Section 2, we provide a characterisation of the task of referring expression generation, situating it within the larger task of natural language generation as a whole. Then, in Section 3, we provide a brief survey of the work that has been done in the area to date, emphasising how most recent work has adopted a characterisation of the problem that is fairly widely agreed upon. In Section 4, we attempt to identify outstanding issues in the field and point to possible future research directions. The paper ends with some conclusions in Section 5. 2 2.1
What’s involved in referring expression generation The architecture of a natural language generator
Under a fairly common view of the architecture of the natural language generation process (see, for example, (Reiter & Dale 2000)), the generation of an output text, whether one sentence long or many paragraphs long, consists of three distinct stages:
GENERATING REFERRING EXPRESSIONS
101
Text planning is concerned with taking as input some high level communicative goal—for example, my desire to tell you what I did today— and deriving from this some structured plan for what has to be said to achieve this goal. Typically this plan is considered to be tree-like in nature, corresponding to the hierarchical decomposition of the original goal into subgoals, subgoals of those subgoals, and so on. So, for example, my plan to tell you about my activities today might be decomposed into the three subgoals of telling you what I did in the morning, what I did in the afternoon, and what I did in the evening; and each of these subgoals might in turn consist of lower-level goals, and so on recursively until some level of atomicity, perhaps corresponding to utterances that can be realised as single sentences or clauses, is reached. We might expect this tree structure to be labelled to indicate the rhetorical relations that hold between the constituent elements of the plan. So, for example, my goal to tell you about a particular event that occurred might be preceded by a goal to explain some relevant background information in order to allow you to understand the significance of the event, and the relationship between these two goals in the plan might be labelled as being one of Background. The terminal nodes or leaves of this plan then correspond to the propositions to be conveyed in the resulting output, each of these propositions being a semantic chunk that might correspond to a major clause in the language. These proposition-like elements are often referred to as messages. Micro-planning then uses these messages to build a sequence of sentence plans that determine the content to be realised in each sentence of the output. This may involve combining messages to build complex sentences, a process sometimes referred to as aggregation. So, for example, two propositions that might independently be realised as the sentences I went to the zoo and Mary went to the zoo can instead be combined to produce a more complex proposition that might be realised as the sentence Both Mary and I went to the zoo. More importantly for our purposes, the micro-planning stage is generally considered to be the place where we determine how the entities mentioned in the messages should be referred to. Up until this point the entities are represented by symbolic identifiers, such as person001, person002 and zoo167; in our present example, referring expression generation determines that person001, the speaker, should be identified by a first person singular pronominal form, person002 should be referred to by her given name, and zoo167 should be referred to simply in terms of the basic type of the object in question, with no additional modifiers.
102
ROBERT DALE
Linguistic realisation is the final stage of the process; this maps these sentence plans, which are still in the form of semantic constructs, into the appropriate lexico-syntactic material of the target natural language to produce the final output surface forms. It is at this point that the words I, Mary and zoo are finally selected. Within this view of the architecture of natural language generation, our interest here, then, is in a component process in the micro-planning stage: the system has determined that it wants to refer to a given entity, indicating this by incorporating the internal symbol for that entity in one or more messages, and the task of referring expression generation is then to determine the semantic content of a referring expression that identifies this entity for the hearer.2 2.2
Generating referring expressions
To make this concrete, suppose we are faced with the need to realize a proposition like that in Example (1): (1) owns(j, j1) Here, j is a symbolic identifier corresponding to some individual whose name we will assume is John, and j1 is the symbol corresponding to some particular entity that John owns. The task of the referring expression generator is to determine how to identify j and j1 for the hearer. We will ignore the reference to John and assume that we can straightforwardly decide to use his name in order to refer to him. Our current focus of interest is j1, where in the absence of a proper name that is known to both the speaker and hearer, we have to reason about the properties we know to be true of the entity in order to determine how to identify it for our hearer.3 In order to carry out this task, we need to have access to at least three kinds of knowledge: Knowledge of the domain: We require some representation of the properties of the entities in the domain, and of the relationships between those entities. 2
3
Other architectural models that carve up the process of natural language generation in different ways are possible. In almost all such architectures, however, there is a process whose focus is the mapping from internal symbols to the content of linguistic expressions that identify their referents. Even this is, of course, a simplification: certainly we can use properties that do not hold of the entity in question in order to describe it, just so long as the hearer believes those properties are true of the entity in question. So, when checking into a rather conservative hotel in a small village, I might successfully refer to my partner as my wife, even if we are not in fact married. Successful reference requires that I achieve my intention of singling out for you my intended referent; whether I achieve this by using properties that are true of the intended referent is less important.
GENERATING REFERRING EXPRESSIONS
103
Knowledge of the user: We need to have some representation of the hearer’s knowledge, so that we can avoid making use of terms that are alien to him or her. For example, there is little point in using a highly technical term for a piece of machinery if this term is unfamiliar to the hearer. Similarly, unless there is no alternative, there is little point in referring to some property of the entity that is not easily accessible to the hearer, such as a serial number on the underside of a very heavy object. Knowledge of the discourse: We need to have some representation of the preceding discourse context so we know what entities have already been referred to, and what has been said about them. This informs our use of pronominal forms and of reduced versions of previously used descriptions. The discourse context is a major factor in determining the appropriate form of reference to use on any subsequent reference to an entity, as is demonstrated by the following examples: (2) a. John owns a red jumper. b. He wears it on Sundays. (3) a. John owns a red jumper and a blue one. b. He wears the red one on Sundays. (4) a. John owns a red jumper and a red cardigan. b. He wears the jumper on Sundays. In each of these three examples, the proposition expressed in the second sentence is the same, and might be represented as something like wears(j,j1,on(day0)). However, as the examples demonstrate, the means used to refer to the jumper worn by John depend on the entities that have been mentioned in the previous discourse context. In the first example, a straighforward pronominal reference suffices, since there are no other likely antecedents for a neuter pronoun in the current context; in the second example, there are two jumpers in the context, and so we have to distinguish which of the two we mean by using a property that is unique to one; and in the third example, our context contains two entities of different types, so this can be used as the distinguishing characteristic in singling out our intended referent. In the simple cases demonstrated here, this seems quite straightforward and amenable to computational implementation given appropriate representational mechanisms for the various forms of information involved; but few cases are quite as simple as these. The approaches taken to this problem in the last 20 years have adopted a remarkably consistent characterisation of what the problem involves, and in this regard, the generation of referring expressions is rather unique within the larger portfolio of research questions that make up natural language generation. This shared conception of the problem makes it possible to
104
ROBERT DALE
compare and contrast solutions, at least to a much greater degree than is the case for many other problems. This has resulted in a focus within the literature on the development of precisely-stated algorithms that explore different facets of the problem. 2.3
A formal characterisation
The bulk of research into the generation of referring expressions takes as given the following assumptions. 1. We have a context that consists of a collection of entities, identified by symbolic identifiers, such as e1 through en . 2. One of these entities is our intended referent. 3. The entities in the context are characterized in terms of a set of attributes and the values that the entities have for these attributes. So, for example, our knowledge base might represent the fact that entity e1 has the value pen for the attribute type, and the value red for the attribute color, where the symbols used to designate attribute values correspond to conceptual or semantic elements rather than surface lexical items. We will use the notation hAttribute, Valuei for attribute– value pairs; for example, hcolor, redi indicates the attribute of color with the value red. 4. In a typical context where we want to refer to our intended referent ei , there will be other entities from which the intended referent must be distinguished; these are generally referred to as potential distractors. The goal of referring expression generation can then be stated thus: We want to find some collection of attributes and their values that distinguish the intended referent from all the potential distractors in the context. If such a collection of attributes and their values can be found, then it serves as a distinguishing description. Formally, we can characterize this as follows. Let r be the intended referent, and C be the set of distractors; then, a set L of attribute–value pairs will represent a distinguishing description if the following two conditions hold: • C1: Every attribute–value pair in L applies to r: that is, every element of L specifies an attribute and its value that r possesses. • C2: For every member c of C, there is at least one element l of L that does not apply to c: that is, there is an l in L that specifies an attribute value that c does not possess. l is said to rule out c.
GENERATING REFERRING EXPRESSIONS
105
We can think of the generation of referring expressions as being governed by three principles, referred to by Dale (1992) as the principles of adequacy, efficiency and sensitivity; these are Gricean-like conversational maxims (Grice 1975) framed from the point of view of the specific task of generating referring expressions. The first two of these principles are primarily concerned with saying neither too much nor too little. The principle of adequacy requires that a referring expression should contain enough information to allow the hearer to identify the referent, and the principle of efficiency requires that the referring expression should not contain unnecessary information (since, in line with Grice’s observations, the hearer will wonder why this information has been provided). The principle of sensitivity, however, has a different concern: it specifies that the referring expression constructed should be sensitive to the needs and abilities of the hearer or reader. Accordingly, the definition of a distinguishing description specified above should really include a third component: • C3: The hearer knows or can easily perceive that conditions C1 and C2 hold. In other words, the hearer must realize that the distinguishing description matches the intended referent and none of the potential distractors, and ideally this realization should not require a large perceptual or cognitive effort on the hearer’s part. 3 3.1
A history of work in the area Early work
As noted earlier, the generation of referring expressions is a component task in almost all nlg systems. Discussion of relevant issues that arise in the broader context of nlg system construction can be found in the seminal early language generation research of Davey (1979), McDonald (1980) and McKeown (1985). However, in each of these cases, the generation of referring expressions was only one of many tasks the authors set out to address, and so only some aspects of the problem were considered. The first major work to look specifically at the problem of referring expression generation was that of Appelt (1985). This work contains many insights into the range of problems that a thoroughgoing solution to the problem needs to consider, placing the task firmly within the framework of speech acts (Searle 1969). Building on other work that has attempted to model speech acts by means of AI-style planning operators, Appelt developed a plan-based approach to constructing references, whereby the speaker
106
ROBERT DALE
reasons about the effects on the hearer obtained by using particular elements of content in a referring expression. 3.2
Producing minimal distinguishing descriptions
The earlier works referred to above clearly established the need to view the generation of referring expressions as a specific task to be addressed within a generation system; Appelt’s work in particular demonstrated that the problem is a non-trivial one. However, none of these works provided a formally well-specified algorithm for carrying out the task. The principle of efficiency mentioned above suggests that, other things being equal, we desire an algorithm that will deliver a referring expression that mentions as few attributes as possible: a minimal distinguishing description. However, from the point of view of computational complexity, the problem of constructing a minimal distinguishing description is equivalent to finding the minimal size set cover. This problem is known to be NP-Hard (Garey & Johnson 1979), and any such algorithm is probably computationally impractical when entities have any more than a small number of properties that might be used in their descriptions. Much of the work in providing algorithms for referring expression generation has therefore been concerned to find heuristic workarounds that avoid these computational complexity issues. There is insufficient space here to consider each of the algorithms in the literature in detail; however, to provide a flavor of what such algorithms contain, we show here the first of the fully specified algorithms to appear, as published in (Dale 1992); this has served as a starting point for many of the subsequently proposed algorithms. The algorithm, which is essentially a variant of Johnson’s greedy heuristic for minimal set cover (Johnson 1974), is as shown in Figure 1. To see how this works, suppose we have a context that contains the following entities with the properties indicated: • e1 : htype, peni, hcolor, redi, hbrand, Staedtleri, hsize, largei • e2 : htype, peni, hcolor, bluei, hbrand, Staedtleri, hsize, smalli • e3 : htype, peni, hcolor, redi, hbrand, Stabiloi, hsize, largei • e4 : htype, peni, hcolor, greeni, hbrand, Staedtleri, hsize, largei Suppose that our intended referent is e1 . Then, the algorithm first chooses the property of being red, since this rules out more distractors than any other property possessed by e1 : it rules out both e2 and e4 , whereas the brand and size attributes only rule out one distractor each, and the type attribute has no impact at all. We still then have to add additional information to rule out e3 : here, size does not help, but adding the brand does what is
GENERATING REFERRING EXPRESSIONS
107
Let L be the set of properties to be realized in our description; let P be the set of properties known to be true of our intended referent r, where a property is a combination of an attribute and a value for that attribute; and let C be the set of potential distractors in the current context, referred to here as the contrast set. The initial conditions are thus as follows: • C = {hall distractorsi}; • P = {hall properties true of ri}; • L = {} In order to describe the intended referent r with respect to the contrast set C, we do the following. 1. Check Success: if |C| = 0 then return L as a distinguishing description elseif P = ∅ then fail else goto Step 2. 2. Choose Property: for each pi ∈ P do: Ci ← C ∩ {x|pi (x)} Chosen property is pj , where Cj is the smallest set. goto Step 3. 3. Extend Description (wrt the chosen pj ): L ← L ∪ {pj } C ← Cj P ← P − {pj } goto Step 1. Fig. 1: An algorithm for the Greedy Heuristic needed, since e1 and e3 have different brands. So, for e1 , the attributes color and brand provide us with a distinguishing description. On the other hand, if the intended referent was e2 , the color or size alone would be sufficient; and if the intended referent was e3 , the brand alone would be sufficient. Verifying that the algorithm does indeed provide these results is left as an exercise for the reader. In reality, since noun phrases always (or almost always) contain head nouns, the head noun indicating the entity’s type is also incorporated into the resulting description; for completeness, the algorithm in Figure 1 would have to be augmented to ensure this. In the case of e1 above, the resulting description might then be realised as the red Stabilo pen. The algorithm we have just presented is intended to be quite general and domain-independent; wherever we can characterize the problem in terms of the basic assumptions laid out earlier, this algorithm can be used to
108
ROBERT DALE
determine the content of context-dependent referring expressions. However, the algorithm also suffers from some limitations: • As stated, it may satisfy the principles of adequacy and efficiency, but it does not pay heed to the principle of sensitivity: there is no guarantee that the properties selected will even be perceptible to the hearer. In the scenario above, for example, if the labels that indicate the brands of the pens are face down on the table, and the pens have no other brand-distinguishing characteristics, then the brand information is of no real value for the hearer. • The algorithm only makes use of one-place predicates (such as the fact that our whiteboard pen is red), but we can of course also make use of relational properties (such as the fact that the pen is on the table rather than on the floor) when describing entities. Since this early algorithm, a wide range of alternatives have been proposed, either to address these particular problems or to extend the coverage of the approach in other ways. 3.3
More efficient algorithms
Although the algorithm above avoids the computational complexity of a literal interpretation of the principle of efficiency, it is still a relatively expensive way of computing a description. A second algorithm for referring expression generation that has had widespread influence is known as the Incremental Algorithm (Dale and Reiter 1995). This algorithm has three distinctive properties. • First, the algorithm sacrifices the goal of finding a minimal distinguishing description in the interests of efficiency and tractability. It does this by considering the attributes that might be used in a description in a predefined order. So, for example, in describing a physical object, we might first consider its color and then its size as appropriate properties to use in its description. This has the consequence that on occasion the algorithm may include a property whose discriminatory power is made redundant by some subsequently incorporated property; however, Dale and Reiter argue that this is not necessarily a bad thing, since human-generated referring expressions also often contain informational redundancy (see, for example, (Pechmann 1989)). It remains to be seen whether the algorithm provided by Dale and Reiter provides the same kinds of redundancies as those produced by humans. • Second, while the Incremental Algorithm contains a domainindependent, and therefore quite generally applicable, algorithm for
GENERATING REFERRING EXPRESSIONS
109
building up the content of a referring expression, it does so by reference to a predefined list of properties whose content is dependent on the domain. This provides a convenient balance between generality and domain-specificity: in order to use the algorithm in another domain, one simply provides an ordering over the attributes available in that domain. • Third, when the algorithm considers whether a given property would be useful to include in a description, it carries out a check to determine whether the property in question is one that the hearer would be able to make use of in identifying the referent. In so doing, it also explicitly aims to address the principle of sensitivity. The Incremental Algorithm has subsequently served as the basis for a number of other algorithms in the literature. 3.4
Referring to entities using relations
Another deficiency we noted above with respect to the greedy heuristic algorithm was that it could only make use of one-place predicates in determining which attributes of an entity to use in its description. The reason for this limitation is to avoid adding significant complexity to the algorithm: if we want to build referring expressions of the form the red pen next to the coffee cup, note that the entity we are relating the intended referent to (in this example, the coffee cup) is itself an entity for which we have to construct a referring expression. Once the generator decides that some other entity has to be introduced in order to identify the intended referent, a recursive invocation of the process of referring expression generation is required. As noted by Novak (1988), there is clearly the scope here for an infinite regress, as in the red pen next to the coffee cup next to the red pen next to the coffee cup next to . . . . In informal terms it would appear to be straightforward to deal with such cases appropriately, but capturing the required behavior algorithmically is a little more difficult. As a solution, Dale and Haddock (1991) proposed an algorithm based on constraint satisfaction. This effectively builds up collections of properties for each of the entities to be described in parallel, and determines when the properties (or more precisely, the relations) used to describe one entity also support the identification of another entity. More recently, Krahmer et al. (2003) have proposed a solution based on viewing the domain of entities, properties and relations as a labeled graph, with the problem of referring expression generation then being one of constructing an appropriate subgraph. This provides a very elegant solution with well-understood mathematical properties.
110 3.5
ROBERT DALE
Logical extensions: Sets, booleans and quantifiers
So far, we have pointed to algorithms that permit the use of both oneplace predicates and relations in the construction of referring expressions. Although this provides for a wide range of possibilities, it still does not encompass all the devices that we as humans use in constructing referring expressions. Notably, the algorithms discussed so far assume that the intended referent is a singular individual, and that it is to be picked out by the logical conjunction of the properties in the description. There are, of course, other possibilities here. For example, we may sometimes need to refer to sets of entities, as in Please pass me all the pens. Sometimes it may be convenient to refer to entities not just in terms of conjunctions of properties, but also via other Boolean possibilities, such as the use of negation and disjunction. To return to our earlier example, it might in some circumstances be most appropriate for me to say Please pass me the red pen that hasn’t run out of ink; and in cases where we want to refer to a set of entities, there may be no property that is shared by all the intended referents, requiring an expression like the red pen and the blue pen. These ideas are explored in detail by van Deemter (2002), who presents generalizations and extensions of the Incremental Algorithm described above that cater for these possibilities. Creaney (2002) offers an algorithm that allows the incorporation of logical quantifiers, such as all and some, into descriptions. 4
Outstanding issues
In the foregoing we have surveyed, albeit somewhat briefly, the major aspects of referring expression generation that have been the focus of attention in the development of algorithms over the last 15 years. There are many other algorithms in the literature beyond those presented here, but the problems addressed are essentially the same. In this section, we turn to aspects of the problem that have not yet received the same level of detailed exploration. 4.1
Other forms of anaphoric reference
We have focused here on the construction of definite descriptions: references to entities which are in the common ground, either by virtue of their presence in the environment, or because they have already been introduced into the discourse. Definite descriptions are not the only form of anaphoric reference: as we noted at the outset of this paper, we can, of course, also use pronouns such as it and one-anaphoric expressions such as the red one to identify intended referents. Although the literature does contain some work
GENERATING REFERRING EXPRESSIONS
111
on generating these forms of reference, they have not received the degree of attention that has been lavished on definite descriptions, and in particular there has been little work that has seriously attempted to integrate the various forms of reference in a unified algorithmic framework. This is perhaps most surprising in the case of pronominal reference, given that this is seen as such a central problem in natural language understanding. Other forms of definite reference have also been neglected: in particular, there is no substantive work that explores the generation of associative anaphoric expressions, such as the use of a reference like the cap in a context where the hearer can infer that this must be a first mention of the cap of a particular pen that is already salient in the discourse. 4.2
Initial reference
An even more striking gap in the literature is any serious treatment of initial reference, as opposed to subsequent or anaphoric reference to entities which are already in the common ground. Subsequent references to entities have a tendency to reuse information used in earlier references; so, for example, I might introduce an entity as that Stabilo pen on the table and subsequently refer to it as the Stabilo pen. The generation literature not only tends to ignore this influence of the initial form of reference on subsequent referring expressions, but is in general quite silent on the problem of how initial references can be constructed. There are a great many ways in which I might refer to a specific entity when introducing it into the discourse, and when we move beyond simple examples of collections of similar objects, as are often used in the literature, it becomes clear that the form of initial reference used has a lot to do with the purpose of the discourse and other aspects of the discourse context. Thus, my decision as to whether to introduce someone into our conversation as a man I met at the bus stop, a mountain climber I met last night, or an interesting guy I met, will depend on what I have planned in terms of content for the rest of the discourse. The complexities of planning and reasoning required to explain how this selection process works are far beyond our current understanding. 4.3
The pragmatics of reference
Following on from the previous point, relatively little work has looked at the broader context of reference, and in particular the fact that, when we construct a referring expression, we may be doing many things simultaneously and attempting to achieve a variety of purposes. Building on the earlier explorations of Appelt (1985), the work of Kronfeld (1990) is a notable exception here; but we lack a worked-out computational theory of how reference functions within the wider context of
112
ROBERT DALE
language use, and in particular we have no formally well-specified explanations of the variety of purposes of reference. The bulk of the work in the field has focused on referring expressions as linguistic devices to distinguish intended referents from other entities in a given context, but this is a rather narrow and low-level view. We also use reference to maintain topic, to shift topic, to introduce contrasting entities, and to navigate a hearer’s focus of attention in various ways, amongst a range of other higher level purposes. These are all relatively unanalyzed notions from a formal perspective, and the relationships between them need to be worked out more clearly before we can say that we have a proper computational theory of the use of reference and the part it plays in language. As is often the case, approaching this problem from the perspective of natural language generation may provide insights that can also enrich our understanding of natural language analysis. 4.4
Evaluation
Despite the limitations discussed in the preceding sections, it turns out that there are, as yet, no significant nlg applications that would provide a thorough testing ground for the wide range of algorithms that have been proposed; for most currently practical applications of language generation, relatively simple techniques will suffice. As a consequence, the bulk of the algorithms we have mentioned are explored by their designers in relatively simple toy domains. However, there would clearly be some advantage in being able to compare the performance of different algorithms against some standard data set of desirable referring expressions. This raises a number of questions about how such a corpus might be constructed, and how the performance of any given algorithm might be measured; some of these issues are discussed further in (van Deemter et al. 2006) and (Viethen & Dale 2006). 5
Conclusions
As we have tried to show in the preceeding, a significant body of work in the generation of referring expressions has been developed over the last 20 years, almost all of which has the same core conception of the task. This makes it possible for new work to build on existing work, and provides an excellent basis for comparison of algorithms, although it is not yet the case that such comparisons have been carried out in an empirically driven fashion using large data sets of human-generated referring expressions.4 4
A significant exception here is the work reported in Albert Gatt’s forthcoming Ph.D. (Gatt 2007).
GENERATING REFERRING EXPRESSIONS
113
An intriguing question is why it is that a consensus view of the problem of referring expression generation has developed, where this has not happened for many other subtasks in nlg. For example, there is much less agreement on the exact nature of the tasks of text planning, document structuring, aggregation, and linguistic realisation. Sometimes we find divergence in terms of the nature of the processing that falls within a particular component’s responsibilities; but in many cases the disagreements arise in terms of the nature of the inputs and the outputs of the process. There are, indeed, other conceptions of the process of referring expression generation that we might consider; why, for example, do we draw a distinction between the determination of the semantic content of a referring expression and the mapping of this semantic content into a surface linguistic form? Why do we take the input to the process to be an internal symbol, rather than some partially developed semantic construct whose content has been derived to meet other communicative needs? It is likely that existing work in the area can be extended, without too much effort, to be used in these alternative architectural models; from this perspective, the current model of referring expression generation is a useful simplification that allows for the development of more complex solutions. Ultimately, however, our algorithms will be truly validated when we have functioning systems that are required to make decisions about forms of reference in rich semantic contexts. The applications we alluded to at the beginning of this paper—embodied conversational agents who take part in complex dialogues, and personalisation technology for delivering emails and web pages—will provide these requirements; within the next five to ten years we can expect to see the presently theoretically-driven results of research in referring expression generation put to the test in real applications. REFERENCES Appelt, Douglas E. 1985. Planning English Sentences. Cambridge: Cambridge University Press. Creaney, Norman. 2002. “Generating Descriptions Containing Quantifiers: Aggregation and Search”. Information Sharing: Reference and Presupposition in Language Generation and Interpretation ed. by Kees van Deemter & Rodger Kibble (= CSLI Lecture Notes, 143), 265-294. Stanford: CSLI Publications. Dale, Robert. 1992. Generating Referring Expressions: Constructing Descriptions in a Domain of Objects and Processes. Cambridge, Mass.: MIT Press. Dale, Robert & Nicholas Haddock. 1991. “Content Determination in the Generation of Referring Expressions”. Computational Intelligence 7:4.252-265.
114
ROBERT DALE
Dale, Robert & Ehud Reiter. 1995. “Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions”. Cognitive Science 19:2.233-263. Davey, Anthony C. 1979. Discourse Production. Edinburgh: Edinburgh University Press. Garey W. & D. Johnson. 1979. Computers and Intractability: A Guide to the Theory of NP-Completeness. San Francisco: W. H. Freeman. Gatt, Albert. 2007. Generating Coherent References to Multiple Entities. Ph.D. Dissertation, University of Aberdeen, Aberdeen, Scotland. Grice, H. P. 1975. “Logic and Conversation”. Syntax and Semantics Volume 3: Speech Acts ed. by P. Cole and J. Morgan, 43-58. New York: Academic Press. Johnson, D. S. 1974. “Approximation Algorithms for Combinatorial Problems”, Journal of Computer and Systems Sciences, vol. 9, 256-278. Krahmer, Emiel, Sebastiaan van Erk & Andr´e Verleg. 2003. “Graph-based Generation of Referring Expressions”. Computational Linguistics 29:1.53-72. Kronfeld, Amichai. 1990. Reference and Computation. Cambridge: Cambridge University Press. McDonald, David D. 1980. Natural Language Production as a Process of Decision Making under Constraints. Ph.D. dissertation, MIT, Cambridge, Mass. McKeown, Kathleen R. 1985. Text Generation: Using Discourse Strategies and Focus Constraints to Generate Natural Language Text. Cambridge: Cambridge University Press. Novak, Hans-Joachim. 1988. “Generating Referring Phrases in a Dynamic Environment”. Advances in Natural Language Generation vol. 2 ed. by Michael Zock and Gerard Sabah, 76-85. London: Pinter. Pechmann, Thomas. 1989. “Incremental Speech Production and Referential Overspecification”. Linguistics 27:1.89-110. Reiter, Ehud & Robert Dale. 2000. Building Natural Language Generation Systems. Cambridge: Cambridge University Press. Searle, J. R. 1969. Speech Acts: An Essay in the Philosophy of Language. Cambridge: Cambridge University Press. van Deemter, Kees. 2002. “Generating referring expressions: Boolean extensions of the Incremental Algorithm”. Computational Linguistics 28:1.37-52. van Deemter, Kees, Ielka van der Sluis & Albert Gatt. 2006. “Building a Semantically Transparent Corpus for the Generation of Referring Expressions”. Proceedings of the International Conference on Natural Language Generation 15-16 July, 130-132. Sydney, Australia. Viethen, Jette & Robert Dale. 2006. “Algorithms for Generating Referring Expressions: Do They Do What People Do?”. Proceedings of the International Conference on Natural Language Generation 15-16 July, 63-70. Sydney, Australia.
A Data-driven Approach to Pronominal Anaphora Resolution for German Erhard W. Hinrichs, Katja Filippova & Holger Wunsch SfS-CL, University of T¨ ubingen Abstract This paper reports on a hybrid architecture for computational anaphora resolution (Car) of German that combines a rule-based pre-filtering component with a memory-based resolution module (using the Tilburg Memory Based Learner – TiMBL). The Car experiments performed on the T¨ uBa-D/Z treebank data corroborate the importance of modelling aspects of discourse structure for robust, data-driven anaphora resolution. The best result with an F-measure of 0.734 achieved by these experiments outperforms the results reported by Schiehlen (2004), the only other study of German Car that is based on newspaper treebank data.
1
Introduction
The present study focuses exclusively on the resolution of pronominal anaphora with NP antecedents for German, where the term pronoun is used as a cover term for 3rd person reflexive, possessive, and personal pronouns. The purpose of this paper is threefold: (i) to apply the machine learning paradigm of memory-based learning to the task of Car for German, (ii) to provide a series of experiments that corroborate the importance of modelling aspects of discourse structure for robust, data-driven anaphora resolution and that induce more fine-grained information from the data than previous approaches, (iii) to apply Car to a corpus of German newspaper texts, yielding competitive results for a genre that is known to be considerably more difficult than the Heidelberg corpus of tourist information texts (see Kouchnir (2003) for more discussion on this issue). 2
Previous research on Computational Anaphora Resolution
While early work on Car was carried out almost exclusively in a rule-based paradigm, there have been numerous studies during the last ten years that have demonstrated that machine-learning and statistical approaches to Car can offer competitive results to rule-based approaches. In particular, this more recent work has shown that the hand-tuned weights for anaphora
116
ERHARD W. HINRICHS, KATJA FILIPPOVA & HOLGER WUNSCH
resolution introduced by Lappin & Leass (1994), by Kennedy & Boguraev (1996), and Mitkov (2002) can be successfully simulated by data-driven methods (Preiss 2002b). While there is a rich diversity of methods that have been applied to Car, there is also a striking convergence of grammatical features that are used as linguistic knowledge across different algorithms. Most approaches base their resolution algorithm on some combination of distance between pronouns and potential antecedents, grammatical agreement between pronouns and antecedents, constituent structure information, grammatical function assignment for potential antecedents, and the type of NP involved (e.g. whether it is definite or indefinite). The combined effect of these features is to establish a notion of discourse salience that can help rank potential antecedents. An important aspect of discourse salience is its dynamic character since there seems to be a strong correlation between salience and discourse recency. This aspect of salience was first captured by Lappin & Leass (1994) and by Kennedy & Boguraev (1996) through the use of a decay function that decreases the score of a potential antecedent each time a new sentence is processed. In data-driven approaches this decay function is simulated by the distance measure between pronoun and antecedent. Previous studies of Car have focused on English and have been based on text corpora of fairly modest size, however see Ge et al. (1998) for an exception. The only previous studies for German have been presented by Strube & Hahn (1999), based on centering theory, M¨ uller et al. (2002), using co-training, and by Kouchnir (2003), who applies boosting. Schiehlen (2004) provides an overview of adapting Car algorithms to German that were originally developed for English. While memory-based learning (Mbl) has been successfully applied to a wide variety of Nlp tasks, there has been only one previous study of Car using Mbl (Preiss 2002a). 3
Data
The present research focuses on German and utilizes the T¨ uBa-D/Z (Hinrichs et al. 2004), a large treebank of German newspaper text that has been manually annotated with constituent structure and grammatical relations such as subject, direct object, indirect object and modifier. These types of syntactic information have proven crucial in previous Car algorithms. More recently, the T¨ uBa-D/Z annotations have been further enriched to also include anaphoric relations (Hinrichs et al. 2004), thereby making the treebank suitable for research on Car. German constitutes an interesting point of comparison to English since German exhibits a much richer inflectional morphology and a relatively free word order at the phrase level.
PRONOMINAL ANAPHORA RESOLUTION FOR GERMAN
117
The sample sentences in (1) illustrate the annotation of referentially dependent relations in the T¨ uBa-D/Z anaphora corpus. (1) [1 Der neue Vorsitzende der Gewerkschaft Erziehung und Wissenschaft] heißt [2 Ulli Th¨ one]. [3 Er] wurde gestern mit 217 von 355 Stimmen gew¨ahlt. ‘The new chairman of the union of educators and scholars is called Ulli Th¨one. He was elected yesterday with 217 of 355 votes’. In (1) a coreference relation exists between the noun phrases [1] and [2], and an anaphoric relation between the noun phrase [2] and the personal pronoun [3]. Since noun phrases [1] and [2] are coreferential, there exists an implicit anaphoric relation between NP [1] and NP [3], with all three NPs belonging to the same coreference chain. In keeping with the Muc-6 annotation standard1 , the anaphoric relation of a pronoun is established only to its most recently mentioned antecedent. The T¨ uBa-D/Z currently consists of 766 newspaper texts with a total of 15260 sentences and an average number of 19.46 sentences per text. The T¨ uBa-D/Z contains 7606 reflexive and personal pronouns, 2195 possessive pronouns, and 99585 markables (i.e. potential antecedent NPs). The number of pronouns in the T¨ uBa-D/Z corpus is considerably larger than in the hand-annotated portion of the German Negra newspaper corpus (2198 possessive pronouns, 3115 personal pronouns) utilized in Schiehlen (2004) and substantially larger than the German Heidelberg tourism information corpus (36924 tokens, 2179 anaphoric NPs) used by M¨ uller et al. (2002) and by Kouchnir (2003). 4
Experiments
The experiments are based on a hybrid architecture that combines a rulebased pre-filtering module with a memory-based resolution algorithm. The purpose of the pre-filtering module, which has been implemented in the Xerox Incremental Deep Parsing System (Xip) (A¨ıt-Mokhtar et al. 2002), is to retain only those NPs as potential antecedents that match a given pronoun in number and gender. Due to the richness of inflectional endings in German, this pre-processing step is crucial for cutting down the size of the search space of possible antecedents. Without Xip pre-filtering, the T¨ uBa-D/Z corpus yields a total of 1,412,784 of anaphor/candidateantecedent pairs. This number represents all possible ways of pairing a pronoun with an antecedent NP in each of the 766 texts of the T¨ uBa-D/Z corpus. After pre-filtering this number is reduced to appr. 190,000 pairs. 1
See www.cs.nyu.edu/cs/faculty/grishman/COtask21.book 1.html.
118
ERHARD W. HINRICHS, KATJA FILIPPOVA & HOLGER WUNSCH
pronoun/antecedent discourse history pronoun
cataphoric On Mod Title reflexive
parallel Od Opp Conj possessive
clause-mate Oa Fopp Hd
distance Pred App Other
Table 1: Feature set
The memory-based resolution module utilizes the Tilburg Memory Based Learner (TiMBL), version 5.1 (Daelemans et al. 2005). Unless otherwise specified, the experiments use the default settings of TiMBL. 4.1
Feature set
In the experiments, the TiMBL learner was presented with the set of features summarized in Table 1. The features on line 1 all refer to relational properties of the pronoun and potential antecedents. The feature parallel encodes whether the anaphor and the potential antecedent have the same grammatical function. The features on line 3 refer to the pronoun alone and encode whether it is possessive or reflexive. The features on line 2 are designed to model the discourse history in terms of the grammatical functions of NPs that are in the same coreference class as the candidate antecedent. The grammatical functions are those provided by the syntactic annotation of the T¨ uBa-D/Z treebank: On (for: subject), Oa (for: direct object), Od (for: dative object), Pred (for: predicative complement), Mod (for: modifier ), etc. The main purpose of the experiments reported here was to systematically study the impact that information about discourse context has on the performance of data-driven approaches to Car. To this end, we designed two experiments that differ from each other in the amount of information about the coreference chains that are encoded in the training data. 4.2
Knowledge-rich encoding of instances – Experiment I
In Experiment I, complete information about coreference chains is used for training. In example (1) the three bracketed NPs form one coreference chain since the first two NPs are coreferent and the pronoun is anaphoric to both. Accordingly, for example (1), two positive instances are created as shown in Table 2. Binary features are encoded as yes/no. Features related to distances are given numeric values from 1 to 30, with a special value of 31 reserved for the value undefined. Inspection of the data showed that a context window of size 30 contains the antecedent in more than 99% of all
PRONOMINAL ANAPHORA RESOLUTION FOR GERMAN
119
cataphoric parallel clause mate distance On Od Oa Pred Mod no no no -1 -1 -31 -31 -31 -31 no no no -1 -1 -31 -31 -1 -31 Opp Fopp App Title Conj Hd Other refl poss class -31 -31 -31 -31 -31 -31 -31 no no yes -31 -31 -31 -31 -31 -31 -31 no no yes
Table 2: Positive knowledge-rich sample instances
cases. For technical reasons, the numeric values are prefixed by a dash in order for TiMBL to treat them as discrete rather than continuous values. The first vector in Table 2 displays the pairing of the pronoun with the NP der neue Vorsitzende der Gewerkschaft Erziehung und Wissenschaft, the first NP in the text. This NP is the subject (On) of its clause. The value for this grammatical function is −1 since the NP occurs in the clause immediately preceding the pronoun. The second vector pairs the two preceding NPs with the pronoun er. Since the NP Ulli Th¨ one is in predicative position (Pred) and occurs in the same clause as the subject NP der neue Vorsitzende der Gewerkschaft Erziehung und Wissenschaft, the value for these two grammatical functions On and Pred is −1. Thus, the intended semantics of the features for each grammatical function is to encode the distance of the last occurrence of a member of the same coreference class with that particular grammatical function.2 Using mention counts of grammatical functions instead of distances did not significantly change the results, and were therefore omitted from the experiments. The sample vectors in Table 2 illustrate the incremental encoding of instances. The initial vector encodes only the relation between the pronoun and the antecedent first mentioned in the text. Each subsequent instance adds one more member of the same coreference class. This incremental encoding follows the strategy of Kennedy & Boguraev (1996) and reflects a dynamic modelling of the discourse history. The last item in the vector indicates class membership. In the memory-based encoding used in the experiments, anaphora resolution is turned into a binary classification problem. If an anaphoric relation holds between an anaphor and an antecedent, then this is encoded as a positive instance, i.e., as a vector ending in yes. If no anaphoric relation holds between a pronoun and an NP, then this encoded as a negative instance, i.e., as a vector ending in no. 2
A similar encoding is also used by Preiss (2002a).
120 4.3
ERHARD W. HINRICHS, KATJA FILIPPOVA & HOLGER WUNSCH
Knowledge-poor encoding of instances – Experiment II
Experiment II uses a more knowledge-poor encoding of the data and pairs each pronoun only with the most recent antecedent in the same coreference class, thereby losing both information inherent in the entire coreference class and at the same time truncating the discourse history. Using example (1) once more as an illustration, two positive instances are created. The first vector is the same as in Experiment I. The second vector retains value −1 only for Pred, the grammatical function of the candidate itself. The value of On is now undefined (−31). 4.4
Two variants
For each of the two experiments described above, two variants were conducted. In one version, the evaluation focused on the closest antecedent to calculate the result for recall, precision and F-measure.3 In a second variant, the most confident antecedent was chosen. The confidence measure was calculated by the function conf(t, ck ) := Pndk di for classes c1 . . . cn , and class i=1 distributions d1 . . . dn (where di is the number of neighbors that classified the test instance t as belonging to class ci ). 5
Evaluation
To assess the difficulty of the pronoun resolution task for the T¨ uBa-D/Z corpus, we established as a baseline a simple heuristic that picks the closest preceding subject as the antecedent. This baseline is summarized in Table 3 together with results of the experiments described in the previous section. For each experiment ten-fold cross-validation was performed, using 90% of the corpus for training and 10% for testing. 5.1
Results of Experiments I and II
Both experiments significantly outperform the baseline approach in F-measure. The findings summarized in Table 3 corroborate the importance of modelling the discourse history for pronoun resolution since the results of Experiment I are consistently better than those of Experiment II. The present paper does not have to rely on the hand-coding of a decay function. Rather, it induces the relevant aspects of the discourse history directly from the instance base used by the memory-based learner. 3
Throughout this paper the term F-measure implies the parameter setting of β = 1.
PRONOMINAL ANAPHORA RESOLUTION FOR GERMAN
Baseline Experiment I closest antecedent most conf. antecedent Experiment II closest antecedent most conf. antecedent
av. precision 0.500
av. recall 0.647
av. F-measure 0.564
0.826 0.801
0.640 0.621
0.721 0.700
0.779 0.786
0.600 0.606
0.678 0.684
121
Table 3: Summary of results 6 most informative features: 3 least informative features:
clause-mate,parallel,possessive,Fopp,On,Od Title, distance,Conj
Table 4: Summary of feature weights based on GainRatio values
It is also noteworthy that in Experiment I the strategy of picking the closest antecedent outperforms the strategy of picking the most confident antecedent chosen by TiMBL. 5.2
Benchmarking feature impact
It is instructive to benchmark the importance of the features used in the experiments. This can be ascertained from the weights that the gain ratio measure (as the default feature weighting used by TiMBL) assigns to each feature. Gain ratio is an entropy-based measure that assigns higher weights to more informative features. Table 4 displays the top six most informative features and the three least informative features in decreasing order of informativeness. The fact that the features clause-mate, parallel, and possessive are the three most informative features concurs with the importance given to such features in hand-crafted algorithms for Car. However, the ranking of some of the features included in Table 4 is rather unexpected. The fact that the grammatical function Fopp (for: optional PP complement) outranks the grammatical function subject (On) runs counter to hand-coded salience rankings found in the literature which give the feature subject the highest weights among all grammatical functions. That the Fopp feature outranks the function subject is due to the fact that the presence of an optional PP-complement is almost exclusively paired with negative instances. This finding points to an important advantage of data-driven approaches over hand-crafted models. While the latter only take into account positive evidence, data-driven models can profit from considering positive and
122
ERHARD W. HINRICHS, KATJA FILIPPOVA & HOLGER WUNSCH
Baseline Experiment I closest antecedent
av. precision 0.500
av. recall 0.647
av. F-measure 0.564
0.827
0.661
0.734
Table 5: Summary of best results
negative evidence alike. Perhaps the most surprising result is the fact that distance between anaphor and antecedent is given the second lowest weight among all eighteen features. This sharply contrasts with the intuition often cited in hand-crafted approaches that the distance between anaphor and antecedent is a very important feature for an adequate resolution algorithm. The reason why distance receives such a low weight might well have to do with the fact that this feature becomes almost redundant when used together with the other distance-based features for grammatical functions. The empirical findings concerning feature weights summarized in Table 4 underscore the limitation of hand-crafted approaches that are based on the analysts’ intuitions about the task domain. In many cases, the relative weights of features assigned by data-driven approaches will coincide with the weights assigned by human analysts and fine-tuned by trial and error. However, in some cases, feature weightings obtained automatically by data-driven methods will be more objective and diverge considerably from manual methods, as the weight assigned by TiMBL to the feature distance illustrates. 5.3
Optimization by fine-tuning of TiMBL parameters
It has been frequently observed (e.g. by Hoste et al. (2002)) that the default settings provided by a classifier often do not yield the optimal results for a given task. The Car task for German is no exception in this regard. TiMBL offers a rich suite of parameter settings that can be explored for optimizing the results obtained by its default settings. Some key parameters concern the choice of feature distance metrics, the value of k for the number of nearest neighbors that are considered during classification as well as the choice of voting method among the k-nearest neighbors used in classification. TiMBL’s default settings provide the feature distance metric of weighted overlap (with the gain ratio measure for feature weighting), k = 1 as the number of k-nearest neighbors, and majority class voting. To assess the possibilities of optimizing the results of Experiments I and II, the best result (Experiment I with closest antecedent) was chosen as a starting point. The best results, shown in Table 5, were obtained by us-
PRONOMINAL ANAPHORA RESOLUTION FOR GERMAN
123
ing TiMBL with the following parameters: modified value distance metric (Mvdm), no feature weighting, k = 3, and inverse distance weighting for class voting. The optimizing effect of the parameters is not entirely surprising.4 The Mvdm metric determines the similarities of feature values by computing the difference of the conditional distribution of the target classes for these values. For informative features, δ(v1 , v2 ) will on average be large, while for less informative features δ will tend to be small. Daelemans et al. (2005) report that for Nlp tasks Mvdm should be combined with values of k larger than one. The present task confirms this result by achieving optimal results for a value of k = 3. 6
Comparison with related work
The only previous study of German Car that is based on newspaper treebank data is that of Schiehlen (2004). Schiehlen compares an impressive collection of published algorithms, ranging from reimplementations of rulebased algorithms to reimplementations of machine-learning and statistical approaches. The best results of testing on the Negra corpus were achieved with an F-measure of 0.711 by a decision-tree classifier, using C4.5 and a pre-filtering module similar to the one used here. The best result with an F-measure of 0.734 achieved by the memory-based classifier and the Xipbased pre-filtering component outperforms Schiehlen’s results, although a direct comparison is not possible due to the different data sets. 7
Summary and future work
The current paper presents a hybrid architecture for computational anaphora resolution (Car) of German that combines a rule-based pre-filtering component with a memory-based resolution module (using the Tilburg Memory Based Learner – TiMBL). The data source is provided by the T¨ uBa-D/Z treebank of German newspaper text that is annotated with anaphoric relations. The Car experiments performed on these treebank data corroborate the importance of modelling aspects of discourse structure for robust, datadriven anaphora resolution. The best result with an F-measure of 0.734 achieved by the memory-based classifier and the Xip-based pre-filtering component outperforms Schiehlen’s results, although a direct comparison is not possible due to the different data sets.
4
See Hoste et al. (2002) for the optimizing effect of Mvdm in the word sense disambiguation task.
124
ERHARD W. HINRICHS, KATJA FILIPPOVA & HOLGER WUNSCH
REFERENCES A¨ıt-Mokhtar, Salah, Jean-Pierre Chanod & Claude Roux. 2002. “Robustness Beyond Shallowness: Incremental Deep Parsing”. Natural Language Engineering 8:2-3.121-144. Daelemans, Walter, Jakub Zavrel, Ko van der Sloot & Antal van den Bosch. 2005. TiMBL: Tilburg Memory Based Learner – version 5.1–Reference Guide. Technical Report ILK 01-04. Induction of Linguistic Knowledge, Computational Linguistics, Tilburg University. Ge, Niyu, John Hale & Eugene Charniak. 1998. “A Statistical Approach to Anaphora Resolution”. Sixth Workshop on Very Large Corpora, 161-170. Montreal, Canada. Hinrichs, Erhard, Sandra K¨ ubler, Karin Naumann, Heike Telljohann & Julia Trushkina. 2004. “Recent Developments in Linguistic Annotations of the T¨ uBa-D/Z Treebank”. Third Workshop on Treebanks and Linguistic Theories, 51-62. T¨ ubingen, Germany. Hoste, Veronique, Iris Hendrickx, Walter Daelemans & Antal van den Bosch. 2002. “Parameter Optimization for Machine-Learning of Word Sense Disambiguation”. Natural Language Engineering 8:4.311-325. Kennedy, Christopher & Branimir Boguraev. 1996. “Anaphora for Everyone: Pronominal Anaphora Resolution Without a Parser”. 16th International Conference on Computational Linguistics, 113-118. Copenhagen, Denmark. Kouchnir, Beata. 2003. A Machine Learning Approach to German Pronoun Resolution. Master’s thesis. School of Informatics, University of Edinburgh. Lappin, Shalom & Herbert Leass. 1994. “An Algorithm for Pronominal Anaphora Resolution”. Computational Linguistics 20:4.535-561. Mitkov, Ruslan. 2002. Anaphora Resolution. Amsterdam: John Benjamins. M¨ uller, Christoph, Stefan Rapp & Michael Strube. 2002. “Applying co-training to reference resolution”. 40th Annual Meeting of the Association for Computational Linguistics (ACL’02 ), 352-359. Philadelphia, Penn. Preiss, Judita. 2002a. “Anaphora Resolution With Memory-Based Learning”. 5th UK Special Interest Group for Computational Linguistics (CLUK5 ), 1-8. Preiss, Judita. 2002b. “A Comparison of Probabilistic and Non-Probabilistic Anaphora Resolution Algorithms”. Student Workshop at the 40th Annual Meeting of the Association for Computational Linguistics (ACL’02 ), 42-47. Philadelphia, Penn. Schiehlen, Michael. 2004. “Optimizing Algorithms for Pronoun Resolution”. 20th International Conference on Computational Linguistics (COLING-2004 ), 515-521. Geneva, Switzerland. Strube, Michael & Udo Hahn. 1999. “Functional Centering - Grounding Referential Coherence in Information Structure”. Computational Linguistics 25:3.309-344.
Efficient Spam Analysis for Weblogs through URL Segmentation Nicolas Nicolov & Franco Salvetti Umbria, Inc. & University of Colorado at Boulder Abstract This paper shows that in the context of spam classification of weblogs, segmenting the urls beyond the standard punctuation is helpful. Many spam urls contain phrases in which the words are glued together in order to avoid spam filtering techniques based on unigrams and punctuation segmentation. An approach for segmenting tokens into words forming a phrase is proposed and validated. The accuracy of the implemented system is better than that of humans (78% vs. 76%).
1
Introduction
The blogosphere, which is the part of the web consisting of personal electronic journals (weblogs) currently encompasses 107.7 million pages [200710-02] and doubles in size every 5.5 months (Technorati 2006). The large amount of information contained in the blogosphere has been proven valuable for applications such as market intelligence, trend discovery, and opinion tracking (Hurst 2005). Unfortunately in the last several years the blogosphere has been heavily polluted with spam. For certain subsets of the blogosphere the spam content can be as high as 70-80%. Spam weblogs (called splogs) are weblogs used for the purposes of promoting affiliated websites. Splogs skew statistics. Thus, methods for filtering out splogs on large collections of weblogs are needed. Sophisticated content-based methods or methods based on link analysis (Gy¨ongyi et al. 2004, Andras et al. 2005) can be slow. We propose a fast, lightweight and accurate method merely based on analysis of the url of the web page itself without considering (nor downloading) the content of the page (Kan 2004, Kan & Thi 2005). For qualitative analysis of the contents of the blogosphere it is acceptable to eliminate data from analysis as long as the remaining part is representative and spam-free. For example, in determining the public opinion about a new product it is sufficient to determine the ratio of the people that like the product vs. the people that dislike it rather than the actual number. Therefore, considering the vasts amount of weblogs available it is acceptable for a spam classifier to lower recall in order to improve precision, especially
126
NICOLAS NICOLOV & FRANCO SALVETTI
when used as a pre-filter. Our method reaches 93.3% of accuracy in classifying a web page in terms of spam or good if 49.1% of the data are left aside (labeled as unknown). If all data needs to be classified our method achieves 78% accuracy which is higher than the average accuracy of humans (76%) on the url spam classification task. Spammers, in creating splogs (Wikipedia 2006:Splog), aim to increase the traffic to specific websites. To do so, they frequently communicate a concept (e.g., a service or a product) through a short, sometimes nongrammatical phrase embedded in the url of the page (e.g., http://adult-video-mpegs.blogspot.com). We want to build a classifier using machine learning techniques which leverages the language used in these descriptive urls in order to classify weblogs as spam or good. We built an initial Language Model (lm) based classifier on the tokens of the urls after tokenizing on punctuation. We ran the system and got an accuracy of 72.2% which is close to the accuracy of humans—76% (the baseline is 50% as the training data is balanced). When we did error analysis on the misclassified examples we observed that many of the mistakes were on urls that contain whole phrases (in the sense of sequence of words rather than syntactic constituents) glued together as one token (e.g., www.dailyfreeipod.com). Had the words in these phrases been segmented the initial system would have classified the url correctly. We, thus, turned our attention to additional segmenting of the urls beyond just punctuation and using this ‘aggressive’ segmentation in classification. Training a segmenter on standard available text collections (e.g., The Penn TreeBank or The British National Corpus) seemed not the way to procede because the lexical items used, and the sequence in which words appear differ from the usage in the urls. Given that we are interested in unsupervised lightweight approaches for url segmentation one possibility is to use the urls themselves after segmenting on punctuation and to try to learn the segmenting (the majority of urls are naturally segmented using punctuation as we shall see later). We trained a segmenter on the urls, unfortunately this method did not provide sufficient improvement over the system that does not use additional segmentation. We hypothesized that the content of the spam pages corresponding to the urls could be used as a corpus to learn the segmentation. We crawled the 20K pages corresponding to the 20K urls labeled as spam and good in the training set, converted them to text, tokenized and used the token sequences as training data for the segmenter. This led to a statistically significant improvement of 5.8% of the accuracy of the spam filter. The structure of the paper is as follows: in Section 2 we discuss the notion of splogs in the blogosphere and how it is reflected in urls. We then describe segmentation in Section 3 (the heart of the paper) and a classi-
127
SPAM CLASSIFICATION OF WEBLOGS
fication approach (Section 4) which uses the segmentation. Experimental results including the performance of humans are shown in Section 5 and discussed in Section 6. We highlight future directions in Section 7 and conclude in Section 8. 2
Engineering of splogs
“Splogs, are weblog sites which the author uses only for promoting affiliated websites. The purpose is to increase the PageRank of the affiliated sites, get ad impressions from visitors, and/or use the blog as a link outlet to get new sites indexed. Content is often nonsense or text stolen from other websites with an unusually high number of links to sites associated with the splog creator which are often disreputable or otherwise useless websites.” (Wikipedia 2006:Splog). Spammers indicate the semantic content of the webpage directly in the url. A Uniform Resource Locator (url), or web address, is a standardized address name layout for a resource (a document, image, etc.) on the internet (or elsewhere). The currently used url syntax is specified in the internet standard RFC 17381 : resource type://hostname.domain:port/file path name#anchor
While the resource type and domain are a small set, the hostname, file path name and anchor present a perfect opportunity to indicate semantic content by using descriptive phrases—often noun groups (non-recursive noun phrases) like http://www.adultvideompegs.blogspot.com.2 There are different varieties of sites promoted by spam: 1. commercial products (especially electronics); 2. vacations; 3. mortgages; 4. adult-related. Users don’t want to see spam web pages in their search results and market intelligence applications are affected when data contains spam. To address this, search engines provide filtering. Simple approaches to filtering use keyword spotting (or unigram models)— if the webpage or its url contain certain tokens the page is eliminated from the results (or ranked lower). For spotting keywords in the urls a url is often tokenized on punctuation symbols. To avoid being identified as a splog one of the creative techniques that spammers use is to glue words 1 2
See also RFC 3986 on Uniform Resource Identifiers (URIs). Non-spam sites also communicate concepts through http://showusyourcharacter.com.
the
urls:
128
NICOLAS NICOLOV & FRANCO SALVETTI
together into longer tokens which will not match the filter keywords (e.g., http://businessopportunitymoneyworkathome.coolblogstuff.com). Another approach to dealing with spam is having a list of spam websites (SURBL 2006). Such approaches based on blacklists are now less effective because the available free bloghost tools for creating blogs have made it very easy for spammers to automatically create new splogs which are not on the blacklist. The spam classifier uses a segmenter which aggressively splits the url (Section 3), and then the token sequence is used in a supervised learning framework for classification (Section 4). 3
URL segmentation
The url segmentation problem at a first glance looks deceptively simple: given a sequence of words without spaces between them identify the words. But what is a word? One of the approaches considered below will take the stance that a word is an entry in a dictionary. Yes, in advance we already know that dictionaries don’t list all words. Another question is even if somehow all possible ‘words’ which are substrings of the initial string are identified (some words can overlap with others) how does the system pick the right sequence of consecutive words. If an input token has n characters there can be 2n−1 possible segmentations (there are n−1 positions and each position is a potential break point). Thus, due to their large number considering all possible segmentations and picking the best one is not practical. The segmenter first does the easy job—it tokenizes the urls on puctuation symbols (., -, , /, ?, =). Then the current url tokens are examined for further possible segmentation. We explore a number of techniques: 1. dictionary-based; 2. symmetric sliding window: 6-gram with single break, and 6-gram sliding window with a backoff of a symmetric 4-gram (again single break); 3. non-symmetric, multi-break, 7-gram sliding window with 6-, and 5-gram backoff; and 4. contextual.
For all techniques we first generalize certain characters (e.g., digits). 3.1
Dictionary-based URL segmentation
With the dictionary-based approach the segmenter goes through the current token from the left to right and in a greedy manner looks for the longest match with a word in a known set of words. If a match is found the current position advances after the match; otherwise the current position moves
SPAM CLASSIFICATION OF WEBLOGS
• ↑ ini pos
•
abc |{z}
matched word
129
remaining characters
↑ new pos
Fig. 1: Forward maximum match
to the next character. Such an algorithm is also called forward maximum match. Segmenting known words that are pre-defined in a dictionary is often referred to as ‘word breaking’ (Gao et al. 2005). Two issues with the above formulation are: 1. No dictionary is exhaustive. In particular, dictionaries don’t include all morphological variations (especially productive morphological processes); 2. The greedy technique might initially select a longer word while the solution might involve a shorter form (loved•ays vs. love•days; we use the symbol ‘•’ to indicates a point where a break occurs or is predicted).
To address these issues we consider extending a found match: ing, ed, s, doubling final consonant, etc. and backtracking to explore shorter matches. The maximum match idea can be applied backwards from right to left (backward maximum match). 3.2
Symmetric sliding window
The segmenter uses a sliding window of n characters (we used n = 6). Going from left to right the segmenter decides whether to split after the current third character. The segmentation decisions are based on counts collected 1.
d i e ? t s t hatwork
2.
d i e ◦ t s thatwork
3.
d i e t ◦ s t i atwork
4.
di e t s • t h a twork
5.
diets • t h a ? t w o rk
6.
diets • t h a ◦ t w ork
7.
diets • t h a t • w o r k
Fig. 2: Workings of the symmetric sliding window with back-off
130
NICOLAS NICOLOV & FRANCO SALVETTI
during training. Figure 2 illustrates the initial steps of the processing of www.dietsthatwork.com when considering the token dietsthatwork. The ‘?’ indicates that the left and right tri-grams have not been encountered in the training data; the symbol ‘◦’ indicates that the left and right tri-grams are kept together; and as above ‘•’ indicates a break. For example, in the case of d i e ? t s t on line 1 in Figure 2 the algorithm considers how many times we have seen in the training data the n-gram ‘dietst’ vs. ‘die tst’. If both counts are zero (as on line 1) the 6-gram symmetric sliding window algorithm simply moves to the next character position. An extension we have explored backs-off to a more general symmetric context of 4-gram characters (shown on lines 2 and 6). If the counts for the 4-grams are zero the default decision is not to split at the current position. 1. i m i n h e a ven 2. i m i n h e aven 3. i m i n h eaven 4. i m i n heaven 5. • i • m • in • h e a v e n 6. • i • m • in • • heaven • 7. i • m • in • heaven
Fig. 3: Workings of the multi-break segmenter
3.3
Non-symmetric, multi-break
Investigating the output of the symmetric window approach we observed that the contexts were too specific—precision was high but recall was low (as we will see in Section 5). Hence, we considered the 4-gram back-off. This substancially improved recall but precision was lowered as the 4-gram contexts are beginning to be too general. There are non-symmetric contexts which are not as general—after the symmetric 6-gram A B C - X Y Z we considered non-symmetric contexts A B C - X Y and B C - X Y Z . We generalized this notion of non-symmetric contexts and allowed multiple breaks to be predicted simultaneously (including at the boundaries). We also started with longer contexts of 7-grams. This algorithm also addressed cases of short words as in: www . i • m • in • heaven . com. Figure 3 exemplifies the entire processing. In the step between lines 6 and 7 the system simply removes consecutive breaks and breaks at the boundaries.
SPAM CLASSIFICATION OF WEBLOGS
feature L known word L known word (morph. extended) R known word L known suffix L maximal known suffix R known prefix Lw & Rw Ls & Rp L & R char unigrams L & R char bigrams L & R char trigrams L bigram & R trigram (char) L trigram & R bigram (char)
131
abbr Lw Lmw Rw Ls Lms Rp LwRw LsRp 1-1 2-2 3-3 2-3 3-2
Table 1: Segmentation features for MaxEnt; L = ‘to the left there is a . . . ’; R = ‘to the right there is a . . . ’ 3.4
Contextual
Finally we considered an approach where diverse information about the segmentation can be used by the system in a unified way. We employed a Maximum Entropy framework and at a position between two characters in the string we considered the following features (contextual predicates) again motivated by error analysis of the previous systems as well as linguistic intuitions (cf. Table 1). feature Lw Lmw Rw Ls Lms Rp LwRw LsRp
example complex|andconfused weekly |tipsandtricks speed|racer wellness|formulas institutionalized|models quick|antiaging keydata|recovery dating|unleashed
Table 2: Examples for the MaxEnt features Table 2 shows examples of the word and affix features. The ‘|’ symbol indicates the current position. The underlined substrings refer to the triggers for the corresponding features. ‘Lmw’ refers to there being a morphological variant (extension) of a known word to the left of the current position, e.g., weekly |. . . ; a maximal suffix is a suffix which is not a prefix of another
132
NICOLAS NICOLOV & FRANCO SALVETTI
suffix (ing is a suffix but it is not maximal because of the composite suffix ings). Note how the character n-grams subsume the symmetric sliding window approaches above in Section 3.2. 4
URL classification
For spam classification a Na¨ıve Bayes classifier is used. Given a token sequence T = ht1 , . . . , tn i, as provided by the segmenter, the class cˆ ∈ C = {spam, good} is decided as:
cˆ = arg max P (c|T ) = arg max c∈C
c∈C
P (c) · P (T |c) P (T )
= arg max P (c) · P (T |c) c∈C
= arg max P (c) · c∈C
n Y
P (ti |c)
i=1
In the last step we assume conditional independence between the features. We use simple Laplace smoothing (add one) and the individual probabilities are computed as: P ∗ (ti |c) =
count(ti , c) + 1 Nc + Vc
where: count(ti , c) is the number of time the token ti occurred in in the collection for category c; Nc is the total number of tokens in the collection for category c; Vc is the number of distinct tokens in the collection for category c. We have also explored simple voting techniques for determining the class cˆ:
cˆ =
(
spam, if sgn
Pn
i=1
sgn (P (ti|spam) − P (ti |good)) = 1
good, otherwise
Because we are interested in having control over the precision of the classifier we introduce a score meant to be used for deciding whether to label a url as unknown. P (spam|T ) − P (good|T ) score(T ) = P (spam|T ) + P (good|T )
SPAM CLASSIFICATION OF WEBLOGS
133
If the score(T ) exceeds a certain threshold τ we label T as spam or good using the greater probability of P (spam|T ) or P (good|T ). To control the accuracy of the classifier we can tune τ . For instance, in order to achieve 93.3% of accuracy we set τ = 0.75 which implied a recall of 50.9%. # of splits 1 2 3 4 5 6 8 Total
# spam URLs 2,235 868 223 77 2 4 3 3,412
# good URLs 2,274 459 46 7 1 1 2,788
Table 3: Number of splittings in a URL 5
Experiments and results
First we discuss the segmenter. 10,000 spam and 10,000 good weblog urls and their corresponding webpages were used for the experiments. The 20,000 weblog HTML pages are used to induce the segmenter. The first experiment was aimed at finding how common was segmentation as a phenomenon. The segmenter was run on the actual training urls. The number of urls that are additionally segmented besides the segmentation on punctuation are reported in Table 3. The multiple segmentations need not all occur on the same token in the url after initial segmentation on punctuation. The various segmenters were then run on a separate test set of 1,000 urls for which the ground truth for the segmentation was marked. For the MaxEnt model we used a feature count cutoff of 7—this affects mostly the lexical and chaacter n-gram features. The results are in Table 4. OOV System Dict (no OOV) Dict 6-gram 6-gram & 4-gram Multi-break Contextual
Precision 95.79% 43.33% 84.31% 70.92% 82.67% 87.96%
Recall 97.13% 87.86% 48.84% 75.85% 90.18% 91.07%
F-measure 96.45% 58.04% 61.85% 73.30% 86.26% 89.48%
Table 4: Performance of the segmenter
134
NICOLAS NICOLOV & FRANCO SALVETTI
means ‘out-of-vocabulary’ and refers to the dictionary-based system which knows all the tokens but uses a greedy search. Figure 4 shows long tokens which are correctly split. cash • for • your • house unlimitted • pet • supllies jim • and • body • fat weight • loss • product • info kick • the • boy • and • run bringing • back • the • past food • for • your • speakers
Fig. 4: Correct segmentations The spam classifier was then run on the test set. The results are shown in Table 5. accuracy prec. spam rec. spam f-measure spam prec. good rec. good f-measure good
78% 82% 71% 76% 74% 84% 79%
Table 5: Classification results The performance of humans on this task was also evaluated. Eight individuals performed the spam classification just looking at the unsegmented urls. The scores for the human annotators are given in Table 6. The average accuracy of the humans (76%) is slightly lower than that of the system (78%). From an information retrieval perspective if only 50.9% of the urls are retrieved (labelled as either spam or good and the rest are labelled as unknown) then of the spam/good decisions 93.3% are correct. This is relevant for cases where a url spam filter is in cascade followed by, for example, content-based spam classifier. 6
Discussion
The system performs better with the aggressive segmentation because the system has been forced to smooth on fewer occasions. For instance given
SPAM CLASSIFICATION OF WEBLOGS
accuracy precision spam recall spam f-measure spam precision good recall good f-measure good
Mean 76% 83% 65% 73% 71% 87% 78%
135
σ 6.71 7.57 6.35 7.57 6.35 6.39 6.08
Table 6: Results for the human annotators; µ is the mean; σ is the standrd deviation the input url www.ipodipodipod.com in the system which segments solely on punctuation both the spam and the good model will have to smooth and the results depend merely on the smoothing technique. Even if we reached the average scores of humans we expect to be able to improve the system further as the maximum accuracy among the human annotators is 90%. Among the errors of the segmenter the most common were related to plural nouns (‘girl•s’ vs. ‘girls’) and past tense of verbs (‘dedicate•d’ vs. ‘dedicated’) whereas common mistakes for the url classifier involve urls that do not contain enough information even for humans to classify correctly. Kolari et al. (2006) mention classification of spam weblogs based on urls. Mishne et al. (2005) present work on spam comment identification. Shih & Karger (2004) use urls for classification. 7
Future work
We are exploring other classifiers for the contextual segmentation model, including Robust Risk Minimization, Support Vector Machines, etc. For the segmentation task we are also exploring finite state approaches. As mentioned above we consider the url spam detector as a filter followed by content models that consider the text in the weblog, structural characteristics of the HTML, the outward links from a page, etc. 8
Conclusions
We have presented techniques for identifying whether a url indicates spam or not. We considered an approach where we initially segment the url and then perform the classification. The technique is simple and efficient, yet very effective—our system reaches an accuracy of 78% while humans perform at 76%.
136
NICOLAS NICOLOV & FRANCO SALVETTI
REFERENCES Andras, T. S., A. Bencz´ ur, K´ aroly Csalog´ any & M. Uher. 2005. “SpamRank — Fully Automatic Link Spam Detection”. 1st Int. Workshop on Adversarial Information Retrieval on the Web, WWW-2005. Gao, J., M. Li & C.-N. Huang. “Chinese Word Segmentation and Named Entity Recognition: A Pragmatic Approach”. Computational Linguistics 31:4.531574, December 2005. Gy¨ ongyi, Z., H. Garcia-Molina & J. Pedersen. 2004. “Combating Web Spam with TrustRank”. 30th Int. Conference on Very Large Data Bases (VLDB), 271-279. Toronto, Canada. Hurst, Matthew. 2005. “Deriving Marketing Intelligence from Online Discussion”. 11th ACM SIGKDD Int. Conference on Knowledge Discovery in Data mining (KDD’05 ), 419-428. Chicago, Illinois, U.S.A. Kan, M.-Y. 2004. “Web Page Classification without the Web Page”. 13th Int. World Wide Web Conference (WWW’2004 ), New York. poster. Kan, M.-Y. & H. O. N. Thi. 2005. “Fast Webpage Classification Using URL Features”. Conference on Information and Knowledge Management (CIKM’05 ), 325-326, Bremen, Germany. Kolari, P., T. Finin & A. Joshi. 2006. “SVMs for the Blogosphere: Blog Identification and Splog Detection”. Computational Approaches to Analyzing Weblogs, number SS-06-03, pages ed. by N. Nicolov, F. Salvetti, M. Liberman & J. H. Martin, 92-99. Menlo Park, Calif.: AAAI Press. Mishne, Gilad, D. Carmel & R. Lempel. 2005. Blocking Blog Spam with Language Model Disagreement. 1st International Workshop on Adversarial Information Retrieval on the Web (AIRWeb’05), at WWW2005, ed. by B. D. Davison, 1-6. Chiba, Japan. Shih, L.K. & D. Krager. 2004. “Using URLs and Table Layout for Web Classification Tasks”. 13th Int. World Wide Web Conference (WWW’2004 ), New York. SURBL. 2006. “Surbl – Spam Uri Realtime Blocklists”. http://www.surbl.org. Technorati. 2006. “State of the Blogosphere February 2006 part 1: On Blogosphere Growth”. http://technorati.com/weblog/2006/ 02/81.html. Wikipedia. 2006. “Splog (spam blog)”. http://en.wikipedia.org/wiki/Splog.
Document Classification Using Semantic Networks with an Adaptive Similarity Measure Filip Ginter, Sampo Pyysalo & Tapio Salakoski Turku Centre for Computer Science (TUCS ) University of Turku, Finland Abstract We consider supervised document classification where a semantic network is used to augment document features with their hypernyms. A novel document representation is introduced in which the contribution of the hypernyms to document similarity is determined by semantic network edge weights. We argue that the optimal edge weights are not a static property of the semantic network, but should rather be adapted to the given classification task. To determine the optimal weights, we introduce an efficient gradient descent method driven by the misclassifications of the k-nearest neighbors (kNN) classifier. The method iteratively adjusts the weights, increasing or decreasing the similarity of documents depending on their classes. We thoroughly evaluate the method using the kNN classifier and show it to statistically significantly outperform the bag-of-words representation as well as the more advanced hypernym density representation of Scott & Matwin (1998).
1
Introduction
Semantic networks have been shown to offer opportunities for improving the performance of machine learning methods in a variety of classification tasks. Several semantic similarity measures have been proposed and applied, in particular to word sense disambiguation-type problems (see e.g., Patwardhan, Banerjee & Pedersen (2003) for a recent evaluation). Methods applying semantic networks to document classification have also been proposed, although they are not as widely studied. For instance, semantic networks have been used to augment terms with their synonyms (G´omezHidalgo & deBuenaga Rodr´ıguez 1997) and hypernyms (Scott & Matwin 1998), thus incorporating the semantic network information on the level of features. Here we consider the special case of similarity through the hyponymy/hypernymy relation, which is the focus of most proposed measures of semantic relatedness. The similarity of terms is typically presented as a static property that can be directly measured either from the semantic network (Agirre & Rigau 1996), from external unlabeled data (Resnik 1995), or using a combination of the two (Jiang & Conrath 1997). We have previously
138
FILIP GINTER, SAMPO PYYSALO & TAPIO SALAKOSKI
air traveller astronaut
traveller space traveller cosmonaut
Fig. 1: Hyponymy relationships in a semantic network
argued that in supervised classification tasks the similarity of terms should be considered dependent on the task and data (Ginter, Pyysalo, Boberg, J¨arvinen & Salakoski 2004). Simply put, terms commonly related to documents of the same class should be considered similar, while terms related to documents of different classes should be considered dissimilar to aid the classification method in distinguishing between the classes. Consider the fragment of a semantic network shown in Figure 1. Common measures of semantic similarity would assign high relatedness to the terms astronaut and cosmonaut as they are immediate hyponyms of the same term, space traveller. In most classification tasks, considering astronaut and cosmonaut essentially synonymous terms would be appropriate. In a document representation, this can be naturally realized by considering the term space traveller to be highly relevant to documents containing either of its two hyponyms. However, we suggest that in a hypothetical document classification task where the goal is to distinguish between documents about American and Russian space efforts, space traveller should not be considered relevant to documents containing either astronaut or cosmonaut in order to avoid increasing the similarity between documents of different classes. We now discuss some desirable properties for a data-dependent semantic document representation and means of realizing them. We assume that each document has been assigned a set of direct terms from a semantic network (e.g., terms that are mentioned in the document). The representation should then determine the relevance of each semantic network term to each document. It is natural to limit this measure of relevance between 0 and 1, and to assign the value 1 to each direct term. Hypernyms of direct terms are typically relevant and their relevance values should be allowed to vary in a data-dependent fashion. We suggest that terms that are neither direct terms nor hypernyms of direct terms in a document are not relevant to that document and can be assigned the relevance value 0: for example, if astronaut is the only direct term, there is no reason to assume that either cosmonaut or air traveller are relevant. Finally, relevance should not increase with distance from the direct term: if, for example, astronaut is the only direct term, traveller should be considered at most as relevant as space traveller. This implies a representation where relevance propagates
DOCUMENT CLASSIFICATION USING SEMANTIC NETWORKS
139
from direct terms to more general terms, decreasing according to the datadependent strengths of connections between hyponyms and hypernyms. We have previously introduced a data-driven method for determining hypernym relevance for document classification, where relevance was limited to the two cases ‘fully relevant’ and ‘irrelevant’ (Ginter, Pyysalo, Boberg, J¨arvinen & Salakoski 2004). In this paper, we present a method that applies a finer-grained concept of relevance and shows a more substantial performance advantage. 2
Document representation
Let T be a set of possible terms that are organized in a semantic network according to the hyponymy relation. Let t, t′ ∈ T be terms. We denote by t′ ≺⋆ t the relation when t′ is a hyponym of t. Further, t′ ≺ t denotes the relation when t′ is an immediate hyponym of t, that is, the relation encoded by the semantic network. Hyponymy (≺⋆ ) is the transitive closure of immediate hyponymy (≺). The immediate hyponymy relation is commonly represented using a directed graph, such as in Figure 1, with an edge from t′ to t whenever t′ ≺ t. Hyponymy (≺⋆ ) is by definition an asymmetric relation, and the corresponding directed graph is thus acyclic. We now define a document representation that implements the intuitions discussed in Section 1. Let D be a set of documents and let d ∈ D be a document with the set of direct terms T (d) ⊆ T . As discussed previously, the document d is represented by the direct terms T (d) and their hypernyms. We implement this property through the notion of activation at (d) ∈ [0, 1] of a term t ∈ T with respect to the document d, which represents the relevance of t to d. For any t ∈ T (d), at (d) is by definition set to 1. The activation of any other term recursively depends on the activations of its immediate hyponyms so that the activation of hypernyms of direct terms typically results in a non-zero value. The activation of the remaining terms is zero by definition. A term t ∈ T is affected by a document d if t ∈ T (d) or ∃ t′ ∈ T (d) : t′ ≺⋆ t. That is, t is affected by d if t is either a direct term of d or a hypernym of a direct term. The set of terms affected by a document d is denoted Aff (d). Let further the base of a term t ∈ T with respect to a document d be Baset (d) = {t′ | t′ ≺ t, t′ ∈ Aff (d)}. Unless t is a direct term, at (d) is based on the activations of the terms in Baset (d). For each t′ ∈ Baset (d), the contribution of t′ to the activation of t is controlled by a weight wt′ t associated with the relationship t′ ≺ t. By definition, 0 ≤ wt′ t ≤ 1 for all weights. The activation at (d) is computed as the weighted sum of the activations of the terms in Baset (d). Thus,
140
FILIP GINTER, SAMPO PYYSALO & TAPIO SALAKOSKI
1/4
Aff (d) = {e, a, k, b, r, f } , Baser (d) = {a, b} , Basea (d) = {e} ,
r
1/2
Baseb (d) = {e, f } , Basef (d) = {k} , Baseg (d) = Baseh (d) =
1 a 0
b 1 1
d
f 1
h
= Basej (d) = Basek (d) = Basel (d) = Based (d) = Basee (d) =
1
e 1
g
c
1/2 j
1/2 k
1 l
= Basec (d) = ∅, ae (d) = aa (d) = ak (d) = 1, ad (d) = ag (d) = 1 = ah (d) = aj (d) = al (d) = ac (d) = 0, af (d) = wkf ak (d) = , 2 web ae (d) + wf b af (d) 3 war aa (d) + wbr ab (d) 1 = , ar (d) = = ab (d) = 2 4 2 2
Fig. 2: The representation of a document with direct terms T (d) = {e, a, k} (bold circles). Terms affected by d are depicted in gray
at (d) =
1P
if t ∈ T (d), wt′ t at′ (d) |Baset (d)|
t′ ∈Baset (d)
0
if t ∈ Aff (d) \ T (d), otherwise.
(1)
Finally, each document d is represented by an activation vector a(d) = (at1 (d), . . . , atm (d)), where ti ∈ T . Figure 2 illustrates the concepts introduced so far. 3
Weight update algorithm
a(d) . Let a ˆ(d) be the normalized activation vector of d, that is, aˆ(d) = ka(d)k The similarity between any two documents di , dj ∈ D is defined as the dot-product of their normalized activation vectors X sim(di, dj ) = a ˆ(di ) · a ˆ(dj ) = a ˆt (di )ˆ at (dj ) . (2) t∈T
Given a training set of documents D, a document similarity measure, and a document d to be classified, the k-nearest neighbors (kNN) classifier computes a set N(d, k, D) ⊆ D \ {d} of k documents most similar to d, also termed as the k-neighborhood. The class assigned to d is the majority class among the documents in its k-neighborhood. The weight update algorithm is based on kNN and implements the following intuition: a misclassification of a document d means that the majority of the documents in N(d, k, D) are of a different class than d. The misclassification could therefore be corrected by modifying the k-neighborhood so that it would contain a majority of documents with the same class as
DOCUMENT CLASSIFICATION USING SEMANTIC NETWORKS
141
that of d. This can be achieved by adjusting the semantic network weights so that the similarity between d and its k-neighbors with a different class decreases and with the same class increases. As there is only one, global set of weights, any change affects all the documents and therefore directly optimizing the similarity of d with its k-neighbors also indirectly affects the similarity of d with all other documents. Generally, documents with the same class are “pulled” towards d while documents with another class are “pushed” away from d. Naturally, this effect is strongest for the k-neighbors of d, whose similarity with d is optimized directly. Other variations of the general scheme are possible as well. For example, the k-neighborhoods could be optimized for all documents rather than only for those that were misclassified. Let us consider two documents di , dj ∈ D. The objective is to either increase or decrease sim(di , dj ) by modifying the semantic network weights. Let w = (w1 , . . . , wn ) , where n is the total number of weights, be the vector of all weights in the semantic network in an arbitrary but fixed order. We then define the weight gradient ∇w(di , dj ) with respect to sim(di , dj ) as ∂ sim(di , dj ) ∂ sim(di , dj ) ,..., . ∇w(di , dj ) = ∂w1 ∂wn Adding the gradient ∇w(di , dj ) to the weight vector w leads to an increase of sim(di , dj ), while subtracting ∇w(di , dj ) from w leads to a decrease of ∂ sim(di , dj ) sim(di , dj ). The formula to compute the partial derivative of ∂ wrs sim(di , dj ) with respect to a weight wrs is specified jointly by Equations 3, 7, and 8 in Appendix 1 which also details the derivation of the formula. A learning rate constant η ∈ R, η > 0, is introduced to control the magnitude of the weight adjustment by the gradient. The weight vector w is then updated according to the rule w ← w + δη∇w(di , dj ) , where δ = +1 (resp. δ = −1) if sim(di , dj ) is to be increased (resp. decreased). The complete weight update algorithm is introduced in Figure 3. 4
Evaluation
We evaluate the proposed representation using ten datasets consisting of articles from the PubMed biomedical literature database, where each article has been manually assigned a set of relevant terms from the MeSH ontology (we use the 2005 version). Each of the ten datasets, corresponding to the randomly selected journals in Ginter, Pyysalo, Boberg, J¨arvinen & Salakoski (2004), consists of
142
FILIP GINTER, SAMPO PYYSALO & TAPIO SALAKOSKI w←¯ 1 until stopping criterion satisfied do: w′ ← ¯ 0 for each document di ∈ D: classify di using D \ {di } as training set if misclassified di then: for each dj ∈ N (di , k, D \ {di }): if class (di ) = class (dj ) then δ ← +1 else δ ← −1 w′ ← w′ + δ∇w(di , dj ) w ← w + η · w′ for each weight wk in w: wk ← max{0, min{1, wk }}
Fig. 3: Pseudocode of the weight update algorithm
2000 randomly selected articles from the journal and 2000 randomly selected articles that have appeared elsewhere. Additionally, we form for each dataset seven down-sampled training sets, the largest consisting of 1000 positive and 1000 negative examples (the other 2000 being used for testing). Each task is then a binary classification problem where the documents must be classified either as originating from the journal or not. Since the journals are usually focused on a subdomain, these classification problems model document classification by topic. We evaluate the proposed document representation with and without the adaptive component. In the fixed representation, the semantic network weights are all set to one constant value wf ix , 0 ≤ wf ix ≤ 1, determined from the data. In the adaptive representation, the weights are computed using the algorithm introduced in Section 3, with a stopping criterion where iteration ends when the average performance increase on the training set over the last three rounds drops below 0.05%. We compare the performance of the fixed and adaptive representations against two baselines, the commonly used bag-of-words (BoW) representation and a modification of the hypernym density (HD) representation of Scott & Matwin (1998) in which each document di is represented by a multiset consisting of all direct terms of di , together with their hypernyms up to a distance h from any of the direct terms. We found that in our case coercing the multiset into a set results in an improvement of performance, and thus we apply this step in our evaluation. Further, infrequent terms are not discarded and the HD normalization step is performed by the classifiers. The main evaluation is performed using the kNN classifier. The parameters of the various methods — k for BoW, k, h for HD, k, wf ix for fixed, and k, η for adaptive — are selected by cross-validated grid search on the training set. We also perform a limited evaluation using Support Vector
5
8 7
6
3
8
4
5
8
2
0
1
2
3
4
1 1 1 0
1
3
0
0
8
5
6
5 3
4 2
5
3 3 2 3 5
3
5
143
Adaptive vs. BoW HD vs. BoW Fixed vs. BoW Adaptive vs. Fixed Adaptive vs. HD HD vs. Fixed
5
3 5 4
0
1
1
0
3 2
125
250
500
1000
2000
0
Absolute difference [percentage units]
DOCUMENT CLASSIFICATION USING SEMANTIC NETWORKS
31
62
Training set size [number of documents]
Fig. 4: Pairwise method differences and their per-dataset and overall statistical significances for kNN. Results averaged over all datasets. The number displayed by each difference denotes the number of datasets for which the difference was statistically significant (p < 0.05, 5 × 2cv test). Full — as opposed to empty — circle denotes the average difference over all ten datasets being statistically significant (p < 0.05, t-test) Machines (SVM) (Vapnik 1998). For this evaluation, only the SVM regularization parameter C is separately selected, while other parameters are set to their kNN optimum values. We measure the performance using average 5 × 2 cross-validated accuracy. To assess the statistical significance for individual datasets, we use the robust 5 × 2 cross-validation test of Alpaydin (1999). To assess the overall significance across all datasets, we use the two-tailed paired t-test. Average performance differences when using kNN are plotted in Figure 4 and average absolute performance figures are given in Table 1a. The adaptive method statistically significantly outperforms all others for all except the smallest training set size. Further, the relative decrease in error rate grows almost monotonically, indicating that the adaptive method works better given more data. As the documents were assigned on average only 10 MeSH terms and the MeSH ontology contains almost 23000 nodes, reliable optimization of the edge weights is expected to be difficult with very small training sets. Nevertheless, the adaptive method works remarkably well with as few as 62 training examples. When applied to SVMs, the fixed and HD representations outperform the adaptive method (Figure 1b). The difference is statistically significant for most training set sizes larger than 62. Clearly, the adaptive method does not optimize a criterion beneficial for SVM classification, and hence modification of the adaptive strategy is required to increase applicability
144
FILIP GINTER, SAMPO PYYSALO & TAPIO SALAKOSKI BoW HD
31 62 125 250 500 1000 2000
72.5 73.9 75.9 77.8 79.6 81.0 82.4
74.8 77.3 79.4 81.8 83.4 84.9 85.9
∆
Fix.
∆
Ad.
∆
8.4 13.0 14.5 18.0 18.6 20.5 19.9
74.4 77.2 79.1 81.4 83.1 84.5 85.7
6.9 12.6 13.3 16.2 17.2 18.4 18.8
75.9 79.6 82.0 84.0 85.4 86.6 87.4
12.4 21.8 25.3 27.9 28.4 29.5 28.4
(a) kNN
BoW HD
75.6 78.2 80.9 83.2 85.4 87.5 88.9
79.1 81.6 84.0 85.8 87.5 88.9 90.0
∆
Fix.
∆
14.3 15.6 16.2 15.5 14.4 11.2 9.9
78.6 81.5 84.3 85.9 87.8 89.2 90.3
12.3 15.1 17.8 16.1 16.4 13.6 12.6
Ad.
∆
78.1 10.2 80.9 12.4 83.0 11.0 84.7 8.9 86.7 8.9 88.4 7.2 89.7 7.2
(b) SVM
Table 1: Accuracy measurements averaged over all ten datasets for each of the seven training set sizes. The ∆ columns represent the relative error rate decrease in percents over the BoW baseline
to SVM classification. Nevertheless, the fixed representation outperforms both the BoW and HD representations for larger training set sizes (the latter difference is mostly not statistically significant), indicating that the document representation itself benefits also SVMs. 5
Conclusions and future work
We have applied semantic networks to develop an adaptive document representation. We have evaluated the representation and the algorithm against the BoW, fixed and HD representations with ten randomly selected datasets from the PubMed biomedical literature database. Our results indicate that the proposed adaptive representation can statistically significantly outperform the baselines over a range of training set sizes from 62 to 2000, with the relative decrease in error rate ranging between 20–30% against BoW and 10–14% against the fixed and HD representations. A preliminary evaluation with SVMs indicated that while the semantic network-based document representations give a statistically significant improvement over the BoW baseline, the gradient descent component of the adaptive method requires modification. We conclude that the proposed method can successfully determine termdocument relevance in a data-dependent manner, increasing performance in supervised document classification tasks. As future work, several aspects of the method can be studied, such as the setting of the initial weights, the learning rate, and the stopping criterion. An additional natural extension of the method is to consider relationships other than hyponymy as activation paths. Careful analysis of these and other properties may offer further opportunities for the use of semantic networks in document classification.
DOCUMENT CLASSIFICATION USING SEMANTIC NETWORKS
145
REFERENCES Agirre, Eneko & German Rigau. 1996. “Word Sense Disambiguation Using Conceptual Density”. Proceedings of the 16th Conference on Computational Linguistics, Copenhagen, Denmark (COLING’96 ), 16-22. San Francisco: Morgan Kaufmann. Alpaydin, Ethem. 1999. “Combined 5 × 2 cv F -Test for Comparing Supervised Classification Learning Algorithms”. Neural Computation 11:8.1885-1892. Ginter, Filip, Sampo Pyysalo, Jorma Boberg, Jouni J¨ arvinen & Tapio Salakoski. 2004. “Ontology-Based Feature Transformations: A Data-Driven Approach”. Proceedings of the 4th International Conference on Advances in Natural Language Processing, Alicante, Spain (EsTAL’04 ), ed. by Jos´e L. Vicedo et al. (= Lecture Notes in Computer Science, 3230 ), 279-290. Heidelberg: Springer. G´omez-Hidalgo, Jos´e M. & Manuel de Buenaga Rodr´ıguez. 1997. “Integrating a Lexical Database and a Training Collection for Text Categorization”. Proceedings of the ACL/EACL Workshop on Automatic Information Extraction and Building of Lexical Semantic Resources, Madrid, Spain, ed. by Piek Vossen et al., 39-44. Association for Computational Linguistics. Jiang, Jay J. & David W. Conrath. 1997. “Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy”. Proceedings of the International Conference on Research in Computational Linguistics, Tainan, Taiwan (ROCLING’97 ), 19-33. Patwardhan, Siddharth, Satanjeev Banerjee & Ted Pedersen. 2003. “Using Measures of Semantic Relatedness for Word Sense Disambiguation”. Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, Mexico City, Mexico (CICLing’03 ), ed. by Alexander F. Gelbukh, 241-257. Heidelberg: Springer. Resnik, Philip. “Using Information Content to Evaluate Semantic Similarity in a Taxonomy”. 1995. Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, Canada (IJCAI’95 ), ed. by Christopher Mellish, 448-453. San Francisco: Morgan Kaufmann. Scott, Sam & Stan Matwin. “Text Classification Using WordNet Hypernyms”. 1998. Proceedings of the COLING-ACL Workshop on Use of WordNet in Natural Language Processing Systems, Montreal, Canada, ed. by Sanda Harabagiu, 38-44. Association for Computational Linguistics. Vapnik, Vladimir. 1998. Statistical Learning Theory. New York: Wiley.
Appendix 1 Derivation of the formula for
∂ sim(di , dj ) ∂ wrs
This appendix details the derivation of the formula to compute the value of ∂ sim(di , dj ) . The partial derivative is solved and the final formula is obtained ∂ wrs jointly from Equations 3, 7, and 8.
146
FILIP GINTER, SAMPO PYYSALO & TAPIO SALAKOSKI
X∂a ˆt (dj ) ∂ sim(di , dj ) X ∂ aˆt (di ) = a ˆt (dj ) + a ˆt (di) ∂ wrs ∂ w ∂ wrs rs t∈T t∈T
(3)
def
= Q(di , dj ) + Q(dj , di )
We now solve Q(di , dj ); the formula for Q(dj , di ) follows by symmetry. a (d )
t i ∂ at (di ) i )k ka(di )k − at (di ) ∂ ka(d ∂ aˆt (di ) ∂ ka(di )k ∂ wrs = = ∂ wrs ∂ wrs ∂ wrs ka(di )k2 qP 2 ∂ X ∂ ka(di )k 1 ∂ au (di) u∈T [au (di )] = = au (di ) ∂ wrs ∂ wrs ka(di )k u∈T ∂ wrs
Combining (4) and (5) yields ∂ aˆt (di ) = ∂ wrs
∂ at (di ) ka(di )k ∂ wrs
−a ˆt (di )
P
u∈T
ka(di)k2
i) au (di ) ∂ ∂auw(d rs
(4)
(5)
(6)
Substituting from (6) into (3) gives Q(di , dj ) = P P P ∂ at (di ) ∂ au (di ) ka(d )kˆ a (d ) − a ˆ (d )ˆ a (d ) a (d ) i t j t j t i u i t∈T ∂ wrs t∈T u∈T ∂ wrs = (2)
=
u|t
=
=
P
∂ at (di ) at (dj ) t∈T ∂ wrs ka(di )kˆ
P
∂ at (di ) at (dj ) t∈T ∂ wrs ka(di )kˆ
P
P
t∈T
t∈T
ka(di )k2 P P i) − u∈T au (di ) ∂ ∂auw(d ˆt (dj )ˆ at (di ) t∈T a rs ka(di)k2 P − u∈T au (di ) ∂ ∂auw(drsi ) sim(di , dj )
(7)
ka(di)k2 ∂ at (di ) ∂ wrs
(ka(di )kˆ at (dj ) − at (di ) sim(di, dj )) ka(di)k2
∂ at (di ) ∂ wrs
(ˆ at (dj ) − a ˆt (di ) sim(di , dj )) ka(di )k
If t ∈ / Aff (di )\T (di) then at (di) is by (1) constant and consequently ∂∂awt (drsi ) = 0. For t ∈ Aff (di) \ T (di ), ( X ar (di ) if (t′ , t) = (r, s) 1 ∂ at (di ) (1) = (8) ∂ at′ (di ) ∂ wrs |Baset (di )| ′ otherwise. wt′ t ∂ wrs t ∈Baset (di )
The recursion in (8) ends when (t′ , t) = (r, s). Finally, substituting from ∂ sim(d , d ) (8) into (7) and then into (3) completes the derivation of ∂ wrsi j .
Text Summarization for Improved Text Classification Rada Mihalcea & Samer Hassan University of North Texas Abstract This paper explores the possible benefits of the interaction between text summarization and text classification. Through experiments performed on standard data sets, we show that techniques for extractive summarization can be effectively combined with classification methods, resulting in improved performance in a text categorization task. Moreover, comparative results suggest that the synergy between text summarization and text classification can be regarded as a new task-oriented evaluation testbed for automatic summarization.
1
Introduction
Text categorization is a problem typically formulated as a learning task, where a classifier learns how to distinguish between categories in a given set, using features automatically extracted from a collection of training documents. In addition to the learning methodology itself, the accuracy of the text classifier also depends to a large extent upon the classification granularity, and on how well separated are the training or test documents belonging to different categories. For instance, it may be a relatively easy task to learn how to classify documents in two distinct categories such as computer science and music, but it may be significantly more difficult to distinguish between documents pertaining to more closely related topics such as operating systems and compilers. Intuitively, if the gap between categories could be increased, the classification performance would raise accordingly, since the learning task would be simplified by removing features that represent potential overlap between categories. This is in fact the effect achieved through feature weighting and selection (Yang & Pedersen 1997, Ng et al. 1997), which was found to improve significantly over the case where no weighting or selection is performed. We propose a new approach for reducing the potential overlap between documents belonging to different categories, by using a method that extracts the essence of a text prior to classification. In this way, only the important sections of a document participate in the learning process, and thus the performance of the text classification algorithm could be improved. Through experiments on standard test collections, we show that significant improvements can be achieved on a text categorization task by classifying extractive summaries, rather than entire documents. We believe
148
RADA MIHALCEA & SAMER HASSAN
that these results have implications not only on the problem of text classification, where the method proposed provides the means to improve the categorization accuracy with error reductions of up to 19.3%, but also on the problem of text summarization, by suggesting a new application-based evaluation of tools for automatic summarization. 2
Text categorization using extractive summarization
Provided a set of training documents, each document assigned with one or more categories, the task of text categorization consist of finding the most probable category for a new unseen document, based on features extracted from training examples. The classification process is typically performed using information drawn from entire documents, and this may sometime result in noisy features. To lessen this effect, we propose to feed the text classifier with summaries rather than entire texts, with the goal of removing the less-important, noisy sections of a document prior to classification. The text classification process is thus modified to integrate an extractive summarization tool that determines the top N most important sentences in each document. Starting with a collection of texts, every document is replaced by its summary, followed by the application of a regular text categorization algorithm that determines the most likely category for a given test document. Figure 1 illustrates the classification process based on extractive summarization. TRAINING DOCUMENTS
NEW DOCUMENT
Extractive Summarization
SUMMARIES
Learning
SUMMARY
TEXT CLASSIFIER
CATEGORY
Fig. 1: Text classification using extractive summarization
TEXT SUMMARIZATION FOR IMPROVED TEXT CLASSIFICATION
2.1
149
Extractive summarization
To summarize documents, we use a graph-based algorithm that generates extractive summaries by finding the most important sentences in the text, as described in our previous work (Mihalcea & Tarau 2004, Mihalcea & Tarau 2005). Shortly, the summarization algorithm starts by building a graph that represents the text, with sentences stored in the nodes, and edges drawn using a measure of lexical similarity. Next, a random-walk algorithm such as PageRank (Brin & Page 1998) or hits (Kleinberg 1999) is run on the graph, and the sentences with the highest score are selected for inclusion in the summary. The selection of this particular summarization algorithm is motivated by several reasons. First, the decision to use an extractive summarization tool, versus more complex systems that include sentence compression and text generation, is based on the fact that in our experiments text summarization is not an end per se, but rather an intermediate step for document classification. The informativeness of a summary is thus more important than its coherence, and summarization through sentence extraction is sufficient for this purpose. Second, through evaluations conducted on standard data sets, the algorithm was demonstrated to be competitive with the state-ofthe-art in text summarization (Mihalcea & Tarau 2004). Finally, by being an algorithm that can produce a ranking over sentences in a text, it is well suited for our experiments where we want to measure the impact on text classification of summaries of various lengths. 2.2
Algorithms for text classification
There is a large body of algorithms previously tested on text classification problems, due also to the fact that text categorization is one of the testbeds of choice for machine learning algorithms. In the experiments reported here, we compare results obtained with two frequently used text classifiers—Rocchio and Na¨ıve Bayes, selected for the diversity of their learning methodologies. Rocchio. This is an adaptation of the relevance feedback method developed in information retrieval (Rocchio 1971). It uses standard tf.idf weighted vectors to represent documents, and builds a prototype vector for each category by summing up the vectors of the training documents in each category. Test documents are then assigned to the category that has the closest prototype vector, based on a cosine similarity. Text classification experiments with different versions of the Rocchio algorithm showed competitive results on standard benchmarks (Joachims 1997). Na¨ıve Bayes. The basic idea in a Na¨ıve Bayes text classifier is to estimate
150
RADA MIHALCEA & SAMER HASSAN
the probability of a category given a document using joint probabilities of words and documents. Na¨ıve Bayes assumes word independence, which means that the conditional probability of a word given a category is assumed to be independent of the conditional probability of other words given the same category. Despite this simplification, Na¨ıve Bayes classifiers perform surprisingly well on text classification (Joachims 1997). While there are several versions of Na¨ıve Bayes classifiers (variations of multinomial and multivariate Bernoulli), we use the multinomial model (McCallum & Nigam 1998), which was shown to be more effective. 2.3
Data
For the classification experiments, we use the Reuters-21578 1 and WebKB 2 data sets—two of the most widely used test collections for text classification. For Reuters-21578, we use the standard ModApte data split (Apte et al. 1994), obtained by eliminating unlabeled documents, and selecting only categories that have at least one document in the training set and test set (Yang & Liu 1999). For WebKB, we perform a four-fold cross validation after removing the other category, using the 3 : 1 training/test split automatically created with the scripts provided with the collection, which separates the data into training on three of the universities plus a miscellaneous collection, and testing on a fourth held-out university. Both collections are further post-processed by removing all documents with less than 10 sentences, resulting into final data sets of 1277 training documents, 436 test documents, 60 categories for Reuters-21578, and 350 training documents, 101 test documents, 6 categories for WebKB. This last step is motivated by the goal of our experiments: we want to determine the impact of various degrees of summarization on text categorization, and this is not possible with very short documents. 2.4
Evaluation metrics
The evaluation is run in terms of accuracy, defined as the number of correct assignments among the document–category pairs in the test set. For the WebKB data set, since the classifiers assign exactly one category to each document, and a document can belong to only one category, this definition of accuracy coincides with the measures of precision, recall, and F-measure. For the Reuters-21578 data set, multiple classifications are possible for each document, and thus the accuracy coincides with the classification precision. 1 2
http://www.daviddlewis.com/resources/testcollections/reuters21578/ http://www-2.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/
TEXT SUMMARIZATION FOR IMPROVED TEXT CLASSIFICATION
Summary length 1 sentence 2 sentences 3 sentences 4 sentences 5 sentences 6 sentences 7 sentences 8 sentences 9 sentences 10 sentences full document
Reuters-21578 Rocchio Na¨ıve Bayes 70.18% 72.94% 72.48% 73.85% 73.17% 75.46% 72.94% 75.46% 75.23% 75.92% 77.00%∗ 78.38%∗ ∗ 76.38% 77.61%∗ 76.38%∗ 77.38%∗ 75.23% 77.61%∗ 75.23% 77.52%∗ 74.91% 75.01%
Rocchio 60.40% 62.38% 60.40% 63.37% 66.35%∗ 65.37%∗ 65.34%∗ 64.36% 65.34%∗ 65.35%∗ 63.36%
151
WebKB Na¨ıve Bayes 61.39% 71.29% 74.26% 76.24%∗ 79.24%∗ 77.24%∗ 76.24%∗ 76.21%∗ 75.25% 75.25% 74.25%
Table 1: Classification results for Reuters-21578 and WebKB, using Rocchio and Na¨ıve Bayes classifiers, for full-length documents and for extractive summaries of various lengths. Statistically significant improvements (p < 0.05, paired t-test) with respect to full-document classification are also indicated (∗ ) Evaluations figures are reported as micro-average over all the categories in the test set. 3
Experimental results
Classification experiments are run using each of the two learning algorithms, with documents in the collection represented by extracts of various lengths. The classification performance is measured for each experiment, and compared against a traditional classifier that performs text categorization using the original full-length documents. Table 1 shows the classification results obtained using the Reuters-21578 and the WebKB test collections. Note that these results refer to data subsets somewhat more difficult than the original test collections, and thus they are not directly comparable to results previously reported in the literature. For instance, the average number of documents per category in the Reuters21578 subset used in our experiments is only 25, compared to the average of 50 documents per category available in the original data set3 . Similarly, the WebKB subset has an average number of 75 documents per category, 3
Using the same Na¨ıve Bayes and Rocchio classifier implementations on the entire Reuters-21578 data set results in a classification accuracy of 77.42% for Na¨ıve Bayes and 79.70% for Rocchio, figures that are closely comparable to results previously reported on this data set.
152
RADA MIHALCEA & SAMER HASSAN Naive Bayes 80
75
75
70
70 Accuracy (%)
Accuracy (%)
Rocchio 80
65
60
65
60 N most important (graph-based) N least important (graph-based) first N (position based) last N (position based) random N full-length documents
55
50 1
2
3
4
5
6
7
Summary Length (number of sentences)
8
N most important (graph-based) N least important (graph-based) first N (position based) last N (position based) random N full-length documents
55
50 9
10
1
2
3
4
5
6
7
8
9
10
Summary Length (number of sentences)
Fig. 2: Classification accuracy for Reuters-21578 for extractive summaries of various lengths compared to 750 in the original data set. Figures 2 and 3 plot the classification accuracy on the two data sets for each learning algorithm, using extractive summaries of various lengths generated with the graph-based summarization method. For a comparative evaluation, the figures plot results obtained with methods that create an extract by: (a) selecting the N most important sentences using the graph-based extractive summarization algorithm; (b) selecting the first N sentences in the text; (c) randomly selecting N sentences; (d) selecting the N least important sentences using the low-end of the ranking obtained with the same summarization algorithm; (e) selecting the last N sentences. From a text categorization perspective, the results demonstrate that techniques for text summarization can be effectively combined with text classification methods to the end of improving categorization performance. On the Reuters-21578 data set, the classification performance using the Rocchio classifier improves from an accuracy of 74.90% to 77.00% obtained for summaries of 6 sentences. Similar improvements are observed for the Na¨ıve Bayes classifier, where the highest accuracy of 78.38% is obtained again for summaries of 6 sentences, and is significantly better than the accuracy of 75.01% for text classification with full-length documents. The impact of summarization is even more visible on the WebKB data set, where summaries of 5 sentences result in a classification accuracy of 79.24% using a Na¨ıve Bayes classifier, significantly higher than the accuracy of 74.25% obtained when the classification is performed with entire documents. Similarly, the categorization accuracy using a Rocchio classifier on the same data set improves from 63.36% for full-length documents to 66.35% for summaries of 5 sentences4 . The highest error rate reduction observed during these 4
The improvements observed in all classification settings are statistically significant at p < 0.05 level (paired t-test).
153
TEXT SUMMARIZATION FOR IMPROVED TEXT CLASSIFICATION Naive Bayes 80
75
75
70
70
65
65
Accuracy (%)
Accuracy (%)
Rocchio 80
60 55 50
40 1
2
3
4
5
6
7
Summary Length (number of sentences)
8
55 50
N most important (graph-based) N least important (graph-based) first N (position based) last N (position based) random N full-length documents
45
60
N most important (graph-based) N least important (graph-based) first N (position based) last N (position based) random N full-length documents
45 40 9
10
1
2
3
4
5
6
7
8
9
10
Summary Length (number of sentences)
Fig. 3: Classification accuracy for WebKB for extractive summaries of various lengths experiments was 19.3%, achieved with a Na¨ıve Bayes classifier applied on 5-sentence extractive summaries. Another aspect worth noting is the fact that these results can have important implications on the problem of text classification for documents for which only abstracts are available. As it was previously suggested (Hulth 2003), many documents on the Internet are not available as full-texts, but only as abstracts. Similarly, documents that are typically stored in printed form, such as books, journals, or magazines, may have an abstract available in electronic format, but not the entire text. This means that an eventual classification of such documents has to rely on abstract categorization, rather than traditional full-text categorization. The results obtained in the experiments reported here suggest that the task of text classification can be efficiently performed even when only abstracts are available for the documents to be classified. Classification efficiency is also significantly improved when the classification is based on summaries, rather than full-length documents. A clear computational improvement was observed even for the relatively small test collections used in our experiments. For instance, the Na¨ıve Bayes classifier takes 22 seconds to categorize the full-length documents in the Reuters21578 subset on a Pentium IV 3GHz 2GB computer, compared to only 6 seconds when the classification is done using 5-sentence summaries5 . The results have also important implications on the problem of text summarization. As seen in Figures 2 and 3, regardless of the classifier, a performance peak is observed for summaries of 5-7 sentences, which give the 5
One could argue that the summarization-based classification implies an additional summarization overhead. Note however that the summarization process can be parallelized and needs to be performed only once offline. The resulting summaries can be then used in multiple classification tasks.
154
RADA MIHALCEA & SAMER HASSAN
highest increase in classification accuracy. This result can have an interesting interpretation in the context of text summarization, as it indicates the optimal number of sentences required for “grasping” the content of a text. From this perspective, the task of text classification can be regarded as an objective way of defining the “informativeness” of an abstract, and could be used as an alternative to more subjective human-assessed evaluations of summary content. The comparative plots from Figures 2 and 3 also reveal another interesting aspect of the synergy between text summarization and document classification. Different automatic summarization tools – graph-based extractive summarization, heuristics that extract sentences based on their position in the text, or a simple random sentence selection baseline – have different but consistent impact on the quality of a text classifier, which suggests that text categorization can be used as an application-oriented evaluation testbed for automatic summarization. The graph-based summarization method gives better text classification accuracy as compared to the position-based heuristic, which in turns performs better than the simple random sentence selection baselines. This comparative evaluation correlates with rankings of these methods previously reported in the literature (Mihalcea & Tarau 2004), (Erkan & Radev 2004), which were based on human judgments or automatic evaluations such as Rouge (Lin & Hovy 2003). 4
Related work
To our knowledge, the impact of content-based text summarization on the task of text categorization was not explored in previous studies. Sentence importance was considered as an additional factor for feature weighting in work reported in (Ko et al. 2002), where words in a text were weighted differently based on the score associated with the sentence they belong to. Experiments with four different text categorization methods have shown that a weighting scheme based on sentence importance can significantly improve the classification performance. In (Kolcz et al. 2001), text summarization is regarded as a feature selection method, and is shown to improve over alternative algorithms for feature selections. Finally, another related study is the polarity analysis through subjective summarization reported in (Pang & Lee 2004), where the main goal was to distinguish between positive and negative movie reviews by first selecting those sentences likely to be more subjective according to a min-cut algorithm. The focus of their study was the analysis of text style (subjective versus objective), rather than classification of text content. We consider instead the more general text classification problem, combined with a typical text summarization task, and evaluate the role that text summaries can play in document categorization.
TEXT SUMMARIZATION FOR IMPROVED TEXT CLASSIFICATION
5
155
Conclusions
In this paper, we investigated the interaction between document summarization and text categorization, through comparative classification experiments relying on full-length documents or summaries of various lengths. We showed that techniques for automatic summarization can be used to improve the performance of a text categorization task, with error rate reductions of up to 19.3% obtained with a Na¨ıve Bayes or a Rocchio classifier when applied on short extractive summaries rather than full-length documents. Finally, we suggested that the interaction between text summarization and document categorization can be regarded as an application-oriented evaluation testbed for tools for automatic summarization, as summaries produced by different summarization tools were shown to have different impact on the performance of a text classifier. REFERENCES Apte, Chidanand, Fred Damerau & Sholom M. Weiss. 1994. “Towards Language Independent Automated Learning of Text Categorisation Models”. Proceedings of the 17th ACM SIGIR Conference on Research and Development in Information Retrieval, 23-30. Dublin, Ireland. Brin, Sergey & Lawrence Page. 1998. “The Anatomy of a Large-Scale Hypertextual Web Search Engine”. Computer Networks and ISDN Systems 30:1-7.107-117. Erkan, Gunes & Dragomir Radev. 2004. “Lexpagerank: Prestige in MultiDocument Text Summarization”. Proceedings of the Conference on Empirical Methods in Natural Language Processing, 365–371. Barcelona, Spain. Hulth, Anette. 2003. “Improved Automatic Keyword Extraction Given More Linguistic Knowledge”. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2003 ), 216-223. Sapporo, Japan. Joachims, Thorsten. 1997. “A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization”. Proceedings of the 14th International Conference on Machine Learning (ICML 1997 ), 143-151. Nashville, Tennessee. Kamvar, Sepandar, Taher Haveliwala, Christopher Manning & Gene Golub. 2003. “Extrapolation Methods for Accelerating PageRank Computations”. Proceedings of the 12th International World Wide Web Conference, 261-270. Budapest, Hungary. Kleinberg, Jon. 1999. “Authoritative Sources in a Hyperlinked Environment”. Journal of the ACM 46:5.604-632.
156
RADA MIHALCEA & SAMER HASSAN
Ko, Youngjoong, Jinwoo Park & Jungyun Seo. 2002. “Automatic Text Categorization Using the Importance of Sentences”. Proceedings of the 19th International Conference on Computational Linguistics, 1-7. Taipei, Taiwan. Kolcz, Aleksander, Vidya Prabakarmurthi & Jugal Kalita. “Summarization as Feature Selection for Text Categorization.” Proceedings of the 10th International conference on Information and Knowledge Management, 365-370. Atlanta, Georgia. Lin, Chin-Yew & Eduard Hovy. 2003. “Automatic Evaluation of Summaries Using n-gram Co-occurrence Statistics”. Proceedings of the Human Language Technology Conference, 71-78. Edmonton, Canada. McCallum, Andrew & Kamal Nigam. 1998. “A Comparison of Event Models for Naive Bayes Text Classification”. Proceedings of AAAI-98 Workshop on Learning for Text Categorization, 41-48. Madison, Wisconsin. Mihalcea, Rada & Paul Tarau. 2004. “TextRank – Bringing Order into Texts”. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2004 ), 404-411. Barcelona, Spain. Mihalcea, Rada & Paul Tarau. 2005. “An Algorithm for Language Independent Single and Multiple Document Summarization”. Proceedings of the International Joint Conference on Natural Language Processing, 19-24. Jeju Island, Korea. Ng, Hwee Tou, Goh, Wei Booh & Low, Kok Leong. 1997. “Feature Selection, Perceptron Learning, and a Usability Case Study for Text Categorization”. Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 67-73. Philadelphia, Pennsylvania. Pang, Bo & Lillian Lee. 2004. “A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts”. Proceedings of the 42nd Meeting of the Association for Computational Linguistics, 271-278. Barcelona, Spain. Rocchio, Joseph J. 1971. “Relevance Feedback in Information Retrieval”. Englewood Cliffs, New Jersey: Prentice Hall. Yang, Yiming & Xin Liu. 1999. “A Reexamination of Text Categorization Methods”. Proceedings of the 22nd ACM SIGIR Conference on Research and Development in Information Retrieval, 42-49. Berkeley, California. Yang, Yiming & Jan O. Pedersen. 1997. “A Comparative Study on Feature Selection in Text Categorization”. Proceedings of the 14th International Conference on Machine Learning, 412-420. Nashville, Tennessee.
Exploiting Linguistic Cues to Classify Rhetorical Relations Caroline Sporleder∗ & Alex Lascarides∗∗ ∗
Tilburg University &
∗∗
University of Edinburgh
Abstract We propose a method for automatically identifying rhetorical relations. We use supervised machine learning but exploit discourse markers to automatically label training data. Our models draw on a variety of linguistic cues to distinguish between relations. We show that these feature-rich models outperform the previously suggested bigram models by more than 20%, at least for small training sets. Our approach is therefore better suited to deal with relations for which it is difficult to automatically label a lot of training data because they are rarely signalled by unambiguous discourse markers.
1
Introduction
Clauses in a text link to each other via rhetorical relations such as contrast or result (Mann & Thompson 1987). For example, the sentences in (1) are linked by result. Many NLP applications, such as questionanswering and text summarisation, would benefit from a method which automatically identifies such relations. While rhetorical relations are sometimes signalled by discourse markers (also known as discourse ‘connectives’) such as since or consequently, these are often ambiguous. For example, since can indicate either a temporal or an explanation relation (examples (2a) and (2b), respectively). Discourse markers are also often missing (as in (1)). Hence, it is not possible to rely on discourse markers alone. (1) A train hit a car on a level crossing. It derailed. (2) a. She has worked in retail since she moved to Britain. b. I don’t believe he’s here since his car isn’t parked outside. We present a machine learning method which uses a variety of linguistic and textual features (word stems, part-of-speech tags, tense information) to determine the rhetorical relation between two adjacent text spans (sentences or clauses) in the absence of an unambiguous discourse marker. To avoid manual annotation of training data, we train on automatically labelled examples, building on earlier work by Marcu & Echihabi (2002), who extracted examples from large text corpora and used unambiguous discourse markers to label them with the correct rhetorical relation. Before training, the markers were removed to enable the classifier to learn from other features
158
CAROLINE SPORLEDER & ALEX LASCARIDES
and thus identify relations in unmarked examples. This approach works because there is often a certain amount of redundancy between the discourse marker and the general linguistic context. For example, the two clauses in example (3a) are in a contrast relation signalled by but. However, this relation can also be inferred if no discourse marker is present (see (3b)). (3) a. She doesn’t make bookings but she makes itinerary recommendations. b. She doesn’t make bookings; she makes itinerary recommendations. Hobbs et al. (1993) and Asher & Lascarides (2003) propose a logical approach to inferring relations, which in this case would rely on the linguistic cues of a negation in the first span, syntactic parallelism of the two spans, and the fact that they both have the same subject. We explore whether such cues can also be exploited as features in a statistical model for recognising rhetorical relations and whether such a feature-rich model leads to a better performance than Marcu & Echihabi’s (2002) word co-occurrence model. 2
Related research
Marcu & Echihabi (2002) present a machine learning approach to automatically identify four relations (contrast, cause-explanation-evidence, condition and elaboration) from Rhetorical Structure Theory (rst) (Mann & Thompson 1987). Two types of non-relations are also included. The training data are extracted automatically from a large text corpus (around 40 million sentences) using manually constructed extraction patterns containing discourse markers which signal one of these relations. For example, if a sentence begins with but, it is extracted together with the preceding sentence and labelled as contrast. Examples of non-relations are created artificially: one type of non-relation corresponds to randomly picked pairs of non-adjacent spans from the same text, and another type of non-relation to pairs of spans from different texts. The number of extracted examples per relation range from 900,000 to 4 million. After the markers had been removed, several Naive Bayes classifiers were trained to distinguish between relations on the basis of co-occurrences between pairs of words. The reported test set accuracy for the six-way classifier is 49.7%. Lapata & Lascarides (2004) present a method for inferring temporal connectives. They, too, label data automatically, using connectives such as while or since. But their aim is to predict the original temporal connective (which was removed from the test set) rather than the underlying rhetorical relation. For this, they train simple probabilistic models based on nine types of linguistically motivated features. They report accuracies of up to 70.7%.
EXPLOITING LINGUISTIC CUES
159
There have also been non-statistical approaches. Corston-Oliver (1998), for instance, presents a system which takes fully parsed sentences and determines rhetorical relations by applying heuristics based on linguistic cues such as clausal status, anaphora and deixis. Le Thanh et al. (2004) use a similar, heuristics-based approach which first splits sentences into discourse spans and then determines which relations hold between them. 3 3.1
Our approach Relations and discourse marker selection
We chose five sdrt (Asher & Lascarides 2003) relations: contrast, result, explanation, summary and continuation. For each of these, there are unambiguous discourse markers but these relations also frequently occur without such a marker; so it is beneficial to be able to determine them in the absence of an explicit marker. This is in contrast to relations such as condition, which always require a discourse marker (e.g., if. . . then). sdrt relations are defined on the basis of truth conditional semantics and tend to be less fine-grained than those in Rhetorical Structure Theory (rst) (Mann & Thompson 1987). Let R(a, b) denote the fact that a relation R connects two spans a and b. For each of the five relations it holds that R(a, b) is true only if the contents of a and b are true too. In addition, contrast(a,b) entails that a and b have parallel syntactic structures that induce contrasting themes, result(a,b) entails that a causes b, summary(a,b) entails that a and b are semantically equivalent, continuation(a,b) means that a and b have a contingent, common topic and explanation(a,b) means that b is an answer to the question why a? (cf. Bromberger 1962). To identify unambiguous discourse markers for the five sdrt relations, we undertook an extensive corpus study, using 2,000 randomly chosen examples (30 per marker), as well as linguistic introspection. The differences between sdrt and rst mean that some markers which are ambiguous in rst are unambiguous in sdrt. For example, in other words can signal either summary or restatement in rst, but sdrt does not distinguish these since the length of the related spans is irrelevant to sdrt’s semantics. Overall, we identified 55 unambiguous discourse markers for the five relations. These were used in the manually written extraction patterns. Sentences (4) to (8) below show one automatically extracted example for each relation (discourse markers are underlined, spans are indicated by square brackets). (4) [We can’t win] [but we must keep trying.] (contrast)
160
CAROLINE SPORLEDER & ALEX LASCARIDES
(5) [The ability to operate at these temperatures is advantageous,] [because the devices need less thermal insulation.] (explanation) (6) [By the early eighteenth century in Scotland, the bulk of crops were housed in ricks,] [the barns were consequently small.] (result) (7) [The starfish is an ancient inhabitant of tropical oceans.] [In other words, the reef grew up in the presence of the starfish.] (summary) (8) [First, only a handful of people have spent more than a few weeks in space.] [Secondly, it has been impractical or impossible to gather data beyond some blood and tissue samples.] (continuation)
3.2
Data
We used three corpora, mainly from the news domain, to extract our data set: the British National Corpus (BNC, 100 million words), the North American News Text Corpus (350 million words) and the English Gigaword Corpus (1.7 billion words). We preprocessed the corpora by removing duplicate texts and speech transcripts, and running a sentence splitter (Reynar &Ratnaparkhi 1997) on the latter two corpora. We applied the extraction patterns to the raw text to find potential examples. These were then parsed with the RASP parser (Carroll & Briscoe 2002) and the parse trees were processed to (i) identify the two spans and (ii) filter out false positives. For instance, (9) was extracted as an example of summary based on the apparent presence of the discourse marker in short. However, the parser correctly identified this string as part of the prepositional phrase in short order and the example was discarded. We extracted both intra- and inter-sentential relations (see (4) and (7) above, respectively). However, we limited the length of the spans to one sentence, as we specifically wanted to focus on relations between small units of text. (9) In short order I was to fly with ‘Deemy’ on Friday morning. There are three potential sources of errors: (i) the two spans are not related, (ii) they are related but the wrong relation is hypothesised and (iii) the hypothesised span boundaries are wrong. To assess the quality of the extracted data, we manually inspected 100 randomly selected examples (20 per relation). We found 11 errors overall: three of type (i) (no relation) and eight of type (iii) (wrong boundary). No wrongly predicted relation was found. The overall precision was thus 89%. The number of extracted training examples ranged from 1,732 for continuation and around 50,000 for contrast. On the whole, our data set
EXPLOITING LINGUISTIC CUES
161
is much smaller than the one used by Marcu & Echihabi (2002), which contained around 10 million examples for six relations. 3.3
The classifier
Several machine learning schemes can be employed for the task. We used a boosting algorithm with simple decision rules, as implemented in BoosTexter (Schapire & Singer 2000). BoosTexter allows a variety of feature types, such as nominal, numerical or text-based. For the latter, it applies n-gram models when forming classification hypotheses. We implemented 72 linguistically motivated features, roughly falling into 9 classes: positional features, length features, lexical features, part-of-speech features, temporal features, syntactic features and cohesion features. We defined three positional features, encoding whether the relation holds intra- or inter-sententially and whether the example occurs towards the beginning and end of a paragraph. The motivation for these features is that the probability of different relations is likely to vary with both their paragraph position and the position of sentence boundaries relative to span boundaries. For instance, a summary relation is probably more frequent at the beginning or end of a paragraph than in the middle of it. We encoded the length of the two spans to capture possible length effects. For example, it is possible that the spans are longer on average for continuation than for contrast. Lexical information is also likely to provide useful cues (cf. Marcu & Echihabi 2002). For example, a high word overlap between spans may be evidence for summary. Furthermore, while we removed the unambiguous discourse markers from the training examples (as they form the basis for the labelling), these may be accompanied by ambiguous cues (like still, signalling contrast or a temporal relation), which we do not remove. Words like this can and should be exploited by the classifier. Thus we encoded the lemmas and stems of all words and of the content words only. These were encoded as text features, allowing BoosTexter to automatically identify stem or lemma n-gram sequences that may be good cues for a particular relation. We also encoded the stem and lemma overlap between the spans. Some part-of-speech sequences may also be more likely for one relation than for another and were included as a text feature. We also separately encoded specific information about verb, noun and adjective lemmas (Lapata & Lascarides 2004). Furthermore, we mapped the lemmas to their most general WordNet (Fellbaum 1998) class (e.g., verb-of-cognition). Finally, we encoded the overlaps between lemmas and between WordNet classes. Tense and aspect provide clues about temporal relations among events and may influence the probabilities of different rhetorical relations. We
162
CAROLINE SPORLEDER & ALEX LASCARIDES
used simple heuristics to classify verbal complexes in terms of finiteness, modality, aspect, voice and negation (Lapata & Lascarides 2004). Some relations (e.g., summary) may have syntactically less complex spans than others (e.g., continuation). To estimate syntactic complexity, we encoded the number of NPs, VPs, PPs, ADJPs, and ADVPs in each span. We also included argument structure information, e.g., whether a verb has a direct object. Finally, we added information about the subjects, i.e., their part-of-speech tags, whether they have a negative aspect (e.g. nobody, nowhere), and the WordNet classes to which they map (see above). The degree of cohesion between two spans may be another informative feature. To estimate it we looked at the distribution of pronouns and at the presence or absence of ellipses (cf. Hutchinson 2004). 4
Experiments
We conducted three experiments. First, we assessed how well humans can determine rhetorical relations in the absence of discourse markers. This gives a measure of the difficulty of the task. We then tested our model and compared its performance to two baseline classifiers. Finally, we looked at which features are particularly useful for predicting the correct relation. 4.1
Experiment 1: Human agreement
Training on automatically labelled examples will only be successful if there is a certain amount of redundancy between the unambiguous discourse marker, which is used to assign the label and then removed, and other linguistic features of the examples. If discourse markers were only used in cases where a relation cannot be inferred from the linguistic context alone, any approach which aims to train a classifier on automatically extracted examples from which the unambiguous discourse markers have been removed would fail. The presence of redundancy in some cases is evident from examples like (3), where contrast can be inferred even when the discourse marker is removed. However, there may be other cases where this is more difficult. To assess the difficulty of determining the rhetorical relation when the discourse marker has been removed, we conducted a small pilot study with human subjects (cf. Soria & Ferrari 1998). We used our extraction patterns to automatically extract examples for the four rhetorical relations contrast, explanation, result and summary (continuation was added after the pilot study). We then manually checked the extracted examples to filter out false positives and randomly selected 10 examples per relation from which we then removed the discourse markers. We also semi-automatically selected 10 examples of adjacent sentences or clauses which were not related by any
EXPLOITING LINGUISTIC CUES
163
of the four relations. For each example, we included the two preceding and following sentences as context. We then asked three subjects who were trained in discourse annotation to classify each of the 50 examples as one of the four relations or as none. All subjects were aware that discourse markers had been removed from the examples but did not know the location of the removed discourse marker. We evaluated the annotations against the gold standard. The average accuracy was 71.25%, the average, pairwise Kappa coefficient (Siegel & Castellan 1988) was .61. While the agreement is far from perfect, it is relatively high for a discourse annotation task. Hence it seems that the task of predicting the correct relation for sentences from which the discourse marker has been removed is feasible for humans. 4.2
Experiment 2: Comparing the models
The machine learning experiments involved five relations: contrast, explanation, result, summary and continuation. We did not use the complete set of extracted examples, as this was highly imbalanced (see Section 3.2), which can have negative effects on a machine learner (see e.g. Japkowicz 2000). Instead, we created training and test sets which contained an equal number of examples for each relation (i.e., 1,732 examples per relation). We used 90% of this data set for training (7,795 examples) and 10% for testing (865 examples), making sure that the distribution of the relations was uniform in both data sets, and evaluated BoosTexter’s performance using 10-fold cross-validation. For comparison, we also used two baselines. For the first, a relation was predicted at random. As there are five, equally frequent relations in the test set, the average accuracy achieved by this strategy will be 20%. For the second baseline, we implemented a bigram model along the lines of Marcu & Echihabi (2002). Table 1 shows the average accuracies of the three classifiers for all relations and also for each individual relation. The feature-rich BoosTexter model performs notably better than either of the other two classifiers. It outperforms the random baseline by nearly 40% and the bigram model by more than 20%. This difference is statistically significant (χ2 = 208.12, DoF = 1, p <= 0.01). Furthermore, the performance gain achieved by our model holds for every relation with the exception of explanation where the bigram model performs better. While the bigram model is geared towards large training sets and it is possible that it outperforms the BoosTexter model if trained on a lot of data, obtaining large amounts of data is not always feasible, even if training examples are extracted automatically. Some relations occur relatively infrequently or are rarely signalled by an unambiguous discourse marker. This applies to continuation, which is usually signalled by an ambigu-
164
CAROLINE SPORLEDER & ALEX LASCARIDES
Relation contrast explanation result summary continuation all
Avg. Accuracy random bigrams BT 20.00 33.11 43.64 20.00 75.39 64.45 20.00 16.21 47.86 20.00 19.34 48.44 20.00 25.48 83.35 20.00 33.96 57.55
Table 1: Results for BoosTexter (BT ) and two baselines (10-fold cross-validation) ous marker (e.g., and) or no marker at all. For such relations, our approach seems a better choice than the model proposed by Marcu & Echihabi (2002). 4.3
Experiment 3: Feature exploration
To determine which features are particularly useful for the task, we conducted a further experiment in which we trained an individual BoosTexter model for each of the features. These one-feature classifiers were then tested on an unseen test set (again using 10-fold cross-validation). Table 2 shows the 10 best performing features and their average accuracies. Feature left stems left words intra/inter left pos-tags right words right stems right pos-tags left content words left noun lemmas right span length
Avg. Accuracy 42.51 41.79 39.18 34.62 32.82 32.58 31.72 29.78 28.30 28.12
Table 2: The ten best features (10-fold cross-validation) Lexical features (stems, words, lemmas) seem the most useful. Table 3 shows some of the words chosen by BoosTexter as particularly predictive of a given relation. Most of the choices seem fairly intuitive. For instance, an explanation relation is often signalled by tentatively qualifying adverbs (e.g., perhaps), while summary and continuation relations frequently contain pronouns and contrast can be signalled by words such as other or still etc. Table 2 also suggests that the lexical items in the left span are more important than those in the right. For example, left stems is 10% more
EXPLOITING LINGUISTIC CUES
165
accurate then right stems. This makes sense from a processing perspective: if the relation is already signalled in the left span the sentence will be easier to process than if the signalling is delayed until the right span is read. Relation contrast explanation result summary continuation
Predictive Words other, still, not, . . . perhaps, probably, mainly, . . . undoubtedly, so, indeed, . . . their, this, yet . . . you, it, there . . .
Table 3: Words chosen as cues for a relation 5
Conclusion
We have presented a machine learning method for automatically classifying discourse relations in the absence of unambiguous discourse markers, using a model which combines a wide variety of linguistic cues. The model was trained on automatically labelled data and tested on five relations. We found that this feature-rich model significantly outperformed the simpler bigram model proposed by Marcu & Echihabi (2002), at least on our, relatively small training set. Hence, our method is particularly suitable for relations which are rarely signalled by (unambiguous) discourse markers (e.g., continuation). In such cases, it is difficult to obtain sufficiently large training sets that a bigram model will perform well, even if the data is extracted from very large corpora. So far we have only tested our method on automatically labelled data, i.e., examples from which an originally present, unambiguous discourse marker had been removed, not on examples which occur naturally without such a marker. However, the latter are exactly the types of examples at which our method is aimed. In future research, we thus intend to create a small, manually labelled test corpus of examples which are not unambiguously marked and test our method on it to determine whether our results carry over to that data type. The RST Discourse Treebank (Carlson et al. 2002) could be used as a starting point for the corpus creation. Acknowledgements. The research in this paper was supported by epsrc grant number GR/R40036/01. We would like to thank Ben Hutchinson and Mirella Lapata for interesting discussions, participating in our labelling experiment and letting us use some of their Perl scripts. We are also grateful to the reviewers for their helpful comments. REFERENCES Asher, Nicholas & Alex Lascarides. 2003. Logics of Conversation. Cambridge: Cambridge University Press.
166
CAROLINE SPORLEDER & ALEX LASCARIDES
Bromberger, Sylvain. 1962. “An Approach to Explanation”. Analytical Philosophy ed. by Ronald J. Butler, 75-105. Oxford: Oxford University Press. Carlson, Lynn, Daniel Marcu & Mary Ellen Okurowski. 2002. RST Discourse Treebank. Linguistic Data Consortium. Carroll, John & Edward Briscoe. 2002. “High Precision Extraction of Grammatical Relations”. 19th Int. Conf. on Computational Linguistics (COLING-02 ), 134-140. Taipei, Taiwan. Corston-Oliver, Simon H. 1998. “Identifying the Linguistic Correlates of Rhetorical Relations”. Workshop on Discourse Relations and Discourse Markers, 8-14. Montreal, Canada. Fellbaum, Christiane Fellbaum, ed. 1998. WordNet: An Electronic Database. Cambridge, Mass.: MIT Press. Hobbs, Jerry R., Mark Stickel, Douglas Appelt & Paul Martin. 1993. “Interpretation as Abduction”. Artificial Intelligence 63:1-2.69-142. Hutchinson, Ben. 2004. “Acquiring the Meaning of Discourse Markers”. 42nd Annual Meeting of the Association for Computational Linguistics (ACL’2004), 685-692. Barcelona, Spain. Japkowicz, Nathalie. 2000. “The Class Imbalance Problem: Significance and Strategies”. Int. Conf. on Artificial Intelligence (IC-AI-00), 111-117. Las Vegas, Nevada. Lapata, Mirella & Alex Lascarides. 2004. “Inferring Sentence-internal Temporal Relations”. Meeting of the North American Chapter of the Assoc. for Comp. Linguistics (NAACL-04 ), 153-160. Boston, Mass. Le Thanh, Huong, Geetha Abeysinghe & Christian Huyck. 2004. “Generating Discourse Structures for Written Text”. 20th International Conference on Computational Linguistics (COLING-04 ), 329-335. Geneva, Switzerland. Mann, William C. & Sandra A. Thompson. 1987. Rhetorical Structure Theory: A Theory of Text Organization. Technical Report ISI/RS-87-190. Los Angeles, Calif.: ISI. Marcu, Daniel & Abdessamad Echihabi. 2002. “An Unsupervised Approach to Recognizing Discourse Relations”. 40th Annual Meeting of the Association for Computational Linguistics (ACL-02 ), 368-375. Philadelphia, Penn. Reynar, Jeffrey C. & Adwait Ratnaparkhi. 1997. “A Maximum Entropy Approach to Identifying Sentence Boundaries”. 5th Applied Natural Language Processing Conference (ANLP-97 ), 16-19. Washington, DC. Schapire, Robert E. & Yoram Singer. 2000. “BoosTexter: A Boosting-based System for Text Categorization”. Machine Learning 39:2-3.135-168. Siegel, Sidney & N. John Castellan. 1988. Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill. Soria, Claudia & Giacomo Ferrari. 1998. “Lexical Marking of Discourse Relations – Some Experimental Findings”. Workshop on Discourse Relations and Discourse Markers, 36-42. Montreal, Canada.
Tree Edit Distance for Textual Entailment Milen Kouylekov∗,∗∗ & Bernardo Magnini∗∗ ∗
ITC-irst, Centro per la Ricerca Scientifica e Tecnologica ∗∗ University of Trento Abstract
This paper addresses Textual Entailment (i.e., recognizing that the meaning of a text entails the meaning of another text) using a Tree Edit Distance algorithm between the syntactic trees of the two texts. It also compares the contribution of lexical resources for recognizing textual entailment in an experiment carried on over the pascal-rte dataset.
1
Introduction
The problem of language variability (i.e., the fact that the same information can be expressed with different words and syntactic constructs) has been attracting a lot of interest during the years and it poses significant issues in front of systems aimed at natural language understanding. The example below shows that recognizing the equivalence of the statements came in power, was prime-minister and stepped in as prime-minister is a challenging problem. • Ivan Kostov came in power in 1997. • Ivan Kostov was prime-minister of Bulgaria from 1997 to 2001. • Ivan Kostov stepped in as prime-minister 6 months after the December 1996 riots in Bulgaria. While the language variability problem is well known in Computational Linguistics, a general unifying framework has been proposed only recently in (Dagan & Glickman 2004). In this approach, language variability is addressed by defining the notion of entailment as a relation that holds between two language expressions (i.e., a text t and an hypothesis h) if the meaning of h, as interpreted in the context of t, can be inferred from the meaning of t. The entailment relation is directional, as the meaning of one expression can entail the meaning of the other, while the opposite may not. The Recognizing Textual Entailment (rte) task takes as input a T H pair and consists in automatically determining whether an entailment relation holds between T and H or not. The task, potentially, covers almost all the phenomena in language variability: entailment can be due to lexical variations, as it is shown in example (1), to syntactic variation (example 2), to semantic inferences (example 3) or to complex combinations of all such
168
MILEN KOUYLEKOV & BERNARDO MAGNINI
levels. As a consequence of the complexity of the task, one of the crucial aspects for any rte system is the amount of knowledge required for filling the gap between T and H. Examples: 1. T – Euro-Scandinavian media cheer Denmark v Sweden draw. H – Denmark and Sweden tie. 2. T – Jennifer Hawkins is the 21-year-old beauty queen from Australia. H – Jennifer Hawkins is Australia’s 21-year-old beauty queen. 3. T – The nomadic Raiders moved to LA in 1982 and won their third Super Bowl a year later. H – The nomadic Raiders won the Super Bowl in 1982. In example 1 we need to know that T lexically H ; in example 2 we need to understand that the syntactic structure are equivalent; in 3 we need to reason about temporal entities. The aim of this paper is to provide a clear and homogeneous framework for the evaluation of lexical resources for the rte task. The framework is based on the intuition that the probability of an entailment relation between t and h is related to the ability to show that the whole content of h can be mapped into the content of t. The more straightforward the mapping can be established, the more probable is the entailment relation. Since a mapping can be described as the sequence of editing operations needed to transform t into h, where each edit operation has a cost associated with it, we assign an entailment relation if the overall cost of the transformation is below a certain threshold, empirically estimated on the training data. Within the tree edit distance (ted) framework, the complexity of rte is put on the availability of entailment rules and on the definition of cost functions for the three editing operations. In this paper we investigate the role of resources for entailment rules for the definition of cost functions. We have experimented the ted approach with a non annotated document collection and a similarity relation estimated over a corpus of dependency trees. Experiments, carried on the pascal-rte dataset, provide significant insight for future research on rte. 2
Tree edit distance on dependency trees
We adopted a tree edit distance algorithm applied to the syntactic representations (i.e., dependency trees) of both t and h. A similar use of tree edit distance has been presented by (Punyakanok et al. 2004) for a Question Answering system, showing that the technique outperforms a simple bag-of-word approach. While the cost function presented in (Punyakanok et al. 2004) is quite simple, for the rte challenge we tried to elaborate more complex and task specific measures.
TREE EDIT DISTANCE FOR TEXTUAL ENTAILMENT
169
According to our approach, t entails h if there exists a sequence of transformations applied to t such that we can obtain h with an overall cost below a certain threshold. The underlying assumption is that pairs between which an entailment relation holds have a low cost of transformation. The kind of transformations we can apply (i.e., deletion, insertion and substitution) are determined by a set of predefined entailment rules, which also determine a cost for each editing operation. We have implemented the tree edit distance algorithm described in (Zhang & Shasha 1990) and applied to the dependency trees derived from t and h. Edit operations are defined at the level of single nodes of the dependency tree (i.e., transformations on subtrees are not allowed in the current implementation). Since the Zhang & Shasha (1990) algorithm does not consider labels on edges, while dependency trees provide them, each dependency relation R from a node A to a node B has been re-written as a complex label B-R concatenating the name of the destination node and the name of the relation. All nodes except the root of the tree are relabeled in such way. The algorithm is directional: we aim to find the better (i.e., less costly) sequence of edit operation that transform t (the source) into h (the target). According to the constraints described above, the following transformations are allowed: • Insertion: insert a node from the dependency tree of h into the dependency tree of t. When a node is inserted it is attached with the dependency relation of the source label. • Deletion: delete a node N from the dependency tree of t. When N is deleted all its children are attached to the parent of N. It is not required to explicitly delete the children of N as they are going to be either deleted or substituted on a following step. • Substitution: change the label of a node N1 in the source tree into a label of a node N2 of the target tree. Substitution is allowed only if the two nodes share the same part-of-speech. In case of substitution the relation attached to the substituted node is changed with the relation of the new node.
3
System architecture
The system is composed by the following modules, showed in Figure 1: (i) a text processing module, for the preprocessing of the input t/h pair; (ii) a matching module, which performs the mapping between t and h; (iii) a cost module, which computes the cost of the edit operations.
170
MILEN KOUYLEKOV & BERNARDO MAGNINI
Fig. 1: System architecture 3.1
Text processing module
The text processing module creates a syntactic representation of a t/h pair and relies on a sentence splitter and a syntactic parser. For sentence splitting we used MXTerm (Ratnaparkhi 1996), a Maximum entropy sentence splitter. For parsing we used Minipar, a principle-based English parser (Lin 1998) which has high processing speed and good precision. 3.2
Matching module
The matching module finds the best sequence of edit operations between the dependency trees obtained from t and h. It implements the edit distance algorithm described in Section 2. The entailment score of a given pair is calculated in the following way: score(T, H) =
ed(T, H) ed(, H)
(1)
where ed(T, H) is the function that calculates the edit distance cost and ed(, H) is the cost of inserting the entire tree h. A similar approach is presented in (Monz & de Rijke 2001), where the entailment score of two ′ documents d and d is calculated by comparing the sum of the weights (idf)
TREE EDIT DISTANCE FOR TEXTUAL ENTAILMENT
171
of the terms that appear in both documents to the sum of the weights of ′ all terms in d . We used a threshold t such that if score(T, H) < t then t entails h, otherwise no entailment relation holds for the pair. To set the threshold we have used both the positive and negative examples of the training set provided by the pascal-rte dataset (see Section 5.1 for details). 3.3
Cost module
The matching module makes requests to the cost module in order to receive the cost of single edit operations needed to transform t into h. We have different cost strategies for the three edit operations. Insertion. The intuition underlying insertion is that its cost is proportional to the relevance of the word w to be inserted (i.e., inserting an informative word has an higher cost than inserting a less informative word). More precisely: Cost[Ins(w)] = Rel(w)
(2)
where Rel(w), in the current version of the system, is computed on a document collection as the inverse document frequency (idf ) of w, a measure commonly used in Information Retrieval. If N is the number of documents in a text collection and Nw is the number of documents of the collection that contain w then the idf of w is given by the formula: idf (w) = log
N Nw
(3)
The most frequent words (e.g., stop words) have a zero cost of insertion. Substitution. The cost of substituting a word w1 with a word w2 can be estimated considering the semantic entailment between the words. The more the two words are entailed, the less the cost of substituting one word with the other. We used the following formula: Cost[Subs(w1 , w2 )] = Ins(w2 ) ∗ (1 − Ent(w1 , w2 ))
(4)
where Ins(w2 ) is calculated using (3) and Ent(w1 , w2 ) can be approximated with a variety of relatedness functions between w1 and w2 .
172
MILEN KOUYLEKOV & BERNARDO MAGNINI
There are two crucial issues for the definition of an effective function for lexical entailment: first, it is necessary a database of entailment rules with enough coverage; second, we have to estimate a quantitative measure for such rules. We experimented the use of a dependency based thesaurus available at http://www.cs.ualberta.ca/ lindek/downloads.htm. For each word, the ~ thesaurus lists up to 200 most similar words and their similarities calculated on a parsed corpus using frequency counts of the dependency triples. A complete review of the method including a comparison with different approaches is presented in (Lin 1996). Dependency triples consists of the head, a dependency type and a modifier. They can be viewed as features for the head and the modifiers in the triples when calculating similarity. The cost of a substitution is calculated by the following formula: Entsim (w1 , w2 ) = simth (w1 , w2 )
(5)
where w1 is the word from t that is being replaced by the word w2 from h and simth (w1 , w2 ) is the similarity between w1 and w2 in the thesaurus multiplied by the similarity between the corresponding relations. The similarity between relations is stored in a database of relation similarities obtained by comparing dependency relations from a parsed local corpus. The similarities have values from 1 (very similar) to 0 (not similar). If there is no similarity, the cost of substitution is equal to the cost of inserting the word w2. Deletion. In the pascal-rte dataset t is typically shorter than h. As a consequence, we expect that much more deletions are necessary to transform t into h than insertions or substitutions. Given this bias toward deletion, in the current version of the system we set the cost of deletion to 0. This expectation has been empirically confirmed (see results of system 2 in Section 5.3). 4
Experiments and results
We carried out a number of experiments in order to estimate the contribution of different combinations of the available resources. In this section we report about the dataset, the experiments and the results we have obtained.
TREE EDIT DISTANCE FOR TEXTUAL ENTAILMENT
4.1
173
Dataset
For the experiments we have used the pascal-rte dataset (Dagan et al. 2005). The dataset1 has been collected by human annotators and it is composed of 1367 text (T ) - hypothesis (H ) pairs split into positive and negative examples (a 50%-50% split). The examples came from seven different application scenario (i.e., Information Retrieval, Question Answering, Machine Translation, Information Extraction, Reading Comprehension, Paraphrase Acquisition) where textual entailment is supposed to play a crucial role. Typically, T consists of one sentence while H was often made of a shorter sentence. The dataset has been split in a training (576 pairs) and a test (800 pairs) part. 4.2
Experiments
The following configurations of the system have been experimented. System 1: Tree Edit Distance Baseline. In this configuration, considered as a baseline for the Tree Edit Distance approach, the cost of the three edit operations are set as follows: Deletion: always 0 Insertion: the idf of the word to be inserted Substitution: 0 if w1 = w2 , infinite in all the other cases. In this configuration the system just needs a non-annotated corpus for estimating the idf of the word to be inserted. Deletion is 0 because we expect much more deletion that insertions, due to the fact that T is longer than H, implying that much more deletions than insertions are necessary. System 2: Deletion as idf. In this configuration we wanted to check the impact of assigning a cost to the deletion operation. Deletion: the idf of the word to be deleted Insertion: the idf of the word to be inserted Substitution: same as system 1. System 3: Similarity Database. This is the same as system 1., but we estimate the cost of substitutions using the similarity database described in Section 3.3. We expect a broad coverage with respect to the previous system. Deletion: always 0 Insertion: the idf of the word to be inserted Substitution: same as system 2, plus similarity rules. 1
The dataset is available for download at http://www.pascal-network.org/ Challenges/RTE/Datasets .
174 4.3
MILEN KOUYLEKOV & BERNARDO MAGNINI
Results
For each system we have tested we report a table with the following data: • #attempted. The number of deletions, insertion and substitutions that the algorithm successfully attempted (i.e., operations that are included in the best sequence of editing transformations). • %success. The proportion of deletions, insertion and substitutions that the algorithm successfully attempted over the total attempted. • accuracy. The proportion of T-H pairs correctly classified by the system over the total number of pairs. • cws. Confidence Weighted score, The proportion of T-H pairs correctly classified by the system over the total number of pairs. Table 1 shows the results obtained by the systems we experimented. #deletions #insertions #substitutions %deletions %insertions %substitutions accuracy cws
System 1 20325 7927 2686 0.88 0.75 0.25 0.560 0.615
System 2 19984 7586 3027 0.87 0.71 0.29 0.481 0.453
System 3 20101 7703 2910 0.87 0.73 0.27 0.566 0.624
Table 1: Results The hypothesis about the 0 cost of the deletion operation was confirmed by the results of system 2. The similarity database used in system 3 increased the number of the successful substitutions made by the algorithm from 25% to 27%. It also increased performance of the system for both accuracy and cws. The impact of the similarity database on the result is small, because of the low similarity between the dependency trees of h and t. 5
Discussion
The approach we have presented can be used as a framework for testing resources for lexical entailment. The experiments show that using a similarity database with the edit distance algorithm can be of benefit for recognizing textual entailment. In order to test the real performance of resources of lexical entailment rules a set of pair from RTE dataset which require lexical entailment rules must be selected.
TREE EDIT DISTANCE FOR TEXTUAL ENTAILMENT
175
The performance of the system is correlated with the performance of the used resources. A wrong substitution with a low cost can significantly influence the optimal cost of the tree mapping. In order to obtain good results we should consider for substation only pairs with high entailment score (in our experiment similarity). The tree edit distance algorithm is designed to work with substitution on the level of tree nodes while our analysis of the pascal-rte dataset show that sub-tree substitutions are more suitable for the task. Other resources of entailment rules (e.g., paraphrases in (Lin & Pantel 2001), entailment patters as acquired in (Szpektor et al. 2004)) could significantly wide the application of entailment rules and, consequently, improve performances. We estimated that for about 40% of the the true positive pairs the system could have used entailment rules found in entailment and paraphrasing resources. As an example, the pair 565: t - Soprano’s Square: Milan, Italy, home of the famed La Scala opera house, honored soprano Maria Callas on Wednesday when it renamed a new square after the diva. h - La Scala opera house is located in Milan, Italy. could be successfully solved using a paraphrase pattern such as Y home of X <=> X is located in Y, which can be found in (Lin & Pantel 2001). However, in order to use this kind of entailment rules , it would be necessary to extend the “single node” implementation of tree edit distance to address editing operations among sub-trees. A system with an algorithm capable of calculating the cost of substitution on the level of subtrees can be used as a framework for testing paraphrase and entailment acquisition systems. A drawback of the tree edit distance approach is that it is not able to observe the whole tree, but only the subtree of the processed node. For example, the cost of the insertion of a subtree in h could be smaller if the same subtree is deleted from t in prior or later stage. A context sensitive extension of the insertion and deletion module will increase the performance of the system. 6
Conclusion and future work
We have presented an approach for recognizing textual entailment based on tree edit distance applied to the dependency trees of T and H. We have also demonstrated that using lexical similarity resources can increase the performance of a system based on such algorithm. In the future we plan to incorporate more resources for lexical entailment rules. We would like compare the performance obtained with the similarity database to WordNet (Fellbaum 1998), a lexical database which includes
176
MILEN KOUYLEKOV & BERNARDO MAGNINI
lexical and semantic relations among word senses. Moreover, in order to use entailment and paraphrasing resources, we plan to extend the tree edit distance algorithm with sub-tree substitutions. REFERENCES Dagan, Ido & Oren Glickman. 2004. “Generic Applied Modeling of Language Variability”. Proceedings of PASCAL Workshop on Learning Methods for Text Understanding and Mining. Grenoble, France. Dagan, Ido, Oren Glickman & Bernardo Magnini. 2005. “The PASCAL Recognizing Textual Entailment Challenge”. Proceedings of PASCAL Workshop on Recognizing Textual Entailment, 1-8. Southampton, U.K. Fellbaum, Christiane. 1998. WordNet, an Electronic Lexical Database. Cambridge, Mass.: MIT Press. Lin, Dekang. 1998. “Dependency-based evaluation of MINIPAR”. Proceedings of the Workshop on Evaluation of Parsing Systems (LREC-1998 ). Granada, Spain. Lin, Dekang. 1998. “An Information-Theoretic Definition of Similarity”. Proceedings of 15th International Conference on Machine Learning, 296-304. Madison, Wisconsin. Lin, Dekang & Patrik Pantel. 2001. “Discovery of inference rules for Question Answering”. Natural Language Engineering 7.4:343-360. Monz, Christof & Maarten de Rijke. 2001. “Light-Weight Entailment Checking for Computational Semantics”. Proceedings of The third workshop on inference in computational semantics (ICoS-3 ), 59-72. Siena, Italy. Punyakanok, Vasin, Dan Roth & Wen-tau Yih. 2004. “Mapping Dependencies Trees: An Application to Question Answering”. Proceedings of AI & Math 2004. Fort Lauderdale, Florida. Ratnaparkhi, Adwait. 1996. “A Maximum Entropy Part-Of-Speech Tagger”. Proceeding of the Empirical Methods in Natural Language Processing Conference, 491-497. Philadelphia, Pennsylvania. Szpektor, Idan, Hristo Tanev, Ido Dagan & Bonaventura Coppola. 2004. “Scaling Web-based Acquisition of Entailment Relations”. Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-04 ), 41-48. Barcelona, Spain. Zhang, Kaizhong & Dennis Shasha. 1990. “Simple fast algorithms for the editing distance betweentrees and related problems”. SIAM Journal on Computing 18:6.1245-1262.
A Genetic Algorithm for Optimising Information Retrieval with Linguistic Features in Question Answering ¨ rg Tiedemann Jo University of Groningen Abstract In this chapter we describe a genetic algorithm applied to the optimisation of information retrieval (ir) within a Dutch question answering (qa) system. In our approach, information retrieval is used for passage retrieval to filter out relevant text passages for subsequent answer extraction modules in the qa system. We focus on the integration of linguistic features extracted from syntactic analyses of the texts in the retrieval component. For this we apply output of the Dutch dependency parser Alpino together with an off-the-shelf ir engine. Query parameters are optimised using a genetic algorithm to improve the overall performance of the qa system.
1
Introduction
Standard information retrieval is commonly used in question answering systems in order to reduce the search space for the system to find answers to given natural questions in unrestricted texts. Consequently, ir is one of the bottlenecks in such a system because it is bound to fail if ir does not provide relevant passages containing answers. The simplest approach to ir in qa uses words appearing in the question as keywords for standard plain text retrieval. However this is a strong limitation disregarding other information included in a natural language question. Recently, many researchers investigate possibilities of integrating linguistic knowledge into ir especially to facilitate question answering (Strzalkowski 1996, Katz & Lin 2003, Neumann & Sacaleanu 2004, Moldovan et al. 02, Prager et al. 2000, Cui et al. 2001). One reason for the current effort in this field is the increased availability of robust nlp tools with large coverage and improved efficiency. However, results achieved so far are rather modest and simple techniques such as stemming are still the most effective ones. It seems to be important to select appropriate features in relevant cases in order to succeed in a task like information retrieval (Katz & Lin 2003). Therefore, we will concentrate on the optimisation of feature selection and keyword weighting. The remaining part of this chapter is organised as follows: The next section briefly describes the ir component in our qa system. Thereafter, we will discuss the genetic algorithm used for optimisation. Then we will
178
¨ JORG TIEDEMANN
show experimental results on open-domain question answering followed by a summary and some conclusions. 2
Information retrieval with linguistic features
In our qa system we use off-the-shelf ir engines for passage retrieval. We implemented an interface to several open-source ir packages and in this paper we will apply the well-known Lucene system from the Apache Jakarta project (Jakarta 2004). We work on Dutch qa and, therefore, we also apply simple extensions of Lucene for Dutch such as a stemmer and a stop word filter. Our work is focused on open-domain question answering using data from the cross-language evaluation forum clef. As mentioned earlier, we like to integrate linguistic knowledge into the retrieval of relevant passages. For this, we have parsed the entire clef corpus for Dutch with its approx. 1,000,000 paragraphs containing more than 4,000,000 sentences. We applied the wide-coverage dependency parser Alpino (Bouma et al. 2001) for this task, which produces among the dependency relations also part-of-speech labels, named entity labels, and linguistic root forms. An example of the output of Alpino is shown in Figure 1. From the syntactic analyses we extract several combinations of features for each paragraph and store them in different fields in the ir index. Henceforth, we will refer to these fields as “index layers”. Table 1 summarises the layers defined in our index. Feature combinations such as dependent-head bigrams for each word in each paragraph ... text plain text tokens root linguistic root forms RootPOS root + POS tag RootRel root + relation (to its head) RootHead root (dependent) + root (head) RootRelHead dependent + relation + head for selected words in each paragraph ... compound compounds ne named entities neLOC location names nePER person names neORG organisation names neTypes labels of named entities
Table 1: Index layers are simply concatenated using a special delimiter symbol to form single tokens that can be queried.
179
A GENETIC ALGORITHM FOR OPTIMISING IR IN QA top
smain
su
det det het0
hd verb word4
vc
1 np
ppart
hd noun embargo1
hd prep tegen2
mod
obj1
pp
1
obj1 name Irak3
det det de7
hd verb stel in5
mod
pp
hd prep na6
hd noun inval8
hd prep in9
obj1
np
mod
mod
pp
pp
obj1 name Koeweit10
hd prep in11
obj1 noun 199012
Fig. 1: A dependency tree produced by Alpino: Het embargo tegen Irak werd ingesteld na de inval in Koeweit in 1990. (The embargo against Iraq has been declared after the invasion of Kuwait in 1990.) The index described above is filled with the data from the machine annotated clef corpus. Now it is possible to send complex queries to the ir engine using the powerful query language supported by Lucene. There are several features that we explore in our experiments: Field restrictions: This is the basic parameter needed to search in the multi-layer index. We have to specify the field we like to query with the given keywords. For example, we can look into the text layer with a query such as “text:(Irak Verenigde Naties)” Keyword weighting: Each keyword in a query can be weighted with a so-called “boost factor”. For example, we can weight the named entity (ne) “Irak” with a weight of 2: “ne:(Irak^2 Verenigde_Naties)” Required keywords: Lucene’s query language allows one to mark keywords to be required in matching paragraphs using the character ’+’.
180
¨ JORG TIEDEMANN
For example, to require the named entity “Verenigde Naties” we can write “ne:(Irak^2 +Verenigde_Naties)”. Questions are parsed by Alpino producing similar annotations as done for the sentences in the clef corpus. Now, the main idea is to match features extracted from the analysed question with paragraphs in the index. As we argued earlier, it is important to select appropriate features (in terms of keywords) in order to improve retrieval performance. Hence, we do not want to simply use all possible feature combinations matching one of the fields in the index. Rather, we want to perform a fine-grained selection of features to combine the most appropriate keywords in a query to the ir engine. For this we can not only select one or more index layers to be queried but also restrict the set of selected keywords by additional constraints. For example, we can restrict the query to only include plain text tokens that have been labelled as nouns. Any kind of restriction can be used to define a subset of keywords, which then can be combined with other subsets. Now, we can also set boost factors for certain selections of keywords to weight them differently than other keywords. Furthermore, we can mark keyword selections as required. In our experiments we use constraints on part-of-speech labels and relation types (dependency relations of words to their head words). Of course, both types of constraints can also be combined. In this way we have a large variety of keyword types (index layer + constraints) that can be weighted in various ways. We use some simple preference heuristics to keep queries at a reasonable size. For instance, we define that weights of more “specific” keywords (with stronger constraints) overwrite weights of less specific ones. We also define that restrictions on relation names are more specific than POS restrictions and required markers overwrite weights for keywords with identical specificity. We also use some heuristics to prune queries. For a given set of questions, we do the following: Keyword types are discarded if 1. they do not produce any keywords for all questions, i.e. no keywords are selected using the given constraints for any question 2. they do not add any new keyword nor they change any parameter (weight/required marker) of any query (i.e. all keywords selected are already contained in the existing queries with exactly the same weights) 3. they do not produce any ir output for all questions 3
A Genetic Algorithm for query optimisation
In our approach, we use ir as a black box filled with our data. We can modify the input parameters and inspect the output produced but we do not
A GENETIC ALGORITHM FOR OPTIMISING IR IN QA
181
know exactly how the internal ranking is influenced by the given parameters. However, we want to perform a systematic search for optimised queries to improve the performance of the qa system. Here, we can use a randomised optimisation process to iteratively search through possible settings with the goal to find the optimal parameter setup. Genetic algorithms have proven to be useful for such a task, running in a manner of a “focused trial-and-error” beam search through a pre-defined hypothesis space. The goal in genetic algorithms is to let a population of individuals find a way to better fitness in their environment using evolutionary principles. Useful genes (=settings) are kept and combined in the population from generation to generation. Random changes are introduced using mutation. Here, individuals refer to qa processes trying to answer a given set of questions. Each individual differs in the settings used for ir. The parameters are simply the keyword types used in the ir queries as defined in the previous section. Here, each of these keyword types may have exactly one weight or a required marker. Fitness is determined by the quality of the result when running qa with the individual’s ir settings. We use mean reciprocal ranks of the top five answers to measure this quality. One important principle in genetic algorithms is “crossover/reproduction”, i.e. means to produce new children for coming generations. Here, we define crossover as the creation of a new qa process using the ir settings of two parents from the current population. The settings of the two parents are simply merged. In cases of overlaps we compute the arithmetic mean of parameters (i.e. weights). Again, required markers overwrite weights on identical keyword types. Parents are selected randomly without any preference mechanism. Another important principle in genetic algorithms is “natural selection”. This is usually done in a probabilistic way. However, we simplify this function to a top N search for the individuals with the highest fitness scores in the current population for reasons we will explain below. One of the beauties of genetic algorithms is the possibility to run parallel processes in order to explore the space of possible solutions. This is especially useful in our case because question answering requires a lot of processing power and time. It is crucial that we can run through a large number of settings in order to find a (sub-)optimal solution. For our experiments, we use a Beowulf Linux cluster to run up to 50 processes in parallel (that is what we define to be the maximum number of children to “grow up” at a time). We keep the size of the population small with only 25 individuals from which new children are created. This means that we define our selection process to be a top 25 search among “living individuals”. This makes it possible to apply the selection procedure each time a child reports its fitness score. Hence, we do not have to wait for an entire new generation which would be necessary when applying probabilistic selection.
¨ JORG TIEDEMANN
182
We believe that this procedure of gradually changing the population describes a more natural development than jumping from one generation to an entirely new one from time to time. The disadvantage of top N selection is that it is deterministic and, therefore, the optimisation process might get stuck in local maxima more easily. This, however is compensated by probabilistic mutation operations with rather high likelihoods as explained below. As a side effect of the top N search, children that develop faster will also have a larger chance to survive (at least until stronger individuals appear pushing them out of the population). This may reduce the effects of over-fitting because we prefer simple queries to complex ones which usually require more processing time. However, complex ones are not discarded and may overrule simpler ones (and their ancestors) in the long run. Mutation operations are defined to include additional randomness in the optimisation process. Standard approaches apply mappings of parameters to gene sequences. However, we define mutation operations directly on the parameters given. We use the following operations on new indivudals: • a new keyword type is added with a chance of 0.2; • a keyword type is removed with a chance of 0.1; • a keyword weight is modified by a random value between −5 and 5 with a chance of 0.1 (+ the weight has to remain a positive value); • a keyword type is marked as required with a chance of 0.01. As mentioned earlier, we have chosen to set rather high likelihoods to the mutation operations to compensate for the deterministic selection process and also to account for the small populations size. We have not experimented with different or even variable settings which might be an interesting investigation in future work. 4
Experiments
For our experiments we used date from the clef competition on Dutch qa. We used questions from the last three years, 2003 – 2005 for training and evaluation. These question sets have been annotated with their correct answers and supporting documents from the clef corpus in which the answer can be found (if they were found by one of the participating teams). We used only those questions which have at least one answer and we use simple string matching to determine if our system produces correct answers (i.e. we used answer strings instead of document IDs for evaluation). We randomly selected 250 questions for evaluation and used the remaining 522 questions for training.
A GENETIC ALGORITHM FOR OPTIMISING IR IN QA
183
The genetic algorithm has been initialised with single keyword type settings using default settings for weights (=1) and no required markers. We limited ourselves to a small set of possible constraints to select keywords of different types. We included constraints on a sub-set of word classes (names, nouns, adjectives and verbs) and a sub-set of dependency relations (direct object, modifier, apposition and subject). Each combination of constraints has been considered and applied to each of the index layers to form the set of initial settings. However, as explained earlier, we pruned this set by discarding keyword types according to the conditions defined in section 2. After running the initial settings the genetic algorithm decides how to combine them and how to develop further. Table 2 shows the results after various numbers of settings tested. number of settings 300 450 600 750 900 1000 1200 1500 1700 baseline (plain text keywords)
training 0.419 0.426 0.429 0.436 0.436 0.441 0.447 0.448 0.449 0.407
evaluation 0.413 0.421 0.429 0.436 0.436 0.452 0.455 0.442 0.442 0.412
Table 2: Improvements of QA performance by optimising IR query parameters We obtained scores above the baseline already after 300 indivuduals tested (initial settings included). Here, the baseline is the best performing initial setting, namely plain text word keywords which is similar to standard ir (Dutch stemming and stop word removal has been applied). In the top part of Figure 2, the training curve is plotted for the 1700 settings. The bottom part shows the corresponding scores on unseen evaluation data. The thin black lines refer to the scores of individual settings tested in the optimisation process. The solid bold lines refer to the top scores obtained using the optimised query settings. The plots illustrate the typical behaviour of genetic algorithms. The random modifications cause the oscillation of the fitness scores plotted with thin black lines. The algorithm picks up the advantageous features and promotes them in the competitive selection process. This gives the gradual improvements in terms of fitness of the top individual tested. The final scores are well above the base line (about 10% improvement in MRR). The largest improvements can be observed in the beginning of the optimisation
¨ JORG TIEDEMANN
184
0.46
answer string MRR
0.44 0.42 0.4 0.38 0.36 0.34 0.32
baseline (0.407) optimized (training)
0.3 0
200
400
600
800
1000
1200
1400
1600
number of settings 0.46
answer string MRR
0.44 0.42 0.4 0.38 0.36 0.34 0.32
baseline (0.412) optimized (evaluation)
0.3 0
200
400
600
800
1000
1200
1400
1600
number of settings
Fig. 2: Parameter optimisation with a genetic algorithm. Training (top) and evaluation (bottom). process which is very common in machine learning approaches. The training curve starts to level out after about 1300 settings. The two plots in Figure 2 also illustrate the relation between the impact of ir variation on training data and on unseen evaluation data. There is a strong correlation between the scores obtained when applying the various ir settings to the two different data sets. The general tendency of evaluation scores is similar to the training curve with step-wise improvements throughout the optimisation process even though the development of
A GENETIC ALGORITHM FOR OPTIMISING IR IN QA
185
the evaluation score is not monotonic. The optimisation process seems to succeed in picking up preferable features for improving qa in general. Besides the small drops of evaluation scores in the beginning of the optimisation process we can also observe a tendency of decreasing values after about 1500 settings which is probably due to over-fitting. This suggests some form of early stopping or a stop condition based on validation data. Note also that the scores on evaluation data using optimised settings do not always reach the top scores seen among all individuals tested on this data set. The highest score observed on evaluation data actually amounts to 0.469 which is clearly above the 0.455 obtained when using settings optimised on the training data. However, the differences are small and the optimised settings produce scores high above the baseline on both, training and evaluation data. 5
Conclusions and future work
In this chapter we described the information retrieval component of our open-domain question answering system. We integrated linguistic features produced by a wide-coverage parser for Dutch, Alpino, in the ir index to improve the retrieval of relevant paragraphs. These features are stored in several index layers that can be queried by the qa system in various ways. We also used word class labels and syntactic relations as constraints in the selection of keywords from analysed natural language questions to build the ir queries. We demonstrated a genetic algorithm for optimising query parameters such as keyword weights and selection constraints to take advantage of the enriched ir index. Query parameters are trained on a set of questions annotated with answers taken from the clef competitions on Dutch question answering. We showed that the performance of the qa system could be improved by about 10% on unseen evaluation data measured with the mean reciprocal rank of retrieved answers. In future work we would like to test further combinations of features and constraints to improve the performance even further. We would also like to experiment with different settings of the genetic algorithm in order to investigate the impact of these parameters on the optimisation process. Acknowledgements. This research was carried out as part of the research program for Interactive Multimedia Information Extraction, imix, financed by nwo, the Dutch Organisation for Scientific Research.
186
¨ JORG TIEDEMANN
REFERENCES Bouma, Gosse, Gertjan van Noord & Robert Malouf. 2001. “Alpino: Wide coverage computational analysis of Dutch”. Computational Linguistics in the Netherlands (CLIN 2000 ), 45-59. Tilburg, The Netherlands: Rodopi. Cui, Hang, Renxu Sun, Keya Li, Min-Yen Kan & Tat-Seng Chua. 2005. “Question Answering Passage Retrieval Using Dependency Relations”. Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR 2005 ), 400-407. Salvador, Brazil. Apache Jakarta. 2004. Apache Lucene - A High-performance, Full-featured Text Search Engine Library. http://lucene.apache.org/java/docs/index.html [Source checked in April 2006]. Katz, Boris & Jimmy Lin. 2003. “Selectively using relations to improve precision in question answering”. Proceedings of the EACL-2003 Workshop on Natural Language Processing for Question Answering, 43-50. Budapest, Hungary. Moldovan, Dan, Sanda Harabagiu, Roxana Girju, Paul Morarescu, Finley Lacatusu, Adrian Novischi, Adriana Badulescu & Orest Bolohan. 2002. “LCC Tools for Question Answering”. Proceedings of the 11th Text Retrieval Conference, TREC, 388-397. Gaithersburg, MD, USA. Neumann, G¨ unter and Bogdan Sacaleanu. 2004. “Experiments on Robust NL Question Interpretation and Multi-Layered Document Annotation for a CrossLanguage Question/Answering System”. Multilingual Information Access for Text, Speech and Images: 5th Workshop of the Cross-Language Evaluation Forum, CLEF, Lecture Notes in Computer Science, Volume 3491, Springer Berlin / Heidelberg, 411-422. Bath, UK. Prager, John, Eric Brown, Anni Cohen, Dragomir Radev & Valerie Samn. 2000. “Question-Answering by Predictive Annotation”. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 184-191. Athens, Greece. Strzalkowski, Tomek, Louise Guthrie, Jussi Karlgren, Jim Leistensnider, Fang Lin, Jos´e P´erez-Carballo, Troy Straszheim, Jin Wang & Jon Wilding. 1997. “Natural Language Information Retrieval: TREC-5 Report”. Proceedings of the 5th Text Retrieval Conference, TREC, 291-314. Gaithersburg, MD, USA.
Lexico-Syntactic Subsumption for Textual Entailment Vasile Rus & Arthur Graesser The University of Memphis Abstract We present here a graph-based approach to the task of textual entailment between a Text and Hypothesis. The approach takes into account the full lexico-syntactic context of both the Text and Hypothesis and relies heavily on the concept of subsumption. It starts with mapping the Text and Hypothesis into graph-structures where nodes represent concepts and edges represent lexico-syntactic relations among concepts. Based on a subsumption score between the Text-graph and Hypothesis-graph an entailment decision is made. A novel feature of our approach is the handling of negation. The results obtained from a standard entailment test data set are better than results reported by systems that use similar knowledge resources. We also found that a tf-idf approach performs close to chance for sentence-like Texts and Hypotheses.
1
Introduction
Recognizing Textual Entailment (rte) (Dagan, Glickman & Magnini 2005) is the task of deciding, given two text fragments, whether the meaning of one text is entailed (can be strongly inferred) from another text (Dagan, Glickman & Magnini 2005). We say that T (the entailing text) entails H (the entailed hypothesis). The task is relevant to a large number of applications, including machine translation, question answering, and information retrieval. In this paper we present a novel approach to rte that uses minimal knowledge resources. The approach relies on lexico-syntactic information and synonymy specified in a thesaurus. The results obtained are better than state-of-the-art solutions that use same array of resources. In our approach each T-H pair is first mapped into two graphs, one for H and one for T, with nodes representing main concepts and edges indicating dependencies among concepts as encoded in H and T, respectively. An entailment score, entail(T, H), is then computed that quantifies the degree to which the T-graph subsumes the H-graph. The score is so defined to be non-reflexive, i.e. entail(T, H) 6= entail(H, T ). We evaluate the approach on the standard rte Challenge data (Dagan, Glickman & Magnini 2005). The application of this work to Intelligent Tutoring Systems (its), namely AutoTutor (Graesser, Lu, Jackson, Mitchell, Ventura, Olney & Louwerse
188
VASILE RUS & ARTHUR C. GRAESSER
2004; Graesser, VanLehn, Rose, Jordan & Harter 2001), is under investigation. AutoTutor is an its that works by having a conversation with the learner. AutoTutor appears as an animated agent that acts as a dialog partner with the learner. The animated agent delivers AutoTutor’s dialog moves with synthesized speech, intonation, facial expressions, and gestures. Students are encouraged to articulate lengthy answers that exhibit deep reasoning, rather than to recite small bits of shallow knowledge. It is in the deep knowledge and reasoning part where we explore the potential help of entailment. The rest of the paper is structured as follows. Section 2 outlines related work. Section 3 describes our approach in detail. Section 4 presents the experiments we performed, results and a comparative view of similar approaches. The Conclusions section wraps up the paper. 2
Related work
The task of textual entailment has been recently treated, in one form or another, by research groups ranging from informational r