Human Language Technology. Challenges for Computer Science and Linguistics (4th Language and Technology Conference - LTC 2009 - Poznan, Poland)

Lecture Notes in Artificial Intelligence Edited by R. Goebel, J. Siekmann, and W. Wahlster Subseries of Lecture Notes ...

Author: Zygmunt Vetulani

50 downloads 1014 Views 8MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Artificial Intelligence Edited by R. Goebel, J. Siekmann, and W. Wahlster

Subseries of Lecture Notes in Computer Science

6562

Zygmunt Vetulani (Ed.)

Human Language Technology Challenges for Computer Science and Linguistics 4th Language and Technology Conference, LTC 2009 Poznan, Poland, November 6-8, 2009 Revised Selected Papers

13

Series Editors Randy Goebel, University of Alberta, Edmonton, Canada Jörg Siekmann, University of Saarland, Saarbrücken, Germany Wolfgang Wahlster, DFKI and University of Saarland, Saarbrücken, Germany Volume Editor Zygmunt Vetulani Adam Mickiewicz University in Poznan Faculty of Mathematics and Computer Science ul. Umultowska 87, 61614 Poznan, Poland E-mail: [email protected]

ISSN 0302-9743 e-ISSN 1611-3349 ISBN 978-3-642-20094-6 e-ISBN 978-3-642-20095-3 DOI 10.1007/978-3-642-20095-3 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011923534 CR Subject Classification (1998): I.2.1, I.2.7, I.2, H.2.5, H.5.2, F.4.2, I.5 LNCS Sublibrary: SL 7 – Artificial Intelligence

© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Preface

Human language technologies emerged in late twentieth century as a natural consequence of the technological progress of the human race. Since the industrial revolution in the eighteenth century, through the nineteenth and twentieth centuries, humans considerably transformed the world and deﬁnitely entered into the role of masters of the game. In an environment dominated by today’s technology, humans eﬀectively took over the totality of the animate world. The key to this success has consisted, since ancient times, in having mastered energy: ﬁre, gunpowder, steam, coal, water power, electricity, nuclear power. The result was the creation of ever more sophisticated tools and artifacts at the high price of the destruction of large areas of the natural environment and of traditional social structures. Humans have become slaves of the technologies they invented and the subjects of new complex social structures. A new kind of resource, with roots dating back to Gutenberg and beyond, was focussed on at the beginning of the twenty-ﬁrst century: information. In the information-rich technological environment, a new sociopolitical concept of the Information Society became a paradigm1. The Information Society implies a novel kind of social relation in the world in which people are surrounded by a mass of information-rich artifacts and technologies designed to be collaborative aids for them. This idea, explicitly addressed by EU policies of the last 20 years, gave a new stimulus for the development of technologies involving (or depending on) various forms of Natural Language Processing and Speech Technology, i.e., the so-called Human Language Technologies2. In the preface to the LTC 2007 Revised Selected Papers3 the editors mentioned a number of challenges which inspire researchers and language engineers in this ﬁeld across the world: - Human languages evolved in such a way that made it well suited for humans but especially unsuited for processing on today’s digital computers. Development of appropriate methodology to face the logical complexity of the human language remains a major challenge both for linguists and computer science engineers, although an essential progress has been achieved during the last 50 years. 1

2

3

Information Society Technologies was one of thematic priorities under the European Six Framework Programme for the period 2002-2006. This term ﬁrst(?) appeared in the name of the ARPA Human Language Technology (HLT) Workshops in 1993 (Former DARPA Speech and Natural Language Workshops). Zygmunt Vetulani and Hans Uszkoreit (Eds.)(2009): Human Language Technology, Challenges of the Information Society, Third Language and Technology Conference, LTC 2007, Poznan, Poland, October 2007, Revised Selected Papers. LNAI 5603. Springer-Verlag, Berlin, Heidelberg.

VI

Preface

- The multilingual global society poses another grand technological challenge because in order to preserve the multicultural setup of the globalizing world all surviving languages need to be supported by language technologies. - Another challenge consists in the integration of language with other media of communication such as gesture, facial expression, pictures and ﬁlms, each of them requiring diﬀerent techniques for encoding and processing that need to interact with language processing. - Combination of human language technologies with a new generation of knowledge technologies which are nowadays called semantic technologies (cf. the Semantic Web). All four of the challenges cited above continue to be valid and stimulating. But this list should not be considered closed. Technologies evolve at a high speed and the day approaches when our technological environment will be populated with devices which are equipped with artiﬁcial but human-like and human-friendly language and speech competences. To provide equal access to this environment for all members of the Information Society, independently of their mother tongue and thus avoiding their technological exclusion, will be a major challenge in the future. The time to start facing this challenge is now. This idea was behind the LTC-FLaReNet joint LRL Workshop “Getting Less-Resourced Languages onBoard!”4 aiming at the international promotion of the development of language resources and technologies for under-resourced languages. The goal of this session was ﬁrst to illustrate the various dimensions of that topic for various technologies (both written and spoken language processing) and for various languages (Indian languages (including Sanskrit), Celtic languages (Welsh, Irish, Scottish Gaelic, Manx Gaelic, Cornish and Breton), Amharic, Luxembourgish, Romani, Basque, Catalan, Galician, Sami and the Ga African language). A ﬁnal panel session allowed for a general discussion and the drafting of a list of recommendations for at a better language coverage in Language Resources, and therefore Language Technologies, as it appears in the ﬁnal report5 . Some texts contained in this volume were presented at the LRL session. In the present volume the reader will ﬁnd the revised and in many cases substantially extended versions of 52 selected papers presented at the 4th Language and Technology Conference. This selection was made from a total of 103 high-quality contributions written by 250 authors qualiﬁed for conference presentation by an international jury on the basis of blind reviews. As a rule these assessments were taken into account for the selection to this volume. Still, we are aware of the partly subjective character of this selection. In a small number 4

5

The Less-Resourced Languages Workshop (LRL) was proposed and set-up by Khalid Choukri, Joseph Mariani and Zygmunt Vetulani. J. Mariani, K. Choukri and Z. Vetulani, “Report on the Special joint LTC-FLaReNet session ‘Getting Less-Resourced Languages On-Board !’ at LTC’09 Conference”. Cf. http://www.ﬂarenet.eu/sites/default/ﬁles/LREC2010 International Cooperation Workshop Mariani-Choukri-Vetulani.pdf (cf. also http://www.ltc.amu.edu.pl).

Preface

VII

of cases some interesting papers which presented partial or preliminary results of on-going research and development did not qualify for this book, but it is hoped that more complete results will be presented soon. The selection of revised papers well illustrates the complexity and diversity of the ﬁeld of Human Language Technologies. The papers collected in this book report on many man-years of hard work by 130 authors representing research institutions from 21 countries6 : Belgium, Canada, Czech Republic, Finland, France, Germany, Greece, India, Italy, Iran, Ireland, Japan, Lithuania, Poland, Portugal, Romania, Russia, Spain, Switzerland, UK, Ukraine7 . The reader will ﬁnd the papers structured into thematic chapters. Clustering papers was a diﬃcult task as in most cases the contributions addressed more than one thematic area so that our decisions should be considered as approximative. In particular, their attributions to chapters do not necessarily correspond to their attribution to the LTC thematic sessions and also may not correspond to the authors’ ﬁrst choice. These chapters are: 1. Speech Processing (9) 2. Computational Morphology/Lexicography (4) 3. Parsing (4) 4. Computational Semantics (6) 5. Entailment (2) 6. Dialogue Modeling and Processing (4) 7. Digital Language Resources (9) 8. WordNet (3) 9. Document Processing (2) 10. Information Processing (IR,IE,other) (7) 11. Machine Translation (2). The ordering of six initial chapters follows the natural order which humans use to process language understanding in NL-communication: starting with speech, and proceeding through morphology, syntax and semantics to dialogue. The next two chapters focus on resources, and the last three on multi-aspectual language engineering tasks. The idea to close this selection with machine translation (MT) papers symbolizes our opinion that machine translation, being (with the Warren Weaver’s Memorandum, 1947)8 the ﬁrst large-scale program in language

6 7

8

Against 250 authors from 38 countries participating in LTC 2009. In fact the geographical/language coverage is larger then what may look like from the presented data, as we know only the present aﬃliation of the data. Also, language coverage is larger than what may be inferred from the list of 21 countries. For example, languages such as Almaric, Bulgarian, Luxembourgish or Sanskrit do not correspond to the aﬃliations of the authors of respective papers. Weaver, W. (1949): ‘Translation’. Repr. in: Locke, W.N. and Booth, A.D. (eds.) Machine translation of languages: fourteen essays (Cambridge, Mass.: Technology Press of the Massachusetts Institute of Technology, 1955), pp. 15-23.

VIII

Preface

engineering, will probably be the last of these research and development programs to reach a fully satisfactory result. There is no signiﬁcant ordering within chapters, where papers are presented in alphabetical order with respect to the ﬁrst author’s family name. The ﬁrst chapter, “Speech Processing,” contains ten contributions. It starts with a text on the evaluation of automatic speech-to-phoneme alignment systems (Baghai-Ravary, Kochanski, Coleman).The next one is a detailed presentation of a speech corpus of European Portuguese for text-to-speech systems (Barros, M¨obius). It is followed by a paper on the quality improvement of a Japanese speech recognition system for the (noisy) car environment designed on the basis of weighted ﬁnite-state transducers (Betkowska Cavalcante, Shinoda, Furui). What follows is a contribution about TTS-oriented corpus design for Bulgarian, a South Slavic language with a relatively low number of speakers (9 million) by national language standards, but with a strong tradition in language engineering, particularly MT (Chalamandaris, Tsiakoulis, Raptis, Karabetsos). Phonetic similarity based on recognized observed properties (in automatic speech recognition, ASR) is the focus of the next paper. Conclusions concerning pronunciation variation modeling are presented (Kane, Mauclair, Carson-Berndsen). This paper is followed by a work on the detection of errors occurring in ASR output considered as a post-processing stage of ASR (Pellegrini, Trancoso). The last of the ﬁve challenges mentioned above is directly addressed in the study of pronunciation and writing variants in Luxembourgish, an under-resourced language spoken by approximately 300,000 speakers (Snoeren, Adda-Decker, Adda). The reader will then ﬁnd a paper on morpheme-based language modeling for speech recognition for Amharic, another less-resourced language represented in this book (Tachbelie, Abate, Menzel). The next paper is on multilevel annotation software for speech corpora, in which the authors present a generic and corpus-independent toolkit supporting de facto standards and commonly used annotation formats (Wilson, Carson-Berndsen). The chapter closes with a report on time duration of phonemes (in Polish). This study is oriented to the development of speech/speaker recognition systems (B. Zi´olko, M. Zi´ olko). In the “Computational Morphology/Lexicography” chapter we present three papers. The ﬁrst is about the classiﬁcation of Japanese polysemous verbs using advanced mathematical methods (Fukumoto, Suzuki, Yamashita). The next paper deals with a problem of the similarity or typological proximity of languages. It proposes to measure the proximity between languages in terms of their vocabulary structure (Lepage, Gosme, Lardilleux). A tool for lexicographic description of multi-word units is presented in the last paper of this chapter (Marciniak, Savary, Sikora, Woli´ nski). Four contributions were accepted for the chapter concerned with “Parsing,” which opens with a presentation of a parsing algorithm for context-free and probabilistic context-free (CFG and PCFG) grammars. This contribution is language independent and therefore of general interest for those who use CFG/PCFG for NL investgations (Hulden). The next article proposes a parsing algorithm described for the Czech language but presented as representative of the family of

Preface

IX

Slavic languages. In order to deal with problems caused by word-order-related phenomena (Slavic languages have a relatively free word order) the authors propose a pattern-matching-based algorithm (Kov´ aˇr, Hor´ ak, Jakub´ıˇcek). Sentence segmentation as a pre-processing stage for higher-level processing (parsing) is considered in the next paper; the SRX standard for sentence segmentation was applied for English and Polish and the results have been compared (Milkowski, Lipski). The last contribution in the parsing chapter is about using lexicon grammar entries for French verbs in a large-coverage parser (Tolone, Sagot). Various issues of computational semantics are of ﬁrst interest in the next chapter of six contributions. This chapter opens with the problem of overt pronouns resolution (Fukumoto, Suzuki). The second one addresses a methodological question of whether sentiment intensity may be considered to be a good summary indicator. This problem, relevant for summarization tasks, is given a negative answer (Kabadjov, Balahur, Boldrini). The third paper contributes to deep semantic analysis which often requires temporal information processing. Temporal information may be inferred from the textual descriptions of events. The paper contributes to classiﬁcation of temporal relations (Mirroshandel, Khayyamian, Ghassem-Sani). Semantic disambiguation is the focus of the next paper where the authors present a platform (applied to Polish) for testing various word sense disambiguation technologies (Mlodzki, Przepi´orkowski). The next paper is about a semantic analyzer used in the translation of natural language texts into the Polish Sign Language (Romaniuk, Suszcza´ nska, Szmal). The last article of this chapter presents a system which outputs semantic representations (logical forms or discourse representation structures) on top of dependency relations produced by a statistical parser (Zouaq, Gagnon, Ozell). A chapter with two papers on text entailment follows. In the ﬁrst one a system for recognizing sentential entailment is presented (B´edaride, Gardent), whereas a formal logical framework for deﬁnition and combination of specialized entailment engines is discussed in the second (Cabrio, Magnini). The next chapter is on “Dialogue Modeling and Processing”. The chapter starts with a paper on Wizard-of-Oz experiments for natural dialogue collection. The collected dialogues were processed and the results were used at the design stage of an artiﬁcial companion project and re-used for creation of an expressive speech corpus for Czech (Gr˚ uber, Leg´at, Ircing, Romportl, Psutka). Dialogue turn analysis for the purpose of automatic summarization of voice conversations was the theme of the second paper (Pallotta, Delmonte, Bristot). The next two papers cover various aspects of a man–machine dialogue system (POLINT112-SMS) designed to understand short messages (SMS) in natural language. The ﬁrst of them covers both methodological aspects of project development and the architecture of the resulting system (Vetulani, Marciniak). The second one focusses on dialogue control and on solutions of several hard problems like anaphora, overt pronouns etc. (Walkowska). Nine papers are collected in the “Digital Language Resources” chapter. It opens with a research report on the development strategy and the HLT resources obtained so far for the Basque language for which the “survival”

X

Preface

program supported by the local administration in the Basque Country is explicitly based on language technologies (Alegria, Aranzabe, Arregi, Artola, D´ıaz de Ilarraza, Mayor, Sarasola). Methodological issues related to the construction of morphosyntactic resources with special attention to the comparison of natural languages were investigated by the authors of the second paper of the chapter. An important potential output of this research is a practical indication of how to predict the costs of development of morphosyntactic resources (Blancafort, De Loupy). The next paper is a presentation of a tool for corpora visualization, especially with respect to diﬀerent types of frequency information (Culy, Lyding). Automatic acquisition of entries for bilingual lexica is a concern of the fourth paper, in which the authors propose an acquisition method consisting in exploration of keyword lists attached to bilingual documents (Grali´ nski, Jassem, Kurc). An experiment in annotating Sanskrit, the oldest documented still spoken Indo-European language, is reported in the next paper (Jha, Gopal, Mishra). Readers concerned with dialectology resources may have interest in the paper on authorizing procedures for e-learning courses on dialectical phonetics (Kedrova, Yegorov, Volkova). The next paper presents a corpus collection exercise whose aim was modeling of user language competence with particular interest in describing spatial relations (Osi´ nski). The following overview of a number of XML standards for multilevel corpus annotation is a contribution to the general problem of standards development, as too many standards mean no standards at all (Przepi´ orkowski, Ba´ nski). The last contribution in the LR chapter is a report on a project aiming at a national-scale corpus of academic Lithuanian, a language for which the existing corpora are not suﬃcient to cover the whole range of scientiﬁc discourse (Usoniene, Butenas, Ryvityte, Sinkuniene, Jasionyte, Juozapavicius). Although many papers in this book refer to WordNet-based methodologies (technologies), WordNet occupies a central position in only three of them. In the ﬁrst of these the author shows how WordNet, independently of the internal organization of its data, may be applied as a tool to enrich a valence dictionary of Polish verbs by adding semantic information (Hajnicz). Using a Princeton WordNet-based sense disambiguation algorithm to evaluate the degree of semantic relatedness between words is the main concern of the second paper (Ion, S¸ tef˘ anescu). The last one in this chapter presents an interface to Polish WordNet (PolNet) and its application within an NL understanding system which uses PolNet as ontology (Kubis). Two papers may be classiﬁed as directly contributing to the “Document Processing” ﬁeld. The ﬁrst of the two presents a Web-accessible tool to support diplomatic transcriptions of historical language data (i.e., transcriptions free of any kind of interpretation involved in the transcribing process) (Dipper, Schnurrenberger). The second one proposes an algorithm of authorship attribution for short texts (Nawrot). Several (7) papers in this selection deal with various aspects of “Information Processing in form of Retrieval, Extraction and Other”. Looking for speculative sentences in scientiﬁc texts (biology) is proposed as a tool for biologists interested

Preface

XI

in ﬁnding new hypotheses published in scientiﬁc literature (J. Descl´es, Alrahabi, J.-P. Descl´es). The second paper presents two Arabic summarization systems based on the extraction of sentences that best match words in the query (El-Haj, Kruschwitz, Fox). Opinion extraction consisting in identiﬁcation of subjectivity expressions is the theme of the third paper (Esuli, Sebastiani). The next paper in the chapter presents a method of analyzing the structure of the titles of research papers by means of information-extraction techniques (Kondo, Nanba, Takezawa, Okumura). The ﬁfth article presents a system for the extraction and presentation of quotations found in French newswire transmissions. This tool is of direct practical interest, e.g., for press agencies (de La Clergerie, Sagot, Stern, Denis, Recourc´e, Mignot). The next contribution shows the reader how to improve the precision of contextual advertising with the help of language technologies (Pak). The closing article of this chapter presents a comparison of four unsupervised algorithms for automatically extracting keywords from the multimedia archive of Belga News Archive (Palomino, Wuytack). Finally, the volume ends with two papers classiﬁed as contributions to “Machine Translation.” Both papers address the issue of MT quality evaluation. The ﬁrst one explores the use of paraphrases for the reﬁnement of traditional methods for text evaluation (valid also for summarization) (Hirahara, Nanba, Takezawa, Okumura). The last paper in the book describes the usage of normalized compression distance as a language-independent machine translation quality evaluation tool (Kettunen). January 2011

Zygmunt Vetulani Joseph Mariani

Organization

Organizing Committee Zygmunt Vetulani - Conference Chair Marek Kubis Piotr Kuszyk Jacek Marciniak Tomasz Obrebski Jedrzej Osi´ nski Justyna Walkowska (All at the Adam Mickiewicz University, Pozna´ n, Poland)

LTC Program Committee Victoria Arranz Anja Belz Janusz S. Bie´ n Krzysztof Bogacki Christian Boitet Leonard Bolc Lynne Bowker Nicoletta Calzolari Nick Campbell Julie Carson-Berndsen Khalid Choukri Adam Dabrowski El˙zbieta Dura Katarzyna Dziubalska-Kolaczyk Tomaz Erjavec Cedrick Fairon Christiane Fellbaum Maria Gavrilidou Dafydd Gibbon Stefan Grocholewski Franz Guenthner

Hans Guesgen Eva Hajiˇcov´a Roland Hausser Steven Krauwer Eric Laporte Yves Lepage Gerard Ligozat Natalia Loukachevitch Wieslaw Lubaszewski Bente Maegaard Bernardo Magnini Joseph Mariani Jacek Martinek Gayrat Matlatipov Keith J. Miller Nicholas Ostler Karel Pala Pavel S. Pankov Patrick Paroubek Stelios Piperidis Emil Plywaczewski Gabor Proszeky

Adam Przepi´ orkowski Reinhard Rapp Zbigniew Rau Mike Rosner Justus Roux Vasile Rus Rafal Rzepka Fr´ed´erique S´egond Zhongzhi Shi Wlodzimierz Sobkowiak Hanna Szafra´ nska ´ Marek Swidzinski Ryszard Tadeusiewicz Dan Tuﬁ¸s Hans Uszkoreit Zygmunt Vetulani - Chair Piek Vossen Tom Wachtel Jan Weglarz Mariusz Zi´olko Richard Zuber

XIV

Organization

LRL Workshop Program Committee LRL Co-chairs: Joseph Mariani, Khalid Choukri, Zygmunt Vetulani N´ uria Bel Gerhard Budin Nicoletta Calzolari Dafydd Gibbon Marko Grobelnik Jan Hajiˇc

Alfed Majewicz Asunci´on Moreno Jan Odijk Nicholas Ostler Stellios Piperidis Gabor Proszeky

Mohsen Rashwan Kepa Sarasola Marko Tadi´c Dan Tuﬁ¸s Cristina Vertan Briony Williams

Marcin Junczys-Dowmunt Besim Kabashi Natalia Kotsyba Anna Kup´s´c Rafal L. G´ orski Penny Labropoulou Kevers Laurent Maciej Lison Jacek Marciniak Montserat Marimon Agnieszka Mykowiecka Tomasz Obrebski Maciej Piasecki

Dawid Pietrala Thomas Proisi Prokopis Prokopidis Michal Ptaszy´ nski Valeria Quochi Flo Reeder Pawel Rydzewski Agata Savary Gra˙zyna Vetulani Marta Villegas Jorge Vivaldi Alina Wr´ oblewska Dapeng Zhang Bartosz Zi´olko

Invited Reviewers Xabier Arregi Richard Beaufort Olivier Blanc Dragos Burileanu Tommaso Caselli Louise-Am´elie Cougnon Jolanta Cybulka ´ Damir Cavar Arantza D´ıaz de Ilarraza Pawel Dybala Nerea Ezeiza Byron Georgantopoulos Filip Grali´ nski Krzysztof Jassem

The reviewing process was eﬀected by the members of Program Committees and Invited Reviewers recommended by PC members.

Table of Contents

Speech Processing Data-Driven Approaches to Objective Evaluation of Phoneme Alignment Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ladan Baghai-Ravary, Greg Kochanski, and John Coleman

1

Phonetically Transcribed Speech Corpus Designed for Context Based European Portuguese TTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maria Barros and Bernd M¨ obius

12

Robust Speech Recognition in the Car Environment . . . . . . . . . . . . . . . . . . Agnieszka Betkowska Cavalcante, Koichi Shinoda, and Sadaoki Furui Corpus Design for a Unit Selection TtS System with Application to Bulgarian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Aimilios Chalamandaris, Pirros Tsiakoulis, Spyros Raptis, and Sotiris Karabetsos Automatic Identiﬁcation of Phonetic Similarity Based on Underspeciﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mark Kane, Julie Mauclair, and Julie Carson-Berndsen Error Detection in Broadcast News ASR Using Markov Chains . . . . . . . . Thomas Pellegrini and Isabel Trancoso Pronunciation and Writing Variants in an Under-Resourced Language: The Case of Luxembourgish Mobile N-Deletion . . . . . . . . . . . . . . . . . . . . . . Natalie D. Snoeren, Martine Adda-Decker, and Gilles Adda Morpheme-Based and Factored Language Modeling for Amharic Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Martha Yiﬁru Tachbelie, Solomon Teferra Abate, and Wolfgang Menzel The Corpus Analysis Toolkit - Analysing Multilevel Annotations . . . . . . . Stephen Wilson and Julie Carson-Berndsen

24

35

47

59

70

82

94

Computational Morphology/Lexicography Time Durations of Phonemes in Polish Language for Speech and Speaker Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bartosz Zi´ olko and Mariusz Zi´ olko

105

XVI

Table of Contents

Polysemous Verb Classiﬁcation Using Subcategorization Acquisition and Graph-Based Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fumiyo Fukumoto, Yoshimi Suzuki, and Kazuyuki Yamashita

115

Estimating the Proximity between Languages by Their Commonality in Vocabulary Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yves Lepage, Julien Gosme, and Adrien Lardilleux

127

Toposlaw – A Lexicographic Framework for Multi-word Units . . . . . . . . . Malgorzata Marciniak, Agata Savary, Piotr Sikora, and Marcin Woli´ nski

139

Parsing Parsing CFGs and PCFGs with a Chomsky-Sch¨ utzenberger Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mans Hulden

151

Syntactic Analysis Using Finite Patterns: A New Parsing System for Czech . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vojtˇech Kov´ aˇr, Aleˇs Hor´ ak, and Miloˇs Jakub´ıˇcek

161

Using SRX Standard for Sentence Segmentation . . . . . . . . . . . . . . . . . . . . . Marcin Milkowski and Jaroslaw Lipski Using Lexicon-Grammar Tables for French Verbs in a Large-Coverage Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elsa Tolone and Benoˆıt Sagot

172

183

Computational Semantics Eﬀect of Overt Pronoun Resolution in Topic Tracking . . . . . . . . . . . . . . . . Fumiyo Fukumoto and Yoshimi Suzuki

192

Sentiment Intensity: Is It a Good Summary Indicator? . . . . . . . . . . . . . . . . Mijail Kabadjov, Alexandra Balahur, and Ester Boldrini

203

Syntactic Tree Kernels for Event-Time Temporal Relation Learning . . . . Seyed Abolghasem Mirroshandel, Mahdy Khayyamian, and Gholamreza Ghassem-Sani

213

The WSD Development Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rafal Mlodzki and Adam Przepi´ orkowski

224

Semantic Analyzer in the Thetos-3 System . . . . . . . . . . . . . . . . . . . . . . . . . . Julia Romaniuk, Nina Suszcza´ nska, and Przemyslaw Szmal

234

Unsupervised and Open Ontology-Based Semantic Analysis . . . . . . . . . . . Amal Zouaq, Michel Gagnon, and Benoˆıt Ozell

245

Table of Contents

XVII

Entailment Non Compositional Semantics Using Rewriting . . . . . . . . . . . . . . . . . . . . . . Paul B´edaride and Claire Gardent Deﬁning Specialized Entailment Engines Using Natural Logic Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elena Cabrio and Bernardo Magnini

257

268

Dialogue Modeling and Processing Czech Senior COMPANION: Wizard of Oz Data Collection and Expressive Speech Corpus Recording and Annotation . . . . . . . . . . . . . . . . . Martin Gr˚ uber, Milan Leg´ at, Pavel Ircing, Jan Romportl, and Josef Psutka Abstractive Summarization of Voice Communications . . . . . . . . . . . . . . . . . Vincenzo Pallotta, Rodolfo Delmonte, and Antonella Bristot Natural Language Based Communication between Human Users and the Emergency Center: POLINT-112-SMS . . . . . . . . . . . . . . . . . . . . . . . . . . Zygmunt Vetulani and Jacek Marciniak Dialogue Organization in Polint-112-SMS . . . . . . . . . . . . . . . . . . . . . . . . . . . Justyna Walkowska

280

291

303

315

Digital Language Resources Valuable Language Resources and Applications Supporting the Use of Basque . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I˜ naki Alegria, Maxux Aranzabe, Xabier Arregi, Xabier Artola, Arantza D´ıaz de Ilarraza, Aingeru Mayor, and Kepa Sarasola

327

Clues to Compare Languages for Morphosyntactic Analysis: A Study Run on Parallel Corpora and Morphosyntactic Lexicons . . . . . . . . . . . . . . . Helena Blancafort and Claude de Loupy

339

Corpus Clouds - Facilitating Text Analysis by Means of Visualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chris Culy and Verena Lyding

351

Acquiring Bilingual Lexica from Keyword Listings . . . . . . . . . . . . . . . . . . . Filip Grali´ nski, Krzysztof Jassem, and Roman Kurc

361

Annotating Sanskrit Corpus: Adapting IL-POSTS . . . . . . . . . . . . . . . . . . . . Girish Nath Jha, Madhav Gopal, and Diwakar Mishra

371

XVIII

Table of Contents

Eﬀective Authoring Procedure for E-learning Courses’ Development in Philological Curriculum Based on LOs Ideology . . . . . . . . . . . . . . . . . . . . . . Galina Kedrova, Anatoly Yegorov, and Maria Volkova

380

Acquisition of Spatial Relations from an Experimental Corpus . . . . . . . . . Jedrzej Osi´ nski

388

Which XML Standards for Multilevel Corpus Annotation? . . . . . . . . . . . . Adam Przepi´ orkowski and Piotr Ba´ nski

400

Corpus Academicum Lithuanicum: Design Criteria, Methodology, Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Aurelija Usoniene, Linas Butenas, Birute Ryvityte, Jolanta Sinkuniene, Erika Jasionyte, and Algimantas Juozapavicius

412

WordNet The EM-Based Wordnet Synsets Annotation of NP/PP Heads . . . . . . . . . El˙zbieta Hajnicz Unsupervised Word Sense Disambiguation with Lexical Chains and Graph-Based Context Formalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Radu Ion and Dan S ¸ tef˘ anescu An Access Layer to PolNet – Polish WordNet . . . . . . . . . . . . . . . . . . . . . . . . Marek Kubis

423

435 444

Document Processing OTTO: A Tool for Diplomatic Transcription of Historical Texts . . . . . . . . Stefanie Dipper and Martin Schnurrenberger

456

Automatic Author Attribution for Short Text Documents . . . . . . . . . . . . . Monika Nawrot

468

BioExcom: Detection and Categorization of Speculative Sentences in Biomedical Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Julien Descl´es, Motasem Alrahabi, and Jean-Pierre Descl´es Experimenting with Automatic Text Summarisation for Arabic . . . . . . . . Mahmoud El-Haj, Udo Kruschwitz, and Chris Fox Enhancing Opinion Extraction by Automatically Annotated Lexical Resources (Extended Version) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrea Esuli and Fabrizio Sebastiani Technical Trend Analysis by Analyzing Research Papers’ Titles . . . . . . . . Tomoki Kondo, Hidetsugu Nanba, Toshiyuki Takezawa, and Manabu Okumura

478 490

500 512

Table of Contents

XIX

Information Processing (IR, IE, other) Extracting and Visualizing Quotations from News Wires . . . . . . . . . . . . . . ´ Eric de La Clergerie, Benoˆıt Sagot, Rosa Stern, Pascal Denis, Ga¨elle Recourc´e, and Victor Mignot

522

Using Wikipedia to Improve Precision of Contextual Advertising . . . . . . . Alexander Pak

533

Unsupervised Extraction of Keywords from News Archives . . . . . . . . . . . . Marco A. Palomino and Tom Wuytack

544

Machine Translation Automatic Evaluation of Texts by Using Paraphrases . . . . . . . . . . . . . . . . . Kazuho Hirahara, Hidetsugu Nanba, Toshiyuki Takezawa, and Manabu Okumura

556

Packing It All Up in Search for a Language Independent MT Quality Measure Tool – Part Two . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kimmo Kettunen

567

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

577

Data-Driven Approaches to Objective Evaluation of Phoneme Alignment Systems Ladan Baghai-Ravary, Greg Kochanski, and John Coleman Oxford University Phonetics Laboratory, 41 Wellington Square, Oxford, OX1 2JD, UK {ladan.baghai-ravary,greg.kochanski,john.coleman}@phon.ox.ac.uk

Abstract. This paper presents techniques for objective characterisation of Automatic Speech-to-Phoneme Alignment (ASPA) systems, without the need for human-generated labels to act as a benchmark. As well as being immune to the effects of human variability, these techniques yield diagnostic information which can be helpful in the development of new alignment systems, ensuring that the resulting labels are as consistent as possible. To illustrate this, a total of 48 ASPA systems are used, including three front-end processors. For each processor, the number of states in each phoneme model, and of Gaussian distributions in each state mixture, are adjusted to generate a broad variety of systems. The results are compared using a statistical measure and a model-based Bayesian Monte-Carlo approach. The most consistent alignment system is identified, and is (as expected) in close agreement with typical “baseline” systems used in ASR research. Keywords: Phonetic alignment, label accuracy, phoneme detectivity, objective evaluation.

1 Introduction How does one find the best Automatic Speech-to-Phoneme Alignment (ASPA) system? It is easy to generate many hidden Markov models (HMMs) for a given corpus of training data. However, even if the sequence of phonemes is considered to be known, the graph of states within each phoneme is largely arbitrary and the acoustic model for each state can be specified in many different ways. So there are many possible ASPA systems for a given corpus, and thus many possible alignments. To date, people have typically approached finding the best ASPA system by manually creating a “gold standard” set of labels, then running several systems on the labelled corpus and computing an error measure of the difference between the system and the human-generated labels (e.g. [3, 9]). The definitions of the both the error measure and the “gold standard” labels themselves are critical to this process. Furthermore, neither of these can be defined unambiguously in any mathematically optimal sense. To define the “gold standard” labels, humans typically segment speech into phonemes via a composite visual-auditory task, working from a spectrogram on a computer Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 1–11, 2011. © Springer-Verlag Berlin Heidelberg 2011

2

L. Baghai-Ravary, G. Kochanski, and J. Coleman

screen with audio playback of the selected segments. While this usually yields intuitively satisfying results, it bears little relationship to the auditory process of real-time speech perception, or to speech production. Thus there is no reason to believe that these labels are optimal in any strict sense, especially when the transition between articulatory gestures is not clear-cut (between one vowel and another, or between glides and vowels, for example). Further, even when using a formal labelling guide (e.g. [8]), different people will label speech differently. The effect of these differences could be minimised by getting a number of labellers to label the data independently, and then taking the median position (for example) of each label as being the “correct” value, but this becomes impractical when the standard corpus must be large, because many systems need to be tested on a wide range of speakers and/or acoustic environments. We use a different approach, designed for cases where internal consistency and robustness of labelling is crucial, and detailed agreement with human labels is secondary. Later, we will show it also gives good agreement with human labels, in fact. Our approach is based on measuring the consistency of pairs of ASPA systems, then searching for the system that is consistent with as many other ASPA systems as possible. This approach allows us to automate more of the labelling process. Not only does this reduce the cost of the process, but it also has the potential to reduce inconsistencies in the labels. Since Automatic Speech-to-Phoneme Alignment (ASPA) systems are invariably deterministic algorithms, they are consistent in the sense that they will always produce the same results when presented with the same data, but most applications, especially research applications, require results to be reproducible, even if the data is not identical. Also, different systems may focus on different aspects of the speech, resulting in discrepancies between the labels, but other researchers should be able to take the ideas behind one's research, follow through, and get similar results. Since exact replications are rare, one would also like other researchers to get similar results even if they do not use the exact same ASPA system. Here, we will search for the system whose results are as insensitive as possible to changes in the number of states per phoneme, the number of mixtures, and the choice of front-end. To identify which of these systems gives the most authoritative labels, we have compared a very large number of alignment results from different ASPA systems, and used these comparisons to evaluate each system’s performance. Those systems that produce labels which are positioned consistently with respect to those of other systems are deemed reliable.

2 Experiments A total of 48 ASPA systems were built using the HTK HMM Toolkit [13]. The systems varied in the number of states per phoneme, the number of Gaussian components per mixture, and the initial front-end processing used to produce the observation vectors. 2.1 Data The experiment used an ad hoc corpus assembled for other purposes. The 34 subjects were all speakers of Southern British English, and the utterances consisted of both

Data-Driven Approaches to Objective Evaluation of Phoneme Alignment Systems

3

single words and complete sentences. The recordings were made with different equipment and at different sampling rates, but all were digitally re-sampled to 16 kHz using the “rate” operation of SoX [12]. Overall, the database consists of just over 23,000 utterances, making a total of 48,000 spoken words taken from a vocabulary of 16,000. 2.2 Front-End Processing The three different pre-processors chosen were Mel-Frequency Cepstral Coefficients (MFCCs), Linear Prediction (LP) Cepstra, and Auditory Description Vectors. All were computed at 12.5 ms intervals. MFCCs are the de facto standard baseline for most speech recognition experiments. The HTK MFCC_D_0 implementation was used, i.e. 13 MFCC coefficients (including the zeroth energy coefficient) with 13 dynamic (“delta”) coefficients, with an analysis window of 25 ms. Linear prediction cepstral coefficients are broadly similar to MFCCs, but with frequency domain smoothing being an inherent part of the spectral analysis, rather than the perceptually based filter-bank smoothing of the MFCC. We used the LPCEPSTRA_E_D front-end of HTK, giving 13 static coefficients, including the energy, together with 13 dynamic coefficients, again with an analysis window of 25 ms. Auditory Description Vectors (ADVs), proposed in [6], were designed to mimic human perception of phonemes. The vector used here is derived from Equation 2 of [6] by a dimensionality-reducing linear transformation which yields 19 coefficients. The transformation is designed to maintain the distance between phonetically dissimilar data in a fashion similar to Linear Discriminant Analysis, as described in [11] and applied to auditory data in [2]. The definition for the initial acoustic description vectors differs slightly from that in [6]: primarily, a shorter time domain smoothing window is used here. The vector consists of: • • • •

The loudness in 1 erb-wide bins [10], smoothed with a 60 ms wide 1+cos2 window. Five broad-band spectral features computed from 20 ms windows. Five broad-band edge detectors, taking a difference across a 40 ms interval. Two entropy features. One is the entropy of the spectrum (computed over a 30 ms window). The other is a space-time entropy in a 70 ms window, which will also pick up changes from one frame to the next. The latter tends to be higher at phoneme edges and in rough voicing. • Two voicing features. One derived from the entire signal, the other from the highfrequency parts (above 1 kHz). • One “roughness” feature inspired by [4]. This is designed to reflect fluctuations in the auditory neural firing rate on 3-30 ms time scales. It would be large if fed a pair of pure tones 50 Hz apart, but small if the tones were 1 Hz apart or 1000 Hz apart. • Finally, one pseudo-duration feature [5] which has a variable window width, but one that is comparable to the phoneme duration. This 51-dimensional representation is then reduced to 19 dimensions by retaining the largest terms in an eigenvalue expansion.

4

L. Baghai-Ravary, G. Kochanski, and J. Coleman

2.3 HMM Training All the phoneme strings to be aligned with the speech were based on a lexicon compiled from diverse sources. An optional short-pause phoneme was added between words. No post-lexical rules were applied, so the phonetic transcriptions will have some inaccuracies. All the aligners compared in this work were based on broadly similar ContinuousDensity Hidden Markov Models (CD-HMMs), implemented using HTK. The number of states and the numbers of Gaussian mixtures in each state were varied to evaluate the effects of changing these parameters. The training process was similar for all the experiments: it used embedded reestimation using the Baum-Welch algorithm [13], applied in three phases [1], with four training iterations at each stage: • Training from flat-start HMMs, initialised to the global means of all the training data, to produce single-mixture phoneme, silence, and short-pause models. • Disambiguation of alternative pronunciations (including presence or absence of inter-word pauses) followed by re-training of the models. • Disambiguation as before, and an increase in the number of mixtures in each state (using a randomised duplication of each existing mixture), followed by final retraining of the full models.

3 Differences between Systems The methods described here assess the relative performance of each alignment system in terms of how consistently the labels are positioned, relative to other occurrences of the same label identified by other systems1. The former will produce close to 2000 potential label identities, making it difficult to present and generalise the results. By using broad phonetic classes for evaluation, instead of phonemes, the number of statistics can be reduced to 55, and the statistics themselves become more reliable, being derived from many more observations. Our approaches to measuring mutual consistency between aligners start by comparing the time of corresponding labels from each pair of systems; we compute the means and variance of these differences within a label class. If there are large mean differences between the labels produced by two systems, it does not necessarily imply that either system is individually poor, just that they have systematically different definitions for where one phoneme stops and the next one starts. It is only if the variance of the time differences is large that one system can be said to be inconsistent with the other. Thus the ability to replicate a set of HMM alignments can be assessed by the variances of the discrepancies in its labels with respect to the other systems. Suppose that one had a “bad” alignment system where label position is very sensitive to small changes in the local acoustic properties2. Contrast this with a “good” system where small changes in the acoustical properties lead to small changes in the 1 2

The word “label” here denotes the boundary between one phoneme and the next. These changes are differences between instances of the same phoneme, so they are all implicitly within the acoustic “borders” of the phoneme.

Data-Driven Approaches to Objective Evaluation of Phoneme Alignment Systems

5

label positions. When we compute the position of different instances of a given label, we are essentially giving a random sample of acoustical properties to the system. The variance of the difference will then be related to the strength of the relationship between acoustical properties and timing. Generally, greater sensitivity of either system will lead to a larger variance of the differences3. A high sensitivity of either system implies a large variance. Conversely, if a given system always yields a small variance when compared with other systems, it is likely to be “good” in the sense of having a relatively small sensitivity of phoneme boundary position to acoustic properties. The variance observed for different phoneme-class to phoneme-class transitions can be quite different. Labels for vowel-to-liquid transitions, for example, often differ widely without being fundamentally “wrong”, because human labellers treat such transitions as broad and unclear. Thus we treat each class-to-class transition separately. To check our conclusions, we compute two parallel analyses in this paper. One is a parametric Bayesian approach that incorporates a model of the systematic differences from one label to another; the other is a non-parametric test based on ranks of variances. 3.1 Non-parametric Ranking of Variances The simplest approach to measuring differences between ASPA systems starts with the mean and variance of corresponding labels for each pair of systems. Within each phoneme-class to phoneme-class transition, we order the different systems according to their variances. Then, to decide which system is most consistent, we average the system's ranking over all label classes. This averaged ranking can then be compared against all other systems. By using this non-parametric approach, we make no assumptions regarding the range of variances, or even the inherent linearity of the scale. A low average ranking implies that the system is generally consistent with many of the other systems and thus that the positions of the labels it generates are relatively insensitive to small changes in acoustic properties4. Such a system presumably responds to the most reliably identifiable features of the acoustic signal. Conversely a system with a high rank may yield labels that depend strongly on small acoustic changes and thus would yield results that would be hard to replicate on a different corpus. 3.2 Parametric Bayesian Models Another, parallel analysis builds a model that predicts where a given aligner will mark a given boundary. We search for the parameters of this model that best explain the observed boundary positions; some of the parameters correspond to systematic differences from one system to the next; other parameters correspond to the consistency of a system. 3

4

Potentially the two systems might have large but nearly identical dependencies on the local acoustical properties, but it is unlikely, given that representations are many-dimensional, and therefore there are many ways to differ. Therefore this possibility is ignored. Alternatively, this system could be part of a large group of systems that all behave similarly. If so, choosing it is a good way to make the experimental results robust, in case other groups who are repeating one's work choose a different ASPA system: they are likely to choose a similar system from the group.

6

L. Baghai-Ravary, G. Kochanski, and J. Coleman

Phoneme “Detectivity” We assume that (for a giveen alignment system) each class of phonemes tends too be systematically longer or shorter than the overall average. This can be thought of aas a class-dependent and alignerr-dependent “detectivity”. A given aligner tends to be m more attuned to certain phonemees: phonemes that an aligner detects more easily tendd to expand in both directions; phonemes p that are not easily detected by an aligner tendd to be shortened. Qualitatively, imagine a slice of sound in an ambiguous region5 between two pphomediate between the two, but whichever phoneme is eaasier nemes: the sound is interm for a given system to detecct will win. Quantitatively, if C is the broad phonetic cllass of a phoneme, we write: (1) We make the assumptions that more detectable phonemes expand in both directiions pansion is proportional to the width of the ambiguous reand that the amount of exp gion. Then, the boundary sh hift is: (2) where σab is the standard deviation d observed between the labels for transitions frrom class Ca to class Cb.; it rep presents the width of the ambiguous region. As a ressult, ambiguous boundaries wou uld be expected to have larger systematic shifts, Δt. Another motivation for this t model is that it expresses diphone properties in terrms of phoneme properties. Thaat means you only need approximately 45 numbers to ppredict the boundary positionss for all (over 1000) diphones. To the extent that it woorks well, it makes the overall reesults easier to compute and understand. Interpretation This model predicts system matic differences between one aligner and another, but thhere are also disagreements betw ween aligners that are not predictable from a knowledgee of the two neighbouring pho ones. The standard deviation of these unpredictable ddisagreements is the σab value mentioned above. b approximated as: We assume that σab can be (3) so that some aligners are prroportionally better than others (expressed by Kσ), and tthat some class-to-class boundaaries are easier to mark than others (KC). The smaller Kσ is, the better the aligner. The bigger b KC is, the more difficult it is to define the bounddary between two phoneme classes. The factor allows us to compare diphones. Finaally, o parameter) is the typical size of disagreements beσglobal (which is a single, overall tween aligners. Overall, between detectiivities, Kσ values, KC values and σglobal, there are 144 parameters in our current analysis. To estimate these parameters we have used an addaptive Markov-Chain Monte--Carlo process [7]. This algorithm can provide statisticaally valid confidence intervals for fo its parameters. 5

The ambiguity can be eitheer intrinsic in the sound or it could be due to a sharp acouustic boundary falling near the middle of a window used for front-end processing.

Data-Driven Approaches to t Objective Evaluation of Phoneme Alignment Systems

7

4 Results 4.1 Non-parametric Ranking Results For each front end, the optiimum (i.e. most mutually-consistent) set of HMM trainning parameters was found using g the approach described in section 3.1 above. The results for each front-end are show wn separately at the end of this section (Figures 1 to 3)). In these figures, a rank of zeero means that the respective ASPA system agrees m more closely than any other with at least one system, for all broad-class transition labels. Each of Figures 1 to 3 was derived from 16 different systems, giving 120 (i.e. (16 * 15) / 2) differentt system pairs, and the maximum possible rank of 1105. Thus a rank of 105 would mean m that the respective ASPA system always had a woorse discrepancy than any of thee others. It is clear from these reesults that, regardless of front-end, 4 states per phoneeme gives the best performancee (or very close to it). The optimal number of mixturees is either 2 or 4 depending on the t front-end. The results for all front ends are shown together in Figure 4; it focuses on 4 staates m components. per phoneme and 2 and 4 mixture

Fig. 1. Inconsistency between labels, expressed as the mean rank of systems with MFCC ffront ends

Fig. 2. Inconsistency between labels of systems with LP Cepstrum front ends

8

L. Baghai-Ravary, G. Kochanski, and J. Coleman

Fig. 3. Inconsisteency between labels of systems with ADV front ends

Fig. 4. Inconsistency between labels of the best systems for each front end (4 states and 2 or 4 mixtures)

4.2 Parametric Bayesian Results The figures below show sy ystem performance vs. front end, number of Gaussians per state, and number of states per p phoneme, as quantified by the equation: (4) as derived from the paraametric Bayesian models. This reflects each alignm ment system's overall inconsisten ncy: it is the geometric mean of σab for a system. The ppanels show the MFCC-based d aligners (Figure 5), LPC-based aligners (Figure 6) and ADV-based aligners (Figurre 7). The vertical axis represents the standard deviationn in seconds. i Figure 7, below, has a very different structure to thosee of Note that the ADV plot in the “traditional” MFCC and d LP Cepstrum front-ends. 4.3 Discussion As can be seen in Figures 5 to 7, the parametric Bayesian analysis gives similar quualitative results to the non-paarametric ranking analysis for the MFCC and LPC-baased

Data-Driven Approaches to t Objective Evaluation of Phoneme Alignment Systems

9

Fig. 5. Alignment discrepancy y standard deviations (in seconds) for systems with MFCC ffront ends

Fig. 6. Alignment discrepancy y standard deviations (in seconds) for systems with LP Cepsttrum front ends

y standard deviations (in seconds) for systems with ADV ffront Fig. 7. Alignment discrepancy ends

analyses: the number of sttates is the critical factor. Three or four states are m much better than one or five, as can c be seen by the lower values in the central two colum mns. Four states are slightly prefferred to three. For both analyses, the number of Gaussiians

10

L. Baghai-Ravary, G. Kochanski, and J. Coleman

per state makes little difference to the performance as long as there is more than one. Some disagreement is apparent in that the ADV results show a preference for two (or even one) mixtures in the parametric Bayesian case, but no real preference when using the non-parametric ranking approach. Overall, MFCC performance is better than LPC by approximately 15% and in this analysis, both LPC and MFCC are dramatically better than the ADV-based systems (with standard deviations roughly three times larger). As most of the windows involved in the ADV calculations are larger than the 25 ms windows used in LPC and MFCC, it is not too surprising that the ADV captures timing less precisely and therefore gives less precise labels. As regards the diagnostic value of these analysis methods, Figures 1 to 4 show that there is a fundamental difference between the ADV processor and the others. This difference manifests itself as a bias towards models with fewer mixture components but more states. This suggests that the ADV processor may be responding to temporal features on a different scale from conventional methods. The ADV front end may be better at resolving sub-phonemic details, and that would also explain its apparent variability in the exact timing of its boundaries: it may be responding to sub-phonemic details in the acoustic signal, which are invisible to the other processors.

5 Conclusions Both the “non-parametric ranking” and “parametric Bayesian” methods show that the LP Cepstrum front end is roughly comparable to MFCCs, but the current implementation of Auditory Description Vectors, at least for the specific phoneme alignment/labelling task described here, gives less consistent labels. For this task, the non-parametric ranking analysis suggests the optimal HMM configuration uses 4 states per phoneme model and 2 mixture components with the LP Cepstrum front end. The parametric Bayesian analysis prefers MFCCs, with 4 states and 4 mixtures, but the differences between the two systems are small for both analysis methods. Thus it can be inferred that MFCCs and LP Cepstra are broadly similar, the best number of states per phoneme is 4, with 2 to 4 Gaussians per mixture. This is slightly more states than are normally used in ASR (i.e. 3), and significantly fewer Gaussians than most speaker-independent ASR systems (8 or more). The most likely reason for this small number of mixtures is that the results presented here were derived from controlled recordings of a single dialect. A more diverse range of speakers and speaking styles would be expected to require more Gaussians per state. More importantly, we have demonstrated two related methods for quantifying the reliability of labelling systems from the discrepancies between labels from a cohort of automatic systems, without any manually created reference labels. This facilitates rapid progress in optimising the design of any alignment system, and frees that process from the need for extensive manual intervention.

References 1. Baghai-Ravary, L.: Multi-dimensional Adaptive Signal Processing, with Application to Speech Recognition, Speech Coding and Image Compression. University of Sheffield PhD. Thesis (1995)

Data-Driven Approaches to Objective Evaluation of Phoneme Alignment Systems

11

2. Beet, S.W., Gransden, I.R.: Interfacing an Auditory Model to a Parametric Speech Recogniser. Proc. Insititute of Acoustics 14(6), 321–328 (1992) 3. Chen, L., Liu, Y., Maia, E., Harper, M.: Evaluating Factors Impacting the Accuracy of Forced Alignments in a Multimodal Corpus. In: 4th International Conference on Language Resources and Evaluation (LREC), ELRA (2004) 4. Hutchinson, W., Knopoff, L.: The Acoustic Component of Western Consonance. Interface 7, 1–29 (1978) 5. Kochanski, G., et al.: Loudness Predicts Prominence; Fundamental Frequency Lends Little. J. Acoustical Society of America 11(2), 1038–1054 (2005) 6. Kochanski, G., Orphanidou, C.: Testing the Ecological Validity of Repetitive Speech. In: Proc. International Congress of Phonetic Sciences (ICPhS 2007), IPA (2007), http://www.icphs2007.de/conference/Papers/1632/1632.pdf 7. Kochanski, G., Rosner, B.S.: Bootstrap Markov Chain Monte Carlo and Optimal Solutions to The Law of Categorical Judgement (Corrected). Submitted to Behavior Research Methods (2010), http://arxiv.org/abs/1008.1596 8. Lander, T.: CSLU Labeling Guide, Center for Spoken Language Understanding, Oregon Graduate Institute (1997) 9. Ljolje, A., Riley, M.D.: Automatic Segmentation of Speech for TTS. In: Proc 3rd European Conference on Speech Communication and Technology (EUROSPEECH 1993), ESCA, pp. 1445–1448 (1993) 10. Moore, B.C.J., Glasberg, B.R.: Suggested Formulae for Calculating Auditory-Filter Bandwidths and Excitation Patterns. J. Acoustical Society of America 74(3), 750–753 (1983) 11. Sebestyen, G.S.: Decision-Making Processes in Pattern Recognition. ACM Monograph Series, pp. 40–47. MacMillan, Basingstoke (1962) 12. SoX Sound eXchange manual (2009), http://sox.sourceforge.net/sox.html 13. Young, S.J., et al.: The HTK Book (for HTK Version 3.4). Cambridge University Engineering Department (2009), http://htk.eng.cam.ac.uk/docs/docs.shtml

Phonetically Transcribed Speech Corpus Designed for Context Based European Portuguese TTS Maria Barros1 and Bernd Möbius2 1

Eurescom GmbH, Wieblinger Weg 19/4, 69123 Heidelberg, Germany 2 University of Bonn, Am Hof 1, 53113 Bonn, Germany [email protected], [email protected]

Abstract. This paper presents a speech corpus for European Portuguese (EP), designed for context based text-to-speech (TTS) synthesis systems. The speech corpus is intended for small footprint engines and is composed by one sentence dedicated to each sequence of two phonemes of the language, incorporating as many language contexts as possible at diphone and word levels. The speech corpus is presented in three forms: its orthographic sentences, its phonetic transcriptions considering the words coarticulation, typical phenomenon in EP and its phonetic transcriptions considering the words coarticulation and the vocalic reduction effects, which is another typical phenomenon of EP language. The paper describes the corpus and presents the results in total number of phonemes and their distribution in the word and the sentence, and the total number of diphones, considering and not considering the vocalic reduction for the phonetic transcription. Keywords: Speech corpus, European Portuguese corpus, text-to-speech corpus, context based corpus.

1 Introduction In a TTS system, the speech synthesis quality is strongly affected by the quality of the speech corpus, the quality of the natural language processing module and the quality of the speech generation module. The speech corpus quality, on its turn, depends on the quality of the corpus design, the quality of the annotations and transcriptions and the quality of the recording system. Nowadays TTS systems are usually corpus based statistical systems. This means that they rely on the quality of the corpus used to produce better or worse results. Besides that, they are often systems using the language contexts for choosing the best speech units for a particular target. This makes it extremely important that the speech corpus design includes as many contexts as possible. Usually, statistical methods are used to search in large amounts of texts, in the intended speech units in the considered language contexts. For this reason the speech corpus for generic vocabulary systems are very large, sometimes too large to be considered. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 12–23, 2011. © Springer-Verlag Berlin Heidelberg 2011

Phonetically Transcribed Speech Corpus Designed for Context Based EP TTS

13

The contextual factors that can be used are many. For example, the contextual factors used for the EP HMM-based speech synthesis system (Barros et al, 2005), considering the hierarchical structure, i.e., phone, syllable, word, phrase and utterance, were the following: At phone level: current, previous and next phones; phones before previous phone and after next phone; and positions (forward and backward) of current phone in current syllable; At syllable level: stress condition of current, previous and next syllables; number of phones in previous, current and next syllable; positions (forward and backward) of current syllable in current word and in current phrase; number of stressed syllables before and after current syllable in the current phrase; syllable counts between the previous stressed syllable and the current syllable and between the current syllable and the next stressed syllable in the utterance; and vowel of the syllable; At word level: number of syllables in the current, previous and next words; and positions (forward and backward) of current word in current phrase; At phrase level: number of syllables and number of words in current, previous and next phrases; and positions (forward and backward) of current phrase in the utterance; At utterance level: number of syllables, words and phrases in the utterance. As a solution towards the best quality language context based systems with small footprint engines, efforts were concentrated in a new, manually designed, speech corpus. The speech corpus intends to achieve a large amount of important language contexts within a limited size. To do so, one sentence was constructed targeting each sequence of two phonemes of the language. In this sentence, it was tried to include as many contexts as possible. Through statistical searches it is possible to obtain as many features as it is needed, but it would need a very large corpus and that is difficult for languages without many resources. This speech corpus is designed for European Portuguese, although the idea can be extrapolated to other languages. The same data should be used to train the natural language processing (NLP) tasks and the synthesis engine, as this way achieves a better synchronization between the units selected by the NLP and the units that can be found in the synthesizer database.

2 Methodology The methodology used to design the corpus took into consideration the 38 phonemes from EP, presented in figure 1, and the silence unit, to construct one sentence dedicated to each of these 39 units combined with each other and themselves. Excluding the combination of silence with itself there were 1520 possible sentences to be constructed. The result is a speech corpus with 1436 sentences, once the rest of the combinations were not possible due to language rules that will be explained in the description section. The sentences were all manually constructed, trying to take into account a number of language context factors and considering, but not limiting to, the most used words in the lists relating to European Portuguese usage of the words, collected from the first Morpholympics for Portuguese (Linguateca, 2003). These lists were extracted from a set of 613 different texts, from different fields (literature, politics, advertising,

14

M. Barros and B. Möbius

informal chats, general news, etc.), collected through different sources (newspapers, books, net, advertising, etc.), containing 80.903 text units correspondent to 17.128 different units. More about the description of the texts and the lists, with statistics, is available in the site (Linguateca, 2003).

Fig. 1. EP Phoneme Inventory, in SAMPA and in IPA

The contextual factors that were considered when building the sentences are: At the diphone level, occurrences of the diphone: positioned at the beginning/end of the sentence; positioned at the beginning/middle/end of the word; and between two words. At the word level, occurrences of the: words containing the target diphone at the beginning/middle/end of the sentence; and combinations with the target diphone between two words positioned at the beginning/middle/end of the sentence. The speech corpus is presented in three forms: ¾ ¾ ¾

The orthographic sentences; A phonetic transcription considering the words coarticulation (that is a natural effect in EP continuous speech), following the grapheme-to-phoneme conversion (G2P) rules for the EP language; A phonetic transcription considering the words coarticulation and the vocalic reduction effects, common in EP language.

Fig. 2. Example of the sentences in the Speech Corpus

Phonetically Transcribed Speech Corpus Designed for Context Based EP TTS

15

Figure 2 shows two examples of sentences, for the phonetic sequences /vu/ and /dg/, marking with circles all the possibilities of the sequence occurrence, independently of have been verified or not. In the example it is possible to see the difference in the number of the sequence occurrences for both types of phonetic transcription and verify that some diphones don’t exist in the language but can appear when considering vocalic reduction.

3 Description The speech corpus presented here has a total of 1436 sentences, comprising 5853 different words, with a total of 21500 words occurrences. Two phonetic transcriptions for the orthographic sentences are provided, one following the G2P rules for EP language and the other considering the vocalic reduction effect that is common in the EP language. The number of words in the phonetic speech corpus transcribed following the G2P rules is 6264 and in the one taking the vocalic reduction into consideration, it is 6415. The difference in the number of words is due to the different transcriptions for some words when considering the vocalic reduction effect. The effect of coarticulation between words is present in EP continuous speech. Regarding to this effect the same word can have different transcriptions, because there are graphemes that have different phonetic transcriptions according to the following one. For example, the grapheme is transcribed as a /S/, if at the end of a sentence or followed by an unvoiced consonant (/p, t, k, s, S, f/ for EP); as a /Z/, if followed by a voiced consonant (/b, d, g, m, n, J, z, v, Z, l, l~, L, r, R/ for EP); or as a /z/, if followed by a vowel. Another example is the grapheme , which is transcribed as a /l~/, if at the end of a sentence or followed by a consonant; or as a /l/, if followed by a vowel. One example of these situations in EP words is the word , which is transcribed as /vajS/, if it is followed by a word starting with an unvoiced consonant or if it is at the end of a sentence; as /vajZ/, if it is followed by a voiced consonant; and as /vajz/, if it is followed by a vowel. Another example is the word , which is transcribed as /mal~/, if it is followed by a word starting with a consonant or if it is at the end of a sentence; and as /mal/, if it is followed by a vowel. The vocalic reduction is another effect present in EP continuous speech. It can be reflected by the suppression of /@/, by the phoneme /u/ reduction or suppression, or by the phonemes /u~/, /i/ and /i~/ reduction. Considering this effect, some language contextual factors are present that otherwise would not be found: The phoneme /@/ suppression can happen in the middle of the words or at the end of the words or sentences. Some examples of this phenomenon are: that would be transcribed as /d/ instead of /d@/; that would be transcribed as /ddik6/ instead of /d@dik6/; that would be transcribed as /pOt/ instead of /pOt@/; and that would be transcribed as /6ptEs/ instead of /6p@tEs@/. The phoneme /u/ reduction or suppression happens in the middle of the words or at the end of the words or sentences. When the vowel /u/ suffers reduction it gives place to the semi-vowel /w/. An example of a word which usually presents /u/ suppression and reduction is the word . Following the G2P rules this word is transcribed

16

M. Barros and B. Möbius

as /sEkuluS/, but is commonly found as /sEklwS/ due to vocalic reduction. The paper (Barros et al, 2001) covers this phenomenon. The phonemes /u~/, /i/ and /i~/ can suffer reduction, giving place to the semivowels /w~/, /j/ and /j~/, respectively, but they cannot suffer suppression. Examples of these situations are: , which would be transcribed as /ko~Zw~tur6/ instead of /ko~Zu~tur6/, , which would be transcribed as /ljtwral~/ instead of /litural~/, and , which would be transcribed as /j~ve~t6/ instead of /i~ve~t6/. In EP many of the consonants can not be found next to each other in the same word. Considering the case of the phoneme /@/ vocalic reduction means to include almost all the combinations between consonants that in other ways would not exist. a)

b)

c)

Fig. 3. Number of: a) Grapheme occurrences; b) Phoneme occurrences, by rules; c) Phoneme occurrences, with vocalic reduction

3.1 The Speech Corpus by Graphemes The 1436 sentences speech corpus comprises of 36 graphemes (considering the graphemes with accents - à, á, â, ã, é, ê, í, ó, ô, õ, ú - as individual units) and a total of 101750 graphemes occurrences. The total number of occurrences of each grapheme is shown in figure 3 a). Although is not a Portuguese grapheme it can appear in some foreigner words adopted in Portuguese vocabulary. In this case it appears in the word . The is a particular case in Portuguese language because it only appears with the contraction between and it's not possible in the middle of a word. For

Phonetically Transcribed Speech Corpus Designed for Context Based EP TTS

17

example, the sentence (meaning ) has a grammar construction that comes from (that would mean: ). The examples present in the speech corpus are found in the words: , , , and . The is the only consonant that cannot appear in the beginning of a Portuguese word. This grapheme is always phonetically transcribed as /s/, which in the beginning of a word is produced by , or if followed by . The , or are the vowels that cannot appear in the beginning of a word. Relating to the last two, the nasal vowels, the is used to nasalize the vowel and in cases with the vowel in the beginning of the word, the vowel is nasalized by using the consonants in front of it. The , and are the most common endings for masculine words, feminine words and plurals, respectively. The infinitive form of Portuguese verbs always ends with , what justifies the large amount of this grapheme’s occurrences in the end of words and sentences. From the Portuguese grammar rules, besides the consonants and , already mentioned above, only the consonants , , and can appear at the end of a word. There are no cases of words ending with or . In the last case, in EP the nasal sound in the end of the word is achieved by following the vowel with . The is a special consonant in EP, once by itself does it not have a sonorous transcription, as it lost its aspiration with the evolution of the language. It is followed only by vowels, not by consonants, and it is used for the combinations , and that produce the phonemes /S/, /J/ and /L/, respectively. 3.2 Phonetic Transcription by Rules The first phonetic transcription presented follows the G2P rules for the Portuguese language taking into consideration the words coarticulation natural in EP continuous speech, but not the vocalic reduction effect. The 1436 sentences speech corpus comprises 38 phonemes and 92618 phonemes occurrences. The total number of occurrences of each phoneme is shown in figure 3 b). In European Portuguese language it is not possible to have words starting with the phonemes /l~/, /r/ or /J/, even considering phonetic effects inherent to continuous speech, unless it is some particular word like words imported from other Portuguese language dialects. This is the case of the word , imported from Angola Portuguese dialect to represent the name of an African bird and a place in Angola. When considering the phonetic transcription following the G2P rules it is not possible to have words starting with a semivowel, although this can happen when considering the vocalic reduction effect. All the vowels except /e~/ can be found at the end of a word or sentence. In the consonants case, the /d/, /k/, /g/, /t/, /J/, /v/, /s/, /L/, /f/, /n/, /m/, /l/, /p/ and /R/ are never found at the end of a word or sentence, unless due to particular cases of foreign words imported to European Portuguese vocabulary like, for instance, , , , and .

18

M. Barros and B. Möbius

The consonants /Z/, /z/ and /l/ cannot be found at the end of a sentence or of an isolated word, but they can be found at the end of a word in continuous speech due to the words’ coarticulation effect. The consonants /S/, /l~/ and /r/ can be found at the end of a sentence or isolated word. Figure 5 presents a table with the total number of occurrences of each diphone when considering the phonetic transcription by rules, where the columns present the first phoneme of each diphone and the rows present the second one. Any non-nasal vowel can only be next to a nasal vowel, or vice versa, if there is a small silence between words, in cases of slow rhythm of speech, because due to the words coarticulation effect a word that starts with a nasal vowel nasalizes the vowel from the precedent word and vice versa. Most of the diphones composed by two consonants are not present in European Portuguese language, but there are some cases that are common: /pr/, /pl/, /br/, /bl/, /fr/ and /fl/; /Z/ followed by a voiced consonant; /S/ followed by an unvoiced consonant; /ks/ that transcribes the for some grammatical cases (Barros and Weiss, 2006); /ps/, /pn/, /pt/, /kt/, /kn/, /gn/, /tn/, /gm/ and /tm/; /bt/, /bS/, /bZ/, /bz/, /bs/, /bv/ and /bm/; /dZ/, /dv/, /dm/ and /dr/; and /l~S/. The followed by a vowel is always transcribed as /l/, even if between words, unless there is a small silence in between the words, caused for instance by a comma, when then it is transcribed by a /l~/. It is also not possible to have /l/ or /l~/ followed by /r/, not because of the construction, that exists, but because in this case the is transcribed by /R/ if in the same word, and there are no words starting with a /r/. Another case related to not possible in EP is having /l/ or /l~/ followed by /J/, because is not an allowed construction and there are no words starting with a /J/. Double "l", , doesn't exist in European Portuguese words, what makes the diphones among /l~/, /l/ and /L/ in the same word not possible. The /l~/ and the /L/ can never be preceded by consonant or a nasal vowel in the same word, as this would implicate a construction of three following consonants in a way that is not possible in EP language. Having a nasal vowel before a /l~/ implicates after the vowel to nasalize it. So, with these cases, as well as with the consonants cases, it would be needed to have a consonant before the and another after, to turn it in /l~/. It is also not possible to find these cases between words because there are no words starting with /l~/. It's not possible to have /J/ preceded by a consonant, as it would lead to another type of construction of three consonants that is not allowed in European Portuguese language, the that gives the /J/ and any other consonant before. The same applies to nasal vowels preceding a /J/, because to have a nasal vowel before a it would have to be constructed by following the vowel with a or . It's not possible to have/r/ in the same word after another /r/ or /R/, because the double is always read as /R/. It is also not possible to have the /r/ after a nasal vowel in the same word, because it's the , or the , after a vowel that makes the nasal sound, but after a consonant the is always read as /R/. There is no possibility of having these cases between words because there are no words starting with /r/.

Phonetically Transcribed Speech Corpus Designed for Context Based EP TTS

19

It's not possible to have /n/ or /m/ preceded by a vowel and followed by a consonant, because in these cases the and the are used to nasalize the precedent vowel. Following this rule, it's not possible to have /n/ or /m/ followed by a consonant, because it would need a word construction of three consonants in a way that is not allowed in European Portuguese. There is one exception to these statements, which is the case of followed by and preceded by , for example in the word or any word using the Portuguese prefix , which are transcribed as /6mniSti6/ and /Omni/, respectively. The phoneme /e~/ is not found in the end of a word or after /e~/, /6~/ and /o~/, unless by influence of some regional accents/dialects, for instance in , and , which are transcribed as /le~6~j~/, /t6~j~6~j~/ and /vo~6~j~/, but could be found in some regional accents as /le~e~/, /t6~e~/ and /vo~e~/, respectively. 3.3 Phonetic Transcription with Vocalic Reduction The other phonetic transcription implemented to present the speech corpus considers the vocalic reduction effect that is common in the EP language, besides the words coarticulation. Due to the vocalic reduction there are less phonemes occurrences in this transcription than in the phonetic transcription by rules. The 1436 sentences speech corpus comprises of 38 phonemes and 84846 phonemes occurrences. The total number of occurrences of each phoneme is shown in figure 3 c). As it was explained before, in EP language it is not possible to have words starting with the phonemes /l~/, /r/ or /J/, even considering phonetic effects inherent to continuous speech, unless it is some particular word like words imported from other Portuguese language dialects. Following the G2P rules it is not possible to have words starting with a semivowel, but this can happen when considering the vocalic reduction effect. Due to the vocalic reduction effect, the vowel /@/ can be suppressed. This makes it possible to find almost all the consonants combinations. Also the consonant /S/ can be found before a voiced consonant from the transcription of or that becomes /S/ instead of /S@/. In European Portuguese language, the diphones /Ou/, /Ej/, /@j/, /aE/, /@E/, /aa/, /6a/, /ea/, /Ea/, /oa/ and /Oa/ aren't found in the same word, but can be found between words, both considering or not considering the vocalic reduction effect. Figure 6 presents a table with the total number of occurrences of each diphone when considering in the phonetic transcription the vocalic reduction effect, where the columns present the first phoneme of each diphone and the rows present the second one. The /@/ is suppressed in most of the cases of continuous speech, if considering the vocalic reduction effect, in any context of word or sentence.

4 The Vocalic Reduction Influence To better understand the influence of the vocalic reduction in EP continuous speech, a comparison between both phonetic transcriptions, following the G2P rules and

20

M. Barros and B. Möbius

considering the vocalic reduction, in different contexts at word and sentence levels is presented. Figure 4 a) presents a chart with the number of occurrences of each phoneme when in the middle of a word, for both phonetic transcriptions, following the G2P rules and considering the vocalic reduction. The same kind of chart is presented for the phonemes at the beginning of a word, in figure 4 b), at the end of a word, in figure 4 c), at the beginning of a sentence, in figure 4 d), and finally at the end of a sentence, in figure 4 e). a)

b)

c)

d)

e)

Fig. 4. Phonemes occurrences at the: a) Middle of word; b) Beginning of word; c) End of word; d) Beginning of sentence; e) End of sentence

From the figures it is possible to see that the biggest differences are in the number of occurrences of the phoneme /@/, the semi-vowels and the correspondent vowels. It is also possible to observe that the number of occurrences of some consonants in a particular context change if considering or not the vocalic reduction.

Phonetically Transcribed Speech Corpus Designed for Context Based EP TTS

Fig. 5. Total number of diphone occurrences, by rules

21

22

M. Barros and B. Möbius

Fig. 6. Total number of diphone occurrences, considering the vocalic reduction

Phonetically Transcribed Speech Corpus Designed for Context Based EP TTS

23

5 Conclusions The paper presents a speech corpus for EP, for small footprint context based text-tospeech TTS systems, with a reduced size, but composed by sentences incorporating each diphone of the language in as many language contexts as possible. Besides considering diphones, it also considers sequences of phonemes that would not constitute a diphone of EP, but appear in EP continuous speech due to the vocalic reduction effect. The speech corpus is presented in its orthographic sentences and in its phonetic transcriptions considering the words coarticulation only and also the vocalic reduction effects. An analysis between both phonetic transcriptions is presented, in order to understand the importance of this effect in EP. The contextual factors considered for the corpus design were the occurrences of each diphone positioned at the beginning and end of the sentence, or at the beginning, middle and end of the word, and between two words, at the diphone level, and the occurrences of the words containing the target diphone at the beginning/middle/end of the sentence, or combinations with the target diphone between two words positioned at the beginning/middle/end of the sentence, at the word level.

References 1. Barros, M.J., Maia, R., Tokuda, K., Freitas, D., Resende, F.G.: HMM-based European Portuguese Speech Synthesis. In: Interspeech 2005, Lisbon, Portugal (2005) 2. Linguateca: The first evaluation contest for morphological analysers of Portuguese (2003), http://www.linguateca.pt/Morfolimpiadas/ (last visited on 02-09-2009) 3. Barros, M.J., et al: Backclose Nonsyllabic Vowel [u] Behavior in European Portuguese: Reduction or Supression. In: ICSP 2001, Taejon, South Korea (2001) 4. Barros, M.J., Weiss, C.: Maximum Entropy Motivated Grapheme-to-Phoneme, Stress and Syllable Boundary Prediction for Portuguese Text-to-Speech. In: IV Biennial Workshop on Speech Technology, Zaragoza, Spain (2006)

Robust Speech Recognition in the Car Environment Agnieszka Betkowska Cavalcante1, , Koichi Shinoda2 , and Sadaoki Furui2 1

Telcordia Poland, ul. Umultowska 85, 61-614 Poznan, Poland [email protected] 2 Tokyo Institute of Technology, 2-12-1, Ookayama, Meguro-ku, Tokyo, 152-8552 Japan {shinoda@cs,[email protected]}.titech.ac.jp

Abstract. In this study we focus on robust speech recognition in car environments. For this purpose we used weighted finite-state transducers (WFSTs) because they provide an elegant, uniform, and flexible way of integrating various knowledge sources into a single search network. To improve the robustness of the WFST speech recognition system, we performed nonlinear spectral subtraction (SS) to suppress noise from the noisy speech. Using the “clean” speech signal obtained from SS, we conducted supervised WFST network adaptation to the characteristics of a given driver. In the best case, for highly noisy conditions, the speaker dependent WFST decoder achieved 70 percentage points improvement when compared with traditional speaker independent speech recognition systems. Keywords: robust speech recognition, car environment, Weighted finitestate transducers (WFSTs).

1

Introduction

Modern cars are being equipped with various electronic devices such as mobile phones, radios, and navigation systems. These give drivers more functionality and help, but their manipulation increases the risk of car accidents because they distract the drivers from the main task: driving. Therefore, a great deal of eﬀort has been devoted to increasing safety by devising unobtrusive devices with hands-free interface. Automatic speech recognition (ASR) technology is the most promising handsfree interface. However, this technology must be robust to be successful, i.e., it must work well in the presence of diﬀerent noises appearing in car environments (street noise, engine noise, stray speech, etc), and it must be able to recognize

This work was done while the first author was a project researcher with the Department of Computer Science, Tokyo Institute of Technology.

Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 24–34, 2011. c Springer-Verlag Berlin Heidelberg 2011

Robust Speech Recognition in the Car Environment

25

speech of diﬀerent speakers, each of whom has a unique speaking style and voice characteristics. Two common approaches to increase the robustness of ASR systems are the manipulation of the input signal, which removes unwanted noise [1], and the optimization of a given speech model, which tries to improve the modeling of the signal for a given environment and speaker [2]. Current state-of-the-art ASR systems, frequently based on Hidden Markov Models (HMM) [3], are complex because they must deal with various knowledge sources at multiple levels, such as an acoustic model (the acoustic signal representation), a dictionary (transcription of words), and a language model (grammar representation), etc. Usually, each level of the ASR system is modeled and optimized separately because they have diﬀerent model representation. Joint optimization of the overall system is diﬃcult because the information about the relations between diﬀerent sources is not available. Recently, a new approach based on weighted ﬁnite-state transducers (WFSTs) [4] has received considerable attention from the speech community. WFST is a ﬁnite state network that encodes mappings between input and output symbol sequences (e.g., the mapping from an acoustic signal to a word sequence), and each mapping can be weighted with a non-negative value. One of the main advantages of this approach compared with traditional systems is that it is an elegant, uniform, and ﬂexible way of integrating various knowledge sources into a single network [5]. Hence, optimizing the overall system is possible by adapting the parameters of a single WFST network instead of adapting the parameters of each knowledge source separately. In this study we focus on robust speech recognition in car environments. We combine signal manipulation and model optimization techniques to increase the speech recognition performance for the individual drivers in the presence of engine/street noise. To improve the robustness of the speech recognition system, we performed nonlinear spectral subtraction (SS) to suppress noise from the noisy speech. Using “clean” speech signal obtained from SS, we conducted the supervised WFST network adaptation to the characteristics of a given driver. The evaluation was done with the “Driver’s Japanese Speech Corpus in Car Environment” [6], which was recorded as a part of the METI project, a project supported by the Japanese Ministry of Economy, Trade and Industry. The results of the initial experiments are promising. In the best case, for highly noisy conditions (signal to noise ratio (SNR) ranging from 0 to −8dB), the speaker dependent WFST decoder achieved 70% improvement when compared with traditional speaker independent speech recognition systems.

2 2.1

Background Spectral Subtraction

In this study, we use two common assumptions in spectral subtraction: clean speech is corrupted by additive noise and both signals are uncorrelated. At each frame i, the clean speech power is estimated by subtracting the estimated noise power from the noisy speech power in the frequency domain, i.e.,

26

A. Betkowska Cavalcante, K. Shinoda, and S. Furui

⎧ ˆi (ω)|2 , ⎨ |Yi (ω)|2 − αi |N 2 ˆ i (ω)| = ˆi (ω)|2 |X if |Yi (ω)|2 > (αi + β)|N ⎩ ˆ 2 β|Ni (ω)| , otherwise,

(1)

ˆ i (ω), Yi (ω), N ˆi (ω), and β are the angular frequency, the estimate of where ω, X clean speech, the noisy speech, the estimate of the noise, and a spectral ﬂooring factor, respectively. The parameter αi , which has to be deﬁned a priori, is a constant called the overestimation factor which depends on the signal to noise ratio (SNR) [1]. The noise signal is unknown, so it has to be estimated by taking averages of frames that are known to be silence [1]. The noise estimation is valid as long as the noise is stationary or its characteristics change relatively slowly compared to those of the speech signal. 2.2

Weighted Finite State Transducers in Speech Recognition

WFST is a ﬁnite state network that encodes mappings between input and output symbol sequences (for example, the mapping from an acoustic signal to a word sequence), and each mapping can be weighted with a non-negative value. WFST algorithms include, among others, composition (combination of transducers), determinization, and minimization operations. Further details on the mathematical representation and algorithms can be found in [5]. The application of WFST to a speech recognition system requires that each knowledge source is represented by a weighted transducer. These transducers are combined using the composition operation (◦) into a single WFST network: R = H ◦ C ◦ L ◦ G,

(2)

where H, C, L and G are WFSTs representing the acoustic model (HMMs), the phoneme context-dependency, the dictionary and the language model, respectively. The network R can be optimized by performing determinization and minimization operations that lead to considerable improvement of the decoder’s search eﬃciency [5]. 2.3

Adaptation of the Acoustic Model HMM

A common method for WFST network adaptation consists of two steps. First, adaptation processes of the acoustic and language models are performed separately. Then the adapted models are combined into a single WFST network. In this section we focus on the user adaptation of the acoustic model represented by HMMs. HMM is a set of N states, each of which can emit an output observation x with a given output probability density function (pdf). For each frame, let x = [x1 , x2 , . . . , xD ]T be the D-dimensional Mel Frequency Cepstral Coeﬃcients (MFCC) vector for the speech signal. Than the output pdf pj (x) for state j (j = 1, . . . , N ) in the HMM is usually represented by a mixture of Gaussians [2]: pj (x) =

M m=1

cjm N (x|μjm , Σjm ),

(3)

Robust Speech Recognition in the Car Environment

27

Fig. 1. Tree structure for shifts estimation

where M is the number of Gaussians in each state, μjm is the mean vector of the m-th mixture components of the state j, and cjm is the m-th mixture coeﬃcient, respectively. We assume that covariance matrix Σjm of the m-th mixture in state j is diagonal. The transitions among states are governed by the stochastic state transition matrix A = {ajk }, where each element ajk represents the probability of transition from state j to state k. For an acoustic model deﬁned in this way, we use an adaptation method proposed by Shinoda et al [7]. In this method, the mean of each Gaussian component in the speaker-independent HMM (SI-HMM) is mapped to the unknown mean of the corresponding Gaussian component in the speaker-dependent HMM (HD-HMM). Let μi and μ ˆi be the mean of the i-th Gaussian component of the SI-HMM and the corresponding Gaussian component of the SD-HMM, respectively. Then, μ ˆi = μi + δi , i = 1, . . . , N × M, where δi is a shift parameter from the mean of the SI-HMM, N is the number of states in the model, and M is the number of Gaussian components in each state. The shift δi is estimated using a training algorithm such as the forward-backward algorithm or the Viterbi algorithm. The number of required shifts δi is very large (N × M ) in general, so the correct estimation of these shifts with a limited amount of adaptation data is often very diﬃcult. To overcome this problem, the proposed method controls the number of shifts to be estimated by using a tree structure of Gaussian components (see Figure 1). This tree is constructed by clustering the Gaussian mixtures of all the states of the SI-HMM with a top-down clustering method that employs

28

A. Betkowska Cavalcante, K. Shinoda, and S. Furui

the k-means algorithm. The symmetric Kullback-Leibler divergence is used as a measure of distance between two Gaussians. In such a tree, each leaf node i corresponds to a Gaussian mixture i, and a tied-shift Δj is deﬁned for each nonleaf node j. Using this tree structure, we control the number of free parameters according to the amount of data available. When we do not have a suﬃcient amount of data, a tied-shift Δj in the upper part of the tree is applied to all the Gaussian components below node j. As the amount of data increases, tied-shifts in the lower levels are chosen for adaptation. To control this process, we use a threshold that deﬁnes the minimum amount of data needed to estimate Δj . This threshold represents the number of data frames needed for the precise estimation of the shifts attached to each node and is chosen experimentally.

3 3.1

Experimental Conditions Drivers Japanese Speech Corpus in Car Environment

For the evaluation we used the “Driver’s Japanese Speech Corpus in Car Environment” recorded by the Asahi Kasei company [6]. The speech of each driver was recorded in the Toyota Vitz car with the microphone installed near the map lamp. For our study we used recordings of 110 nonprofessional female drivers, 110 nonprofessional male drivers, 20 professional female drivers, and 20 professional male drivers. The recordings were taken in three diﬀerent conditions: with the car in an idle state, in a driving school (nonprofessional drivers) or in a city (professional and nonprofessional drivers), and on highways (professional drivers). The recordings taken on highways have low SNR, ranging from (0 to −8 dB). For each condition each driver uttered around 270−450 commands. The commands included navigation commands, hands-free commands, and digit sequences, among others. Each command consists of several words. The samples were digitized at 16 kHz sampling rate and analyzed with a frame of 10 msec. Mel Frequency Cepstral Coeﬃcients (MFCC) consisting of 12 static features, 12 Δ features, 12ΔΔ features, and Δ and ΔΔ energy were used as an input vector in each frame. 3.2

WFST Network Construction

Acoustic Model. We constructed a baseline speech left-to-right HMM with 3 states and 16 Gaussian components per state. The basic recognition unit for the acoustic models were triphones, trained with a Japanese speech corpus for large vocabulary continuous speech recognition (LVCSR) systems called JNAS [8] (Japanese Newspaper Article Sentence). The constructed acoustic model was then converted to a WFST using AT&T FSM tools [9]. Language Model. The language model was constructed as follows. First, a grammar deﬁned with extended Backus Naur Form (BNF) notation was parsed with the HTK HParse tool [10], and a word network in a HTK standard lattice

Robust Speech Recognition in the Car Environment

(a) Female drivers

29

(b) Male drivers

Fig. 2. Baseline WFST system evaluation - non-professional drivers

format (SLF) was generated. Next, the word network was converted to the WFST format. Finally a language WFST model was compiled with the AT&T FSM library. 3.3

Evaluation of the Baseline Model

We built a separate baseline WFST recognition system for navigation, hands-free control, and digit sequence tasks. For each command group a separate language model was developed with a separate dictionary. The vocabulary list consisted of 286, 83, and 13 words for navigation commands, hands-free commands, and digit sequence task, respectively. The resulting WFST decoders shared the same acoustic model, but their language model and dictionary were diﬀerent. Each of these decoders was tested by professional and nonprofessional drivers’ utterances in three conditions: idle state, city/school, and highway. The results of the experiment are shown in Figure 2 and Figure 3. The system achieved the highest performance when the car was in an idle state. This result is expected because those test samples have high SNR. The recognition accuracy varied from 94.5%−98.6% for professional drivers and from 95.5% − 99.1% for nonprofessional drivers, respectively. The worst results were achieved for a digit sequence task. It can be explained by the fact that the grammar is more ﬂexible as compared to other tasks, so the recognition heavily relies on the accuracy of the acoustic model. The WFST decoder achieved slightly worse results in city conditions. Surprisingly, a signiﬁcant accuracy degradation can be seen for professional female drivers (more than 10% absolute). This may be due to the fact that the SNR for these professional female drivers was much lower than for the other speakers. The baseline WFST decoders failed in highway conditions, achieving recognition accuracy below 30% for professional female drivers and around 50% for

30

A. Betkowska Cavalcante, K. Shinoda, and S. Furui

(a) Female drivers

(b) Male drivers

Fig. 3. Baseline WFST system evaluation - professional drivers

professional male drivers. The low accuracy can be explained by a high level of noise due to the speed of the cars. 3.4

Evaluation of Nonlinear Spectral Subtraction

We applied nonlinear spectral subtraction for speech recorded in a city and on a highway. When the car was in an idle state, the noise distortion of the speech was small, so there was no need for performing this procedure. Proper estimation of the noise signal (based on silence frames) is essential for successful spectral subtraction. In the assumed scenario, the driver could speak anytime, so correct detection of silence and the speech part of the signal was crucial. To simplify the problem, the driver was requested to push a button before speaking. All frames of the signal recorded before the button was pushed were considered as silence. These frames were used for noise spectrum estimation. Spectral subtraction was applied to the signal taken after the button was pushed. The results of SS are shown in Figures 4, 5, and 6. A considerable improvement can be seen for both female and male professional drivers on highways. The best results were achieved for the hands-free commands: 24.4 and 22.3 percentage points of absolute gain in accuracy for female and male speakers, respectively. In the city, application of SS caused a slight degradation of the recognition performance of most of the drivers. Taking into account the high recognition performance of the baseline WFST decoder, we can assume that only a small amount of noise is corrupting the speech. Distortions caused by SS (so called musical noise) seem to give more problems for the recognition system than the noise originally corrupting the speech signal.

Robust Speech Recognition in the Car Environment

31

Fig. 4. Spectral subtraction - professional drivers, city

Fig. 5. Spectral subtraction - professional drivers, highway

3.5

Speaker Adaptation of Acoustic Model

We performed speaker adaptation for professional drivers driving on highways. Each speaker data was randomly divided in half to create the adaptation and test sets. For the speaker adaptation, we used Shinoda’s algorithm [7] described in Subsection 2.3. The tree of the Gaussian components had four levels and four branches at each level. The threshold, which deﬁnes the minimum frames needed for the tied-shift Δ estimation, was experimentally set to four. Before adaptation, spectral subtraction was applied to the adaptation and test sets. We created a speaker-dependent acoustic model for each driver by adapting a baseline (speaker-independent) acoustic model. The speaker-dependent acoustic models were combined later with the language model to create a single WFST network. We tested the adapted WFST decoder for each speaker and for each command task. The results are shown in Figure 7. Compared to the baseline WFST decoder, the speaker adaptation improved the WFST decoder’s performance in all three tasks. In case of the digits task

32

A. Betkowska Cavalcante, K. Shinoda, and S. Furui

Fig. 6. Spectral subtraction - nonprofessional drivers, city

Fig. 7. Adaptation of WFST network and SS - Professional drivers, highway

the improvement was 49.4 and 29.9 percentage points for professional female drivers and professional male drivers, respectively. In the hands-free task, the improvement was 70.6 and 46.6 percentage points. The best results, 93.5% and 94% recognition accuracy, were obtained for the navigation command task. The digit sequence task, as in the baseline WFST decoder, was the most diﬃcult to be recognized correctly (e.g. 78% of recognition accuracy for female drivers).

4

Conclusions

We investigated diﬀerent methods for improving the speech recognition performance in car environments. The proposed methods were evaluated with the

Robust Speech Recognition in the Car Environment

33

“Driver’s Japanese Speech Corpus in Car Environment” and compared against a baseline WFST speech recognition system. The baseline WFST system achieved high recognition accuracy (over 95%) for the samples recorded in a car in the idle state. Slightly worse performance was obtained for samples recorded in the city. A signiﬁcant degradation of the recognition performance was observed in highways. However, applying nonlinear spectral subtraction (SS) in highway environment improved the recognition performance of the WFST network by 24.4 and 22.3 percentage points for professional female and male speakers, respectively. We also evaluated the WFST system optimized in highway conditions. The combined optimization method included spectral subtraction and speaker adaptation. In the best case, for professional female drivers with hands-free command task, the performance of the WFST network was increased by 70.6 percentage points. In this study, the WFST adaptation is done in two steps. First, the acoustic model is adapted to the characteristics of the speciﬁc driver. Then this model is combined with language model into a single WFST network. Unfortunately, this scheme for adaptation does not take into consideration the relation between acoustic and language models, and it can only be performed oﬄine. In the future, the supervised and unsupervised adaptation of the whole WFST network should be investigated and compared with traditional optimization methods.

Acknowledgments This works was supported by the Japanese government METI Project ”Development of Fundamental Speech Recognition Technology”.

References 1. Gong, Y.: Speech recognition in noisy environments: A survey. Speech Communication 16(3), 261–291 (1995) 2. Huang, X., Acero, A., Hon, H.: Spoken Language Processing: A Guide to Theory, Algorithm and System Development. Prentice Hall, USA (2001) 3. Rabiner, R.: A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE (2), 257–286 (1989) 4. Caseiro, D., Trancoso, I.: Using dynamic wfst composition for recognizing broadcast news. In: International Conference on Spoken Language Processing, pp. 1301–1304 (September 2002) 5. Mohri, M., Pereira, F.C.N., Riley, M.: Weighted finite-state transducers in speech recognition. Computer Speech and Language 16(1), 69–88 (2002) 6. Kato, T., Okamoto, J., Shozakai, M.: Analysis of Drivers’ Speech in a Car Environment. In: INTERSPEECH 2008, pp. 1634–1637 (September 2008) 7. Shinoda, K., Watanabe, T.: Speaker adaptation with autonomous control using tree structure. In: EuroSpeech 1995, pp. 1143–1146 (September 1995)

34

A. Betkowska Cavalcante, K. Shinoda, and S. Furui

8. JNAS: Japanese newspaper article sentences, http://www.mibel.cs.tsukuba.ac.jp/jnas/ 9. Mohri, M., Pereira, F.C.N., Riley, M.: A rational design for a weighted finite-state transducer library. In: Wood, D., Yu, S. (eds.) WIA 1997. LNCS, vol. 1436, pp. 144–158. Springer, Heidelberg (1998) 10. Young, S., et al.: The HTK Book: HTK Tools and Reference Manuals, Version 3.4. Cambridge University Press, UK (2006)

Corpus Design for a Unit Selection TtS System with Application to Bulgarian Aimilios Chalamandaris, Pirros Tsiakoulis, Spyros Raptis, and Sotiris Karabetsos Institute for Language and Speech Processing – Athena Research Centre, Artemidos 6 & Epidavrou, 15125, Athens, Greece {achalam,ptsiak,spy,sotoskar}@ilsp.gr

Abstract. In this paper we present the process of designing an efficient speech corpus for the first unit selection speech synthesis system for Bulgarian, along with some significant preliminary results regarding the quality of the resulted system. As the initial corpus is a crucial factor for the quality delivered by the Text-to-Speech system, special effort has been given in designing a complete and efficient corpus for use in a unit selection TTS system. The targeted domain of the TTS system and hence that of the corpus is the news reports, and although it is a restricted one, it is characterized by an unlimited vocabulary. The paper focuses on issues regarding the design of an optimal corpus for such a framework and the ideas on which our approach was based on. A novel multistage approach is presented, with special attention given to language and speaker dependent issues, as they affect the entire process. The paper concludes with the presentation of our results and the evaluation experiments, which provide clear evidence of the quality level achieved. Keywords: Text to speech, unit selection TTS, corpus design, concatenative speech synthesis.

1 Introduction Text-to-speech synthesis systems convert textual input into synthetic voice. In recent years, text-to-speech (TTS) systems have shown a significant improvement as far as the quality of the synthetic speech is concerned. This evolution has been sparked mainly by the fact that they rely more and more on data-driven, statistical modeling of the speech, with time or frequency domain algorithms for signal manipulation, such as in [1], [2], [3] and [4]. The majority of current approaches in the field aim to model the speech of a given human speaker, based on recordings of his or her voice, and deliver synthetic speech which manages to capture satisfactorily its original spectral and prosodic characteristics, such as voice timbre, pitch and durations. The key idea of unit selection speech synthesis [5], which currently constitutes the cornerstone of most of the state-of-the-art TTS systems worldwide, is to use a spoken corpus as the acoustic inventory and to select from within, at run-time, different acoustic units that match better according to a metric, so as to capture the characteristics of a targeted synthetic speech and at the same time deliver context-specific prosodic behavior. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 35–46, 2011. © Springer-Verlag Berlin Heidelberg 2011

36

A. Chalamandaris et al.

A more detailed description of a unit-selection TTS system is given in the following paragraphs. Although recent parametric approaches for speech synthesis have emerged with significantly good and promising results, such as HMM-based speech synthesis [6] that aims to model the features of one’s voice through Hidden Markov Models, the unit selection method currently provides a wide and more mature framework for speech synthesis, which has been advocated to best meet the needs of limited domains. The parametric approaches are certainly the future of synthetic speech; however the concatenative unit selection systems still dominate present solutions.

2 Unit Selection Text to Speech The general architecture of a corpus-based TTS system is depicted in Fig. 1. text

N atural Language P rocessing (N LP ) • Parsing and Morpho-syntactic analysis • Text Normalization • Letter to Sound • Prosodic analysis Speech (units) Database Digital Signal Processing (DSP ) • Prosodic Analysis • Speech Representation and Manipulation • Speech Generation

Voice

Fig. 1. General architectural diagram of a corpus-based TTS system

There are two main components most often identified in such a system, namely the Natural Language Processing unit (NLP) and the Digital Signal Processing unit (DSP). This schematic applies for every data driven (i.e. any corpus-based) TTS system, regardless of the underlying technology (e.g., unit selection or parametric) [7]. The NLP component accounts for every aspect of the linguistic processing of the input text, whereas the DSP component accounts for the speech signal manipulation and the output generation. For a unit selection TTS, besides the speech units (usually diphones, i.e. two consecutive semiphones including the transition phase between them) the speech database contains all the necessary data for the unit selection stage of the synthesis [5]. These components deserve more explanation. More specifically, the NLP component is mainly responsible for the parsing, the analysis and the transformation of the input text into an intermediate symbolic format, appropriate to feed the DSP component. Furthermore, it provides all the essential information regarding prosody, that is, pitch contour, phoneme durations and intensity. It is usually composed of a text parser, a morpho-syntactic analyzer, a text

Corpus Design for a Unit Selection TtS System with Application to Bulgarian

37

normalizer, a letter-to-sound module [7] and a prosody generator. All these components are essential for disambiguating and expanding abbreviations and acronyms, for producing correct pronunciation, and also for identifying prosody related anchor points. The DSP component includes all the essential modules for the proper manipulation of the speech signal, that is, prosodic analysis and modification, speech signal processing and generation. Among various algorithms for speech manipulation, Time Domain Pitch Synchronous Overlap Add (TD-PSOLA), Harmonic plus Noise (HNM), Linear Prediction based (LPC-based) and Multiband Resynthesis Overlap Add (MBROLA) are the techniques that are mostly employed. Aside from the aforementioned modules, the DSP component includes also the unit selection module, which performs the selection of the speech units from the speech database using explicit matching criteria [7]. More details about the unit selection module are provided in the following paragraph. 2.1 Unit Selection Module The unit selection module provides a mechanism to automatically select the optimal sequence of database units that produce the final speech output, the quality of which depends on its efficiency. The optimization criterion is the minimization of a total cost function which is defined by two partial cost functions, namely the target cost and the concatenation cost functions [5]. The target cost function measures the similarity of an applicant unit with its predicted specifications (from NLP) and is defined as, p

C t (t i , u i ) = ∑ w tj ⋅ C tj (t i , u i )

(1)

j =1

where, u1n = {u1, u2, …, un} are the candidate (sequence) units, t1n = {t1, t2, …, tn} are the target (sequence) units, C tj (t i , ui ) is a partial target cost, p is the dimension of the target feature vector and

w tj is a weighting factor for every partial target cost. The

target feature vector typically employs target values for prosody and contextual features. The concatenation, or otherwise join, cost function accounts for the acoustic matching between pairs of candidate units and is defined as, q

C c (u i −1 , u i ) = ∑ w cj ⋅ C cj (ui −1 , u i )

(2)

j =1

where, C cj (ui −1 , ui ) is a partial join cost , q is the dimension of the join feature vector and w cj is a weighting factor for every partial join cost. The feature vector typically includes similarity measurements for the spectral, pitch and contextual dimensions. Hence, the total cost is defined as, n

n

i =1

i =2

C (t1n , u1n ) = ∑W t ⋅ C t (ti , ui ) + ∑W c ⋅ C c (ui −1 , ui )

(3)

38

A. Chalamandaris et al.

or based on (1) and (2) it can be written as, n

p

n

q

i =1

j =1

i=2

j =1

C (t1n , u1n ) = ∑W t ⋅ ∑ wtj ⋅ C tj (ti , ui ) + ∑ W c ⋅ ∑ wcj ⋅ C cj (ui −1 , ui )

(4)

where, W t and W c are the weights that denote the significance of the target and the join costs, respectively. The goal of the unit selection module is to perform a search, so as to find the speech unit sequence which minimizes the total cost, hence to specify, n u1 = min C (t1n , u1n ) u1 ...un

(5)

The selection of the optimal speech unit sequence incorporates a thorough search (usually a Viterbi search) which involves comparisons and calculations of similarity measures between all available units, often employing heuristics to guide and/or limit the search [5] for higher efficiency. 2.2 Spoken Corpus for TTS It is a common knowledge that the quality of the synthetic speech provided by a corpus based TTS system highly depends on the quality of its acoustic unit inventory. The most important factor affecting the quality of this inventory is the initial text corpus, namely the set of sentences that the speaker will have to utter and record, in order to create the initial data for the acoustic inventory. There have been several publications on how to create a corpus for a TTS system such as in [8], [9] and [10]. Most of them use greedy algorithms in order to select a sub-corpus from an initial large corpus pool, which would fulfill satisfactorily several different requirements. These requirements can be as simple as word coverage [11], or more complex techniques such as phonetic or prosodic coverage, or the coverage of other language-dependent or domain-dependent parameters [12]. Our methodology attempts to deal with different types of parameters in order to design an efficient spoken corpus for a relatively general-domain TTS system for Bulgarian. One should note here that although the targeted domain, namely the news reports domain is restricted, it can also be regarded as a relatively generic one due to the wide range of topics it covers and to the generic vocabulary it is defined by. The main objectives of our approach were two: (a) to achieve sufficient coverage of the significant language-dependant phenomena identified and (b) to ensure consistently good performance during synthesis. More specifically, the first objective can be projected to the following set of complementary goals: (i) phonetic coverage, (ii) prosodic coverage, and (iii) controlled redundancy, while the second objective aims to compensate for problems that are relevant either to the speaker’s voice characteristics or the actual recordings and which cause inconsistent performance for the TTS system.

3 Corpus Design Strategies In a unit selection TTS system, the corpus design problem can be regarded as a set coverage problem. The target set C is the set of units to be covered. Each sentence in

Corpus Design for a Unit Selection TtS System with Application to Bulgarian

39

the corpus is also a set of units, and the corpus selection problem consists in finding a minimum-size set of sentences which will contain all the units defined in the set C. According to the domain specifics and needs of the application field, the nature and number of units can vary from simply a few words, to many thousands of phonemes, diphones or longer units. In the extreme case of very limited domain [13], it has been shown that if a database is deliberately tailored to the intended application, the TTS system can provide robustly high-quality synthetic speech. In such simple cases, the corpus design process is limited in selecting sentences that include at least one occurrence of each word in the domain, in each desired prosodic context [10]. Manual selection or compilation of the sentences can often be adequate for such limited domains. Different units allow for different strategies for corpus design. In the case of very limited domains, such as weather reporting, the units can be words or even phrases that can be reproduced during text-to-speech synthesis. Such approaches have been very popular for such domains [14], and it has been shown that they provide high quality synthetic speech. However, in the case of a more generic domain, or of a domain with unlimited vocabulary, even though the domain is practically restricted, the units cannot be as long as words, since it is impossible to achieve efficient coverage [15]. Many researches use diphones as basic units for the unit selection and therefore they also use diphones in order to handle the corpus design problem. Others identify severe drawbacks when employing diphones for corpus design based on the theory of Large Number of Rare Events (LNRE) [16] and they suggest either modifying the searching algorithm using diphones, or using triphones as basic units [8]. In our approach the basic selection unit is the diphone, for each of which we employ a feature vector that sufficiently describes its contextual properties, as far as our TTS system is concerned. More details about the employed features are provided in section 4. 3.1 Utterance Selection Methods The corpus design problem as defined in previous paragraphs can be regarded as a process for deriving a minimum-size set of sentences which offer coverage for a target-units set S. The target unit set C incorporates all target units necessary for the TTS system to deliver high quality synthetic speech. The common practice for designing such a corpus automatically is by employing a greedy algorithm [17]. The latter is an iterative technique for compiling a subset of sentences from a large set of sentences (corpus pool) in order to cover the largest unit space C with the smallest number of sentences. Prior to the selection, the corpus pool as well as the target set C must be well defined. Normally, the initial corpus pool is a set of sentences that well define the text style of the targeted domain of the target TTS application. That is, if for example one would aim to create a corpus for sports reports, then the initial corpus pool should contain mainly texts from sports news and reports. As far as the target set C is concerned, it often consists of units that best describe the phonetic content of the targeted domain [8] and [7]. Rational extension to this idea is the inclusion of other important parameters such as prosodic, stress and contextual information of the phonemes to be covered [10] and [12].

40

A. Chalamandaris et al.

The greedy selection algorithm involves the assignment of costs to every sentence of the corpus pool, according to the number of units that are in common with the target set C, and the number of units that are not in their intersection. At each iteration the algorithm selects the sentence with the highest ranking according to the previous criterion, it removes it from the corpus pool, and it updates the target set C by removing the units that have been covered by this sentence. This process continues until a termination criterion is reached, such as maximum number of sentences or efficient coverage of the target set C. The main drawback of this technique is that if the number of factors defining a unit is large, that is, if the target set C is significantly large, then the produced corpus may be prohibitively large. Modified greedy algorithms which mainly aim to indirectly cluster the factors defining the units of the target set C have also been suggested and they have shown to work efficiently [7]. The main aspects, on which most strategies employing the greedy algorithm differentiate from each other, are the statistical properties of the coverage set. Often it is suggested that a phonetic distribution similar to the one of the corpus pool should be aimed in order to better capture the acoustic properties of the initial large corpus, while in other cases, priority is given on the rarest unit classes in order to compensate for LNRE phenomena [8] and [18].

4 The Proposed Corpus Selection Method It has been clear from our experiments and from other researches that the process of corpus design for a unit selection TTS is not trivial and it should be given special attention and effort. It has also been clear that the unit selection TTS systems suffer from limitations as far as the signal modification is concerned, and therefore their final quality is inherently dependent on the abundance and completeness of their database [19]. Nevertheless, it is also obvious that the development of huge databases, aside from being a time and effort consuming process, does not necessarily guarantee proportionally to the size good results. Our corpus design method aims to deliver an efficient corpus for a restricted but unlimited domain, with a special providence to identify and alleviate problems that could cause inconsistency in the final quality. The main idea behind our algorithm is to define a process that will take advantage of available information about the targeted TTS system, and via a post-hoc process, it will be able to act complementarily to the unit selection algorithm’s properties. By doing so, we cater for both the specifics of the domain we are targeting, as well as for the optimization of its behavior in combination with our specific unit selection algorithm. As shown in the following section, our results justify our approach and the underlying hypothesis. The algorithm works in three stages: 1.

From a domain specific, extensively large corpus pool, we identify a set of sentences S which offers the maximum possible coverage of our unit target set C (as described above). 2. After the recording of the S set and the incorporation of it in the unit selection TTS system, we simulate the synthesis of the original large corpus pool, in order to identify possible concatenation problems that are either speaker or sentence specific. By doing so, we automatically select an additional subset

Corpus Design for a Unit Selection TtS System with Application to Bulgarian

41

of sentences S’ that present large number of problems in combination to our unit selection module. 3. In the final stage of our method we identify possible diphones that are missing from the initial large corpus pool and we manually insert them in short non-sense sentences compiling a new subset S’’. The final optimal corpus is then the union of the independently derived sets:

S Final = S + S '+ S ' '

(6)

The stages described above deserve more explanation. Stage 1: Selection of the S corpus set. Initially the large corpus set, serving as the corpus pool from which we automatically select the sentences, has to be collected and processed accordingly in order to be rendered appropriate for this task. In order to break down a large text into a set of sentences two tasks have to be carried out initially: (i) text normalization and (ii) sentence tokenization [2]. The text normalization is responsible for the expansion of numerals, abbreviations and acronyms, as well as dates, addresses etc. It is a rather complex task, based mostly on heuristics and hybrid algorithms, in order to achieve disambiguation when necessary. The sentence tokenization process, although seems to be simple enough, it highly depends upon other text pre-processing modules and its level of complexity is often language-dependent. In order to phonetically transcribe every sentence, we used the grapheme to phoneme module we developed for the Bulgarian language [20]. From the phonetically transcribed corpus pool, where the prosodic characteristics of every unit are also provided by our prosody engine, we greedily select a subset of sentences that offers satisfactory coverage for our target unit set C. In order to define C, we constructed a contextual feature vector for the diphone units that included key prosodic factors, such as word accent status, position in the utterance, distance from prosodic modifiers in the utterance etc. The prosodic factors employed here derive from our prosody engine that is incorporated into our TTS system, which is based on a data-driven prosody modeling approach. By using a greedy selection algorithm as described previously, we produce a set of sentences S which satisfactorily covers our defined target unit set. Stage 2: Enhancement and fine tuning of the S corpus set. This stage of our approach provides a novel method for enhancing the speech corpus. During this stage, after the recording and processing of the sentences and their compilation into a database, ready to be used by our TTS system, we aim to identify problems during actual synthesis, which originate either from speaker-dependent factors, or from other factors, such as misaligned or even bad recordings. This process consists in synthesizing every sentence of the large corpus pool and identifying local and global maxima of the total cost function during the unit selection. By synthesis, we identify additional units that should also be covered by the final corpus, and through an iterative selection algorithm we select a set of sentences S’ that optimally cover these units. Special care has been taken in order to maintain the recording conditions consistent throughout the entire process, especially because different recording sessions were carried out with long intervals between them.

42

A. Chalamandaris et al.

Stage 3: Further enhancement with missing units. The final stage of our selection strategy although may seem trivial, is crucial for the better performance of the TTS system in different domains, or in difficult vocabularies, such as foreign words. This step of the selection process, leading to S’’, practically aims to efficiently handle LNRE aspects as well, since whatever units are missing from the large corpus pool can be well characterized as rare events. These units, even though they are very rare, they can affect the overall quality of the TTS system by fusing inconsistencies and mismatches in the synthesized speech.

5 Results It was decided that the targeted domain would be the news reports, mainly for two reasons: (i) it is of the scope of the authors to develop a synthetic voice for news, and (ii) because this domain is both a restricted and unlimited one, and the usual neutral informative speaking style can also serve in other domains such as dialogues or speaking applications. In order to shape the large corpus pool from which we would extract the optimal corpus, we collected the online news articles from different Bulgarian newspapers for a period of 12 months. The initial large corpus pool consisted of about 54 million words. After having performed text normalization and sentence tokenization onto the corpus set, we ended up with an initial corpus pool of about 4.15 million sentences. Sentences with foreign words, or extremely long or short ones, were discarded. Although other approaches explicitly decide to select from within sentences of reasonably short length, such as of maximum 10-12 words [7], we believe that longer sentences, although they might be sometimes difficult to pronounce or process, they contain features that short sentences lack, such as more variable prosodic structure. For creating the target unit set C, we employed our letter to sound module for the Bulgarian language and the respective prosody engine on the entire corpus [20], and we formed a feature vector for each unit, containing the following parameters: (i) diphone type, (ii) acoustic context, (iii) prosodic cluster type and (iv) intra-prosodic relative position. We identified 204,514 unique units in the large corpus pool, which altogether define our target unit set C. A custom greedy selection algorithm with a termination criterion of 4,000 sentences, selected a subset of sentences (namely set S) from the large corpus pool, which offered not complete but sufficient coverage of the target unit set C. The selection algorithm was designed to pursue full coverage for all unique diphones and efficient unit coverage. One of the major problems in such cases, where a set of sentences is selected automatically with a dual criterion, to maximize the coverage and minimize the size, is the fact that often sentences with spelling errors or others are selected, since they provide coverage of rare acoustic events. In order to alleviate this problem, an additional mechanism has also been implemented that allowed us to review the rare instances in the resulted sentences, and observe the units for which each of them had been selected. During the manual correction of the sentences, if a correction would lead to the exclusion of a covered unit, the system would suggest automatically one or more other sentences that would compensate for all the units that the removed sentence had been selected for. This process is necessary in order to remove any errors

Corpus Design for a Unit Selection TtS System with Application to Bulgarian

43

that could affect the final results at several intermediate stages. After this phase of the process, the corpus set S consisted of 4,083 sentences with 16.13 words on average. The following two stages of the design process are carried out after the recordings and their processing for their incorporation into the TTS system. After the completion of the recordings and the corresponding database, we synthesized every sentence from the large corpus pool and we identified automatically the most problematic units, as far as the synthesis process is concerned. This was carried out automatically by identifying the local maxima in the unit selection total cost function for every synthesized sentence. By this process, after the clustering of the problematic units, we identified a set of units that were necessary to be additionally covered by the recordings. Again, through the means of the greedy selection algorithm and with a termination criterion in the number of the selected sentences, we automatically selected 1000 additional sentences, which were to be included in a following recording session, by the same speaker. It is worth noting here, that special attention was given to carefully profile the recording settings for every session in order to ensure no deviations in the recorded speech. The final stage of the selection process consisted of the research and manual enrichment of a sentence set with sentences containing possible missing diphones that can be met in the Bulgarian spoken language, even if they are only necessary for the pronunciation of foreign words with the Bulgarian phonetic alphabet. The additional sentences were non-sense sentences, of short length, and were produced manually by the concatenation of words which ensured the utterance of such diphones. This stage resulted in an additional set of 32 sentences. In the following table one can see the properties of the selected corpus during the different stages of the process. Table 1. The properties of the resulted corpus during the corpus design process Corpus Design Stage 1st Stage 2nd Stage 3rd Stage

# Sentences 4,083 5,083 5,115

Diphone Coverage 96% 96% 100%

Unit Coverage 61.6% 65.6% 66.1%

6 Experimental Evaluation In order to evaluate the Bulgarian speech synthesis system, a set of acoustic experiments was performed. The experiments targeted different dimensions of the quality by covering naturalness, intelligibility and speech flow. A final set of questions was used to capture the participants’ opinion regarding the appropriateness of the system in different application areas. Finally, the listeners were given the option to provide free-text feedback, should they wish to. The subjects were 30 native Bulgarian speakers, 10 of which had a background in linguistics or previous experience related to the subject and, for the purposes of these experiments, where considered as a distinct group. For every stimulus the subjects were asked to provide a rating from 1 to 5, with 5 meaning perfect and 1 bad. The results of the experiments are illustrated in the following tables. Since it is out of this paper’s scope to describe the evaluation process, a more detailed description of the experiments and the analysis of the respective results can be found in [20].

44

A. Chalamandaris et al. Table 2. The evaluation results with regard to naturalness

Naturalness Non-expert listeners “Expert” listeners Overall

3,53 0,96 3,46 1,00 3,67 0,87

MOS STD MOS STD MOS STD

Experiment 1 (sentence-level) Ease of Articulation listening 4,41 4,13 0,66 0,77 4,39 4,08 0,68 0,81 4,44 4,24 0,56 0,63

Table 3. The evaluation results with regard to speech flow

Quality Non-expert listeners “Expert” listeners Overall

MOS STD MOS STD MOS STD

3,57 0,76 3,54 0,84 3,62 0,55

Experiment 3 (paragraph-level) Ease of PleasantUnderstandlistening ness ability 3,69 3,67 3,75 0,83 0,86 0,70 3,64 3,53 3,72 0,87 0,84 0,75 3,78 3,96 3,80 0,75 0,83 0,59

Pronunciation 3,47 0,78 3,48 0,83 3,46 0,68

In order to investigate how well the resulted TTS system captures the specifics of the targeted domain, namely the news reports, we asked the subjects to rate the appropriateness of the system for different application areas, grading with 5 for perfect for the domain, and with 1 for inappropriate. The results are illustrated in the following table. The latter experiment provides a clear evident that the resulted TTS system captures efficiently the specifics of the targeted domain, providing at the same time enough adversity for coping with other similar domains. Table 4. Mean Opinion Score for rating the appropriateness of the resulted TTS according to the application area Application Domain News Portals Telecom Applications Accessibility Tools Audio Books

Appropriateness (1:poor 5:excellent) 4.13 3.97 4.50 3.76

Nevertheless, other domains, such as book reading, are more demanding areas with many aspects that could not be covered by the resulted speech database, such as expressivity. It is also worth noting here that the high grade in the area of accessibility tools is mainly attributed to the fact that the resulted system delivers a high-quality synthetic speech and meets efficiently the most important requirements of the specific domain: intelligibility, robustness, consistency and pleasantness.

Corpus Design for a Unit Selection TtS System with Application to Bulgarian

45

7 Discussion In this paper we presented a methodology for designing and automatically producing an optimal corpus in the framework of a TTS system, providing a test case for the specifics of the Bulgarian language. Our results depicted that the resulted TTS system, which incorporated the designed spoken corpus, performs significantly well, producing a high-quality, near-natural synthetic speech. In the future we aim to investigate the differentiation of the TTS behavior with smaller databases, which would cover different aspects of our methodology, in order to identify possible prioritization in the aforementioned criteria and the level at which they affect the overall TTS performance. More focus will be given in the case where the spoken corpus design will attempt to explicitly capture the systems characteristics and will provide an optimal spoken corpus for the specific TTS system. Acknowledgements. The work presented in this paper has been co-financed by the European Regional Development Fund and by Greek national funds in the context of the INTERREG IIIA / PHARE CBC Programme 2000-2006 (an inter-regional cooperation program between Greece and Bulgaria). The authors would like to thank Prof. Elena Paskaleva, Ms. Irina Strikova and Ms. Aglika Ilieva Kroushovenska for their valuable help during the experiments, as well as the participants of the evaluation group for their useful feedback.

References 1. Moulines, E., Charpentier, F.: Pitch-synchronous waveform processing techniques for textto-speech synthesis using diphones. Speech Communication 9, 453–467 (1990) 2. Schroeter, J.: Basic Principles of Speech Synthesis. In: Benesty, J., Sondhi, M.M., Huang, Y. (eds.) Springer Handbook of Speech Processing. Springer, Heidelberg (2008) 3. Nagy, A., Pesti, P., Nemeth, G., Bihm, T.: Design issues of a corpus-based speech synthesizer. Hungarian J. Commun. 6, 18–24 (2005) 4. Mobius, B.: Corpus-Based Speech Synthesis: Methods and Challenges. Arbeitspapiere des Instituts fur Maschinelle Sprachverarbeitung (Univ. Stuttgart), AIMS 6(4), 87–116 (2000) 5. Hunt, A., Black, A.: Unit Selection in a Concatenative Speech Synthesis System Using a Large Speech Database. In: Proc. Intl. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), Atlanta, pp. 373–376 (1996) 6. Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T., Kitamura, T.: Speech parameter generation algorithms for HMM-based speech synthesis. In: Proc. of ICASSP, pp.1315– 1318 (June 2000) 7. Dutoit, T.: Corpus-based Speech Synthesis. In: Benesty, J., Sondhi, M.M., Huang, Y. (eds.) Springer Handbook of Speech Processing, Part D, ch. 21, pp. 437–455. Springer, Heidelberg (2008) 8. Black, A., Lenzo, K.: Optimal Utterance Selection for Unit Selection Speech Synthesis Databases. International Journal of Speech Technology 6(4), 357–363 (2003) 9. Bozkurt, B., Ozturk, O., Dutoit, T.: Text Design for TTS Speech Corpus Building Using a Modified Greedy Selection. In: Eurospeech 2003, pp. 277–280 (2003) 10. Matousek, J., Psutka, J., Kruta, J.: Design of Speech Corpus for Text-to-Speech Synthesis. In: Eurospeech 2001, Alborg (2001)

46

A. Chalamandaris et al.

11. Lewis, E., Tatham, M.: Word and Syllable Concatenation in Text-to-Speech Synthesis. In: Eurospeech 2001, vol. 2, pp. 615–618 (1999) 12. Lambert, T.: Automatic construction of a prosodically rich text corpus for speech synthesis systems. In: SP 2006, paper 200 (2006) 13. Yi, J.R.W., Glass, J.R.: Natural-Sounding Speech Synthesis using Variable-Length Units. In: Proc. ICSLP 1998, Sydney Australia, vol. 4, pp. 1167–1170 (1998) 14. Kishore, S.P., Black, A.: Unit Size in Unit Selection Speech Synthesis. In: Eurospeech 2003, pp. 1317–1320 (2003) 15. Schweitzer, A., Braunschweiler, N., Klankert, T., Mobius, B., Sauberlich, B.: Restricted Unlimited Domain Synthesis. In: Eurospeech 2003, pp. 1321–1324 (2003) 16. Mobius, B.: Rare events and closed domains: Two delicate concepts in speech synthesis. International Journal of Speech Technology 6(1), 57–71 (2003) 17. Franois, H., Boffard, O.: The Greedy Algorithm and its Application to the Construction of a Continuous Speech Database. In: 3rd International Conference on Language Resources and Evaluation (LREC 2002), vol. 5, pp. 1420–1426 (2002) 18. Andersen, O., Hoequist, C.: Keeping Rare Events Rare. In: Eurospeech 2003, vol. 2, pp. 1337–1340 (2003) 19. Balestri, M., Pacchiotti, A., Quazza, S., Salza, P., Sandri, S.: Choose the best to modify the least: a new generation concatenative synthesis system. In: Proceedings of the European Conference on Speech Communication and Technology, Budapest, Hungary, vol. 5, pp. 2291–2294 (1999) 20. Raptis, S., Tsiakoulis, P., Chalamandaris, A., Karabetsos, S.: High Quality Unit-Selection Speech Synthesis for Bulgarian. In: Proceedings of SPECOM 2009, St. Petersburg, Russia, June 21-25, pp. 388–393 (2009)

Automatic Identification of Phonetic Similarity Based on Underspecification, Mark Kane1 , Julie Mauclair2 , and Julie Carson-Berndsen1 1

School of Computer Science and Informatics, University College Dublin, Ireland [email protected], [email protected] 2 LIPADE, University Paris Descartes, France [email protected]

Abstract. This paper presents a novel approach to the identification of phonetic similarity using properties observed during the speech recognition process. Experiments are presented whereby specific phones are removed during the training phase of a statistical speech recognition system so that the behaviour of the system can be analysed to see which alternative phone is selected. The domain of the analysis is restricted to specific contexts and the alternatively recognised (or substituted) phones are analysed with respect to a number of factors namely, the common phonetic properties, the phonetic neighbourhood and the frequency of occurrence with respect to a particular corpus. The results indicate that a measure of phonetic similarity based on alternatively recognised observed properties can be predicted based on a combination of these factors and as such can serve as an important additional source of information for the purposes of modelling pronunciation variation. Keywords: speech recognition, phonetic similarity.

1

Introduction

A key challenge in speech recognition is to construct acoustic models which correctly estimate a sub-word unit or phonetic class label within a speciﬁc time interval. The smallest posited linguistically distinctive unit that is typically modelled is the phoneme. However, phonemes that belong to the same acousticarticulatory group (i.e. have similar acoustic or articulatory properties) are easily confused and thus statistical context-dependent phone models are often used as

This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at University College Dublin and Dublin City University. The opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Science Foundation Ireland. The research presented in this paper is a revised and extended version of the paper presented at LTC 2009, Pozna´ n Poland.

Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 47–58, 2011. c Springer-Verlag Berlin Heidelberg 2011

48

M. Kane, J. Mauclair, and J. Carson-Berndsen

a basis for deciding which phoneme seems to be the best according to an acoustic probability and the probability of its occurrence with respect to a language model. Confusability of phonemes and the relationship to underlying phonetic properties of speech sounds remains an important area of research in order to address variability in the domain of speech recognition. The research presented in this paper is motivated by research into patterns which emerge from mis-recognition of phonemes during the speech recognition process. The next step is to employ these mis-recognitions as a useful and additional source of information for modelling pronunciation variation. Experiments presented in [1], demonstrated that almost 80% of all misrecognised phonemes belong to the same phonetic group as the correct phoneme. In [2] this information is used in order to build Broad Phonetic Groups (BPG) that are deﬁned according to a confusability matrix and a new phoneme classiﬁer is proposed consisting of modular arrangements of experts, with one expert assigned to each BPG and focused on discriminating between phonemes within that BPG. The result in PER achieved by that system on the TIMIT corpus [3] is 26.4%. More recently these phonetic and phonological features are used to exploit similarities in phoneme recognition [4] with encouraging results. Neural Networks have also been used to classify between phonetic groups [5] and [6]. In [5] Neural Networks are used to improve the Discrete Wavelet Transform-based phonetic classiﬁcation algorithm. This paper focuses on the identiﬁcation of phonetic similarity using properties observed during the speech recognition process. This approach also uses the notion of mis-recognitions but in a very diﬀerent way from that outlined above. Rather than construct a confusability matrix for the recognised output, the output of two statistical speech recognition systems are compared, one where all phonemes to be recognised are included in the training data and one where individual phonemes are systematically removed from the training data (note that all phones occur in the testing data). This allows the identiﬁcation of the substituted choice where a particular phoneme is not available as the number of substitutions will increase dramatically in the testing phase for the removed phoneme as there is no ASR (hidden Markov) model built to represent that phone from the training data. The domain of the analysis is restricted to speciﬁc contexts and thus the term phone will be used in the remainder of the paper. The substituted phones thus identiﬁed are then analysed with respect to their phonetic properties as given by a phonetic feature classiﬁcation based on the IPA chart [7]. These properties provide a principled way to investigate phonetic similarity, underpinned by insights from experimental phonetics and phonological theory. The method described in this paper presents experiments on American English read speech and German spontaneous speech. Section 2 presents the speech recognition system used in the experiment and section 3 details the experimentation carried out to identify the substituted phones. The results and analysis of the experiment are discussed in section 4 and 5. Some conclusions are drawn and directions for future work are highlighted in section 6.

Automatic Identification of Phonetic Similarity Based on Underspecification

2

49

Speech Recognition System and Corpus

The HMM based speech recognition system used in this experiment is implemented with HTK [8]. The TIMIT [3] and Kiel [9] speech corpora are used for training and testing of the HMM models. For completeness, the more technical details of the speech recognition system and the corpus are presented in the next section in the traditional way. The TIMIT corpus consists of read speech spoken by 630 speakers of American English. The data is split into two sets; training and complete test set. The training set consists of 3696 utterances while the test consists of 1344 utterances. The SA data is not used in this paper. The Kiel corpus consists of spontaneous speech spoken by 26 speakers of German. The data is split into two sets; training and test set. The training set consists of 383 utterances while the test set consists of 142 utterances. There is no overlap between any of the training and test sets used in this paper. All plosives phones represented by a separate closure and associated burst are merged into a single phone. The chosen form of parameterisation of a phone within an utterance is mel frequency cepstral coeﬃcients (MFCCs), with their associated log energy and ﬁrst and second order regression coeﬃcients. Therefore every frame is represented by 39 coeﬃcients. First, each speech waveform is passed through a pre-emphasis ﬁlter. The waveform is then framed at a rate of 10 ms with a frame size of 25 ms where each frame is then windowed using a Hamming window function. These MFCCs representing the phones are then used in the calculation of the HMM models. The HMMs are context-dependent triphone models that were initially calculated by cloning and re-estimating context-independent monophone models. The triphones states were tied using phonetic feature decision trees for clustering. Each model is comprised of 5 states where only the centre 3 states are emitting. The decoding process is implemented with a uni-gram language model to emphasis acoustic variability thus negating the eﬀects of a typical bi/tri-gram. Initially experimentation began with the evaluation of the decoding process using a trigram language model of the phones. However it was found that the substituting choice remained closer to the removed phones phonetic neighbourhood if a unigram language model was used. This process is executed without any pruning. Finally, the number of components in each mixture is set to 4 as this was found to be the optimal number for the Kiel corpus. The experiment detailed in the next section distinguishes between a baseline and an underspeciﬁed system.

3

Experiment

In this paper, experiments based on an underspeciﬁed system for speciﬁc phones are presented. These experiments involve the identiﬁcation of the substituted alternatives when one phone is removed from the training set. The phonetic make-up of each of the phones were classiﬁed with respect to features from the IPA chart.

50

M. Kane, J. Mauclair, and J. Carson-Berndsen

As mentioned in the previous section, two types of speech recognition systems are evaluated: baseline and underspeciﬁed as shown in Figure 1. The baseline system is trained on a speciﬁc corpus training set. The underspeciﬁed system is trained on the same corpus training set but with all instances of a particular phone removed. Both systems are evaluated on the same test set.

Fig. 1. Overview of baseline and underspecified systems

From the baseline system all instances of the correctly recognised phones and all substitutions for the phones are identiﬁed. A certain level of control is required in these experiments and thus substitutions in which the left and right contexts of the substituted phone are correctly recognised are considered. This ensured that preceding errors had a limited aﬀect on results. It is clear that for an underspeciﬁed system for a particular phone, there can be no correctly recognised instances of that phone. The substitution phones act as possible alternatives and serve as a comparator. In this underspeciﬁed system where a phone is removed, it is anticipated that the substituted alternatives should be determined with respect to common phonetic properties, phonetic neighbourhood and perhaps with respect to frequency of occurrence in the full corpus, as this is an indicator of the relative frequency of the sound in the language (in particular the domain of the corpus). Each of these factors are underpinned by experimental phonetics and phonological theory (feature theory, markedness and underspeciﬁcation, in particular [10]) and, in combination, should serve as indicators for phonetic similarity. The method is presented using the American English read speech corpus (TIMIT) and then the commonalities are outlined with the German spontaneous speech corpus (Kiel). Results have been calculated for all of the phones, but ﬁrst the analysis of the phones [f] and [th] from the American English corpus are presented in sections 4 and 5.

Automatic Identification of Phonetic Similarity Based on Underspecification

4

51

Results

Based on the experiments outlined above, the results are presented as follows. Firstly, the results for the recognition of the phones [f] and [th] with the baseline system are presented. Secondly the results for the underspeciﬁed system for the phones [f] and [th] are presented and analysed in the context of phonetic similarity. Finally, the common phones between the American English and German corpora and their substitutions from an underspeciﬁed system are identiﬁed and discussed. 4.1

General Information

The baseline system serves to highlight which substitutions are found for each of the phones when they are not correctly recognised, given a correct left and right context (LR context). Table 1 shows the general ASR statistics for these phones while Table 2 shows the top ﬁve substituted alternatives of these phones. The LR context quantity (a subset of the substitution quantity) is the number of the phones that were recognised incorrectly and were replaced by another phone, where both the phone to the left and right of the substituted phone were correctly recognised. In this table the following are also highlighted; the number of times a phone occurred, how many times it was recognised correctly, how many times it was not recognised correctly and was substituted, how many times it was not recognised but deleted altogether and how many times it was inserted. The sum of the recognised, substituted and deleted is equal to the occurrence of that phone within the test set. Tables 1 and 2 also contain the corresponding information for the underspeciﬁed system. Table 1. General information of the baseline and underspecified (removed) system when evaluated against the test set Baseline Underspecified Qty f th f th occurrence 911 259 911 259 recognised 804 (88.3%) 88 (34%) 0 (0%) 0 (0%) substituted 94 (10.3%) 155 (59.8%) 708 (77.7%) 219 (84.6%) deleted 13 (1.4%) 16 (6.2%) 203 (22.3%) 40 (15.4%) inserted 58 10 0 0 LR context 23 42 206 60

As to be expected, it can be seen from Table 1 that the number of substitutions and LR contexts of a phone is greater in the underspeciﬁed system. This gives a broader range of similar phones as they are generated from a larger portion of data. This extra data allows a more complete pattern of substitutions to be observed.

52

M. Kane, J. Mauclair, and J. Carson-Berndsen

Table 2. Top 5 substituted alternatives and quantities from the baseline and underspecified (removed) system with respect to LR context Substitutions (LR context) Reference Baseline Underspecified f th(6), p(4), t(3), s(2), d(2) th(63), v(24), p(21), t(21), s(19) th f(17), t(9), dh(6), s(5), v(1) f(18), t(11), dh(9), z(5), s(5)

5

Analysis of Underspecified System

In this section the underlying theories are ﬁrst presented with respect to particular examples. Therefore in order to illustrate the type of analyses undertaken, an underspeciﬁed system [f] and an underspeciﬁed system [th] from the American English corpus are analysed. This method of analysis was also performed on the full American English and German Corpora. 5.1

Phone [f ]

As can be seen from Table 1, the baseline system, [f] was substituted 94 times and [th] was substituted 155 times. The number of [f] substitutions that have a correct LR context is 23 and the number of [th] substitutions that have a correct LR context is 42. Due to the fact that in the underspeciﬁed system [f] and [th] are not correctly recognised, the amount of substitutions increased. In this system [f] was substituted 708 times and [th] was substituted 219 times. The number of [f] substitutions that have a correct LR context is 206 and the number of [th] substitutions that have a correct LR context is 60. This extra data helps disambiguate the substitution data as there is more of it to help ascertain a pattern. In this section, the following three factors are considered: 1. common phonetic properties, 2. phonetic neighbourhood and 3. frequency of occurrence. 5.2

Common Phonetic Properties of [f ] and [th]

Phones can also be characterised in terms of the canonical properties or features they possess. Indeed this is typically one of the metrics employed to measure phonetic similarity. While the properties clearly relate to the notion of phonetic neighbourhood, depending on the feature set used, they may oﬀer additional granularity which allows features to be grouped as natural classes. Table 3 provides the subset of relevant features for the phones of Table 2, where + means present and - means not present. Note that other subset groupings of these sounds may also be relevant for further experimentation. 5.3

Phonetic Neighbourhood

The notion of phonetic neighbourhood can be visualised as in ﬁgure 2 where the four planes of the cube indicated in the ﬁgure represent the fricative manner

Automatic Identification of Phonetic Similarity Based on Underspecification

53

Table 3. Subset of features describing phonetic properties of substituted phones for [f] and [th] Phone p f v th dh s t d

fric plosive bi-labial labio-dental dental alveolar voiced + + + + + + + + + + + + + + + + + + +

of articulation, the plosive manner of articulation, the voiced and the unvoiced articulations, where the horizontal dimension indicates the place of articulation within these planes. For the initial experiments, relative positions of phones on this cube were used as a basis for the analysis. A more detailed explication of the role of phonetic neighbourhood may be possible, based on a numerical distance measure and a diﬀerentiation between the relative weightings of the dimensions within the cube. For example, it may be that remaining on the place of articulation axis represents closer proximity than moving to the voicing axis for some occasions; it would certainly appear reasonable that a change in the manner of articulation should involve a greater phonetic distance in comparison to a change in voicing information. This is a topic for future work, however. In the following subsections, the phonetic neighbourhoods of the phones [f] and [th] are discussed in more detail in the context of the substituted phones in Table 1 and 2. Phonetic Neighbourhood of [f ]. The position of phone [f] should be regarded as the starting point in the cube of ﬁgure 2. The consonants which appear as the substitutions for [f] in Table 2 can be seen within the cube and their relationship is represented with respect to the four planes indicated in the ﬁgure and are as follows: unvoiced fricatives([th], [s]), voiced fricatives([v]) and unvoiced plosives([p], [t]). At ﬁrst glance, the LR context substitutes appear to be in rank order from Table 2. Cautiously, due to the small amount of data, the ranking for these substitutes may be partitioned into two groups: ﬁrst - [th]; second - [v], [p], [t] and [s]. The phone [th] is the obvious ﬁrst choice for substitution as its place of articulation on the fricative plane is closet to that of phone [f]. They also share the same manner of articulation and are both unvoiced. According to this depiction, moving one step in any dimension from [f] for example, phones [th], [v] and [p] would appear to have the closest phonetic neighbourhood to the phone [f]. The phonetic neighbourhood may be measured in terms of relative distance between points on the cube. Phonetic neighbourhood alone does not appear to be a complete determining factor (albeit close) for the substituted alternative for phone [f] as the phones [t] and [s] (where both phones are two places removed

54

M. Kane, J. Mauclair, and J. Carson-Berndsen

Fig. 2. Cube representation of phonetic neighbourhood

from the place of articulation of [f] and [t] has a diﬀerent manner of articulation) seem to have as much priority as [v] and [p]. Phonetic Neighbourhood of [th]. The notion of phonetic neighbourhood for [th] is also accounted for in ﬁgure 1. In this case, the position of phone [th] should be taken as the starting point in the cube. Similar to the ranking consideration that was applied to the substitutions of phone [f], the ranking for the [th] substitutes from Table 2 may also be partitioned into two groups: ﬁrst - [f]; second - [t], [dh], [z] and [s]. Again, the obvious ﬁrst choice for substitution is [f] as it is also partially dental. Going one step in either dimension from [th] yields [f], [dh], [s] and almost [t]. Noticeably [t] seems to have as much priority as [dh] and may be attributed to the frequency of occurrence of [t] which will now be explained. 5.4

Frequency of Occurrence

Another factor which is likely to be a determining factor for the substituted alternative is frequency of occurrence in the corpus as a whole. This factor relates to the notion of markedness in phonology which postulates that the most unmarked (or default) sound in a language is also likely to be the sound which is most common. The frequencies of occurrence of the phones in the full American English corpus (as described in section 2) together with their associated percentage of the corpus they represent are presented in Table 4. These frequency of occurrences can also be visualised with respect to the phonetic neighbourhood cube as depicted in ﬁgure 3, where the larger the circle, the greater the frequency of occurrence. As mentioned in section 5.3, [t] and [s] seem to have as much priority as [v] and [p] as substitutions for [f] and may

Automatic Identification of Phonetic Similarity Based on Underspecification

55

Table 4. Frequencies of phones of Table 2 from the combined training and test set of the American English corpus Phone Frequency of phone in Percentage of phone full TIMIT corpus in full corpus t 8578 5.16% s 8348 5.03% d 5918 3.56% p 4015 2.42% dh 3272 1.97% f 3126 1.88% v 2704 1.63% th 1004 0.60%

be attributed to their frequency of occurrence. A region of high frequency of occurrence within the cube has a pseudo gravitational eﬀect, the closer to this region, the greater the inﬂuence this region has on the substituting possibilities; creating a possible default substitution combined with unmarked sounds [10]. The common phonetic properties, the phonetic neighbourhood and the frequency of occurrence in the corpus, together determine how the results of the underspeciﬁed systems for [f] and [th] should be interpreted. The corresponding substitutions of phone [f] for German using the same method are noted in Table 5. 5.5

Discussion

In this section the substituted phones for the underspeciﬁed systems of both the American English and German speech corpora are presented, where the complete set of results for the phones that are common to both corpora are listed in Table 5 with their associated substitutions. However it is important to note that some of the substituting phones are exclusive to one corpus. Also the substituted phones are displayed left to right with a decreasing substitution count. One of the ﬁrst items of note is that the [ ] phone of the American English corpus (known as [ix]) is a frequent substituting phone. The reason for this is clariﬁed when its frequency of occurrence is found to be the highest for its corpus, over twice that of the schwa . This close-central vowel appears to be the default sound for the domain of the American English corpus as it substitutes both vowel and consonants alike and appears as a top ﬁve substitute for several phones as well as appearing as a substitute beyond the top ﬁve list. Looking at [f] from the German corpus it can be seen that in one step in either dimension within the phonetic neighbourhood cube of ﬁgure 2 the obvious voiced step [v] and place of articulation step [s] are found. Plosive phones [t] and [d] of the same place of articulation of [s] are also noted. It is important to note that the [] and [] phones known as [th] and [dh] are exclusive to the American English corpus. Therefore when understanding [f]’s substitutions for the German corpus, [th] and [dh] are not a substituting choice. Looking at these results with respect to their frequency of occurrence, it was found that while [t]

56

M. Kane, J. Mauclair, and J. Carson-Berndsen

Fig. 3. Representation of phonetic neighbourhood with respect to the most frequent phones from the American English corpus, where the larger the circle the greater the frequency of occurrence Table 5. Subset of American English and German phone substitutions (LR context) with respect to the IPA using an underspecified system. Only the top five substitutions are shown. HMM models substitutions HMM models substitutions IPA TIMIT Kiel TIMIT Kiel IPA TIMIT Kiel TIMIT Kiel p p p t,b,k,f,d t,v,b,g,k ax @ ,,,,o n,,t,d, b b b d,p,v,, d,t,v,m,a i iy i: ,e,,,y e,l,n,,t t t t d,k,p,s,t d,k,v,p,s y y y: i,, ,,k ,v,,i,k d d d t,b,,g, t,n,z,g, ih I ,,i,e, ,,e,i,l k k k t,g,p,d, t,g,p,x, u uw u:

, ,o,,l ,o,a g g g k,d,b,t, d,k,z,t,b uh U ,,,o, u, ,a,,n f f f ,v,p,t,s t,v,s,k,d eh E,E: ,,,,e , ,a v v v f,b,,p,d f,d,m,b,t ao O ,o,w,l, a,a, ,n s s s z,t, ,f, t,f,z,,a a aw aU ,, ,,o a, ,, ,o z z z s,d, ,, s,t,d,n,l a ay aI ,,,e, a,, ,l, m m m n,, ,v,b n, ,l,z, l l l ,w,o, , n,t,,d,m n n n m,,,d,t ,m,t,, ng N n,m,,g,k n,k,u,o

substituted [f] the most, it also has the highest frequency of occurrence (percentage of phone in the complete German corpus) of (9.1%). [v] ranked as the second most substituted phone with a frequency of occurrence of 2.85% where [s] ranked the third most substituted phone even though its frequency of

Automatic Identification of Phonetic Similarity Based on Underspecification

57

occurrence is 6.24%. In this case the phonetic neighbourhood similarity between [f] and [v] outweighed the frequency of occurrence of [s] but not [t]. Finally the unvoiced velar plosive [k] with a frequency of occurrence of 1.7% outweighed the voiced alveolar plosive [d] with its frequency of occurrence of 5.07%. It appears that the choice of substituting phone is based on primarily two weights, (phonetic neighbourhood proximity of substituting phone)*(frequency of occurrence of substituting phone).

6

Conclusion

This paper has presented a novel approach to the identiﬁcation of phonetic similarity using properties observed during the speech recognition process. An experiment was presented whereby the phones [f] and [th] were separately removed during the training phase of a statistical speech recognition system so that the behaviour of the system could be analysed to see which alternative phones were selected. The analyses of underspeciﬁed systems was illustrated using the phones [f] and [th] from an American English corpus with respect to three determining factors: common phonetic properties, phonetic neighbourhood and frequency of occurrence of the phone in the full corpus. Other common phones to both the American English and German corpora yielded similar results and are presented in Table 5. The frequency of occurrence of a substituted phone was seen to have a strong eﬀect on its ranking among the alternatives although phonetic neighbourhood and phonetic properties also played an important role. In the context of the results and analysis presented in section 4 and 5, phonetic neighbourhood was interpreted without resort to an exact measure of distance between the points on the cube. As mentioned there, a more detailed explication of the role of phonetic neighbourhood may be possible, based on a numerical distance measure and a diﬀerentiation between the relative weightings of the dimensions of the cube. For example, it may be that remaining on the place of articulation axis represents closer proximity than moving to the voicing axis for some occasions; it would certainly appear reasonable that a change in the manner of articulation would involve a greater phonetic distance in comparison to a change in voicing information. A factor which will be taken into account in future work is the extent to which the context (prosodic position and segmental context [10]) inﬂuences the substituted alternative; for example a substituted phone may have emerged as a result of an inﬂuence of the preceding or following context/phone. All of these points will be considered in the next phase of experimentation and the experiments will be extended to include other broad classes of phones with the aim of providing a principled methodology for the prediction of phonetic similarity for the purposes of speech recognition. A phonetic similarity measure can serve as an important additional source of information for the construction of acoustic models in statistical speech recognition, for enhancing the lexicon with appropriate phonetic variants and for the design of knowledge-based feature detection engines. In summary, future work envisages that these phonetic

58

M. Kane, J. Mauclair, and J. Carson-Berndsen

similarity measures will be used with the output of an ASR system so that if a conﬁdence metric is low for some output, then the phonetic similarity measure will enable the system to oﬀer an appropriate alternative.

References 1. Halberstadt, A., Glass, J.: Heterogeneous acoustic measurements for phonetic classification. In: Eurospeech Proceedings, pp. 401–404 (1997) 2. Scanlon, P., Ellis, D., Reilly, R.: Using broad phonetic group experts for improved speech recognition. IEEE Transactions on Audio, Speech and Language Processing 15(3), 803–812 (2007) 3. Garofolo, J., Lamel, L., Fisher, W., Fiscus, J., Pallett, D., Dahlgren, N.: The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM (1993) 4. Mauclair, J., Aioanei, D., Carson-Berndsen, J.: Exploiting phonetic and phonological similarities as a first step for robust speech recognition. In: EUSIPCO Proceedings (2009) 5. Van Thuan, P., Kubin, G.: Dwt-based phonetic groups classification using neural networks. In: ICASSP Proceedings, pp. 401–404 (2005) 6. Ghiselli-Crippa, T., El-Jaroudi, A.: Voiced-unvoiced-silence classification of speech using neural nets. In: IJCNN Proceedings, pp. 851–856 (1991) 7. The-International-Phonetic-Alphabet (2005), http://www.langsci.ucl.ac.uk/ipa/ 8. Young, S., Evermann, G., Gales, M., Hain, T., Kershaw, D., Liu, X., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev, V., Woodland, P.: Hidden markov model toolkit (htk) (2009), http://htk.eng.cam.ac.uk/, Version 3.4.1 9. IPDS, CD-ROM#2: The Kiel Corpus of Spontaneous Speech, vol. 1, Kiel, IPDS (1995) 10. Chomsky, N., Halle, M.: The sound pattern of english. Harper & Row, New York (1968)

Error Detection in Broadcast News ASR Using Markov Chains Thomas Pellegrini and Isabel Trancoso INESC-ID R. Alves Redol, 9 1000-029 LISBON, Portugal [email protected]

Abstract. This article addresses error detection in broadcast news automatic transcription, as a post-processing stage. Based on the observation that many errors appear in bursts, we investigated the use of Markov Chains (MC) for their temporal modelling capabilities. Experiments were conducted on a large Amercian English broadcast news corpus from NIST. Common features in error detection were used, all decoder-based. MC classiﬁcation performance was compared with a discriminative maximum entropy model (Maxent), currently used in our in-house decoder to estimate conﬁdence measures, and also with Gaussian Mixture Models (GMM). The MC classiﬁer obtained the best results, by detecting 16.2% of the errors, with the lowest classiﬁcation error rate of 16.7%. To be compared with the GMM classiﬁer, MC allowed to lower the number of false detections, by 23.5% relative. The Maxent system achieved the same CER, but detected only 7.2% of the errors. Keywords: Error detection, automatic speech recognition, Markov chains.

1

Introduction

Error detection is an important topic in Automatic Speech Recognition (ASR). Three types of errors can occur in the hypothesized word stream output: substitutions, insertions and deletions. Having a conﬁdence measure indicating a potential substitution or insertion error for each hypothesized word is useful in several applications: to discard sentences with errors in real-time broadcast news subtitling systems, to try to correct errors by searching text material similar to what is being transcribed, to help select automatically material for unsupervised model training or speaker model adaptation, to validate results of keyword spotting, or else to detect out-of-vocabulary words. Conﬁdence measures can be used to classify hypothesized words into two classes, “correct” and “error”. Many statistical tools have been proposed in the literature: generalized linear models [Gillick et al.(1997), Allauzen(2007)], artiﬁcial neural networks [Weintraub et al.(1997)] and more recently conditional random ﬁelds [Xue et al.(2006)]. Conﬁdence estimation is still challenging, since Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 59–69, 2011. c Springer-Verlag Berlin Heidelberg 2011

60

T. Pellegrini and I. Trancoso

one of the diﬃculties remain in the decoding process itself: to allow computation eﬃciency, the search space is pruned. Hence, word posteriors that are the main feature for conﬁdence estimates are over-estimated [Hillard et al.(2006)]. This problem will not be addressed in this article, rather we will focus on a common observation, that errors appear very often in bursts. For example, an out-of-vocabulary word is known to generate between 1.5 and 2 errors [Schwartz et al.(1994)]. Error bursts are well illustrated in the following alignment example, between our ASR decoder output and the corresponding reference. The named entity “John Makero” was not part of our recognition vocabulary and appeared to be responsible of three consecutive errors, indicated by surrounding stars: ref: DR. *JOHN** *MAKERO* *IS*** A PROFESSOR hyp: DR. *ZHANG* *MARKET* *ROSE* A PROFESSOR

The presence of multi-word error sequences in the output word stream justiﬁes the use of statistical tools that model temporal sequences in some way, such as Markov Chains (MC), or linear-chain conditional random ﬁelds. In this study, we propose a two-state MC, with one “error” state, and one “correct” state, respectively trained on only errors and correct words from the decoder output. In the following, features for error modelling will be listed, and the various statistical models will be brieﬂy presented. Section 4 describes the American English HUB-4 NIST corpus used to train and test the models. Then error detection results are provided, based on the automatic transcription of the corpus performed by our in-house decoder. Classiﬁcation results of the various classiﬁers will be compared, and complementary experiments with MC will be presented.

2

Features for Error Detection

The output of the ASR system is a stream of words. For each hypothesized word, various decoder-based features are available. In this study, only words from the best hypothesis are considered. A set of 15 features common in error detection was used: . Length of words in number of decoding frames (20 ms duration) and in number of phones (2) . Final, acoustic and posterior scores (3) . Average phone acoustic and posterior scores (2) . Log of the total and average active states, arcs and tokens (6) . Minimum and average phone log-likelihood ratios (2) Features related to the active states, arcs and tokens for each hypothesized word should be intuitively high to reﬂect a large degree of uncertainty of the recognizer [Gillick et al.(1997)].

Error Detection for ASR

3

61

Models for Error Detection

Many distinct types of statistical classiﬁers can be used. Currently, our in-house ASR system estimates conﬁdence measures with a maximum entropy model. In this study, we compared this discriminant model with generative models, Gaussian Mixture Models and Markov Chain Models. 3.1

Maximum Entropy Models

Maximum Entropy (Maxent) models are very popular models, and are used in many applications, in particular in natural language processing tasks, such as part-of-speech tagging [Ratnaparkhi(1996)]. The Maxent principle states that the correct probability distribution for a given class is the one that maximizes entropy, given constraints on the distribution [Jaynes(1957)]. One advantage of Maxent models is that the training algorithm will determine how to combine the diﬀerent features by estimating the best weights, so that the main user eﬀort will consist of identifying which features are best to be used. In our case, the Maxent was used as the following: when the probability or conﬁdence measure given by the model is lower than 0.5, then the hypothesized word is labeled as an error. In pratice, larger decision thresholds are used: about 0.8 and more to select automatically transcribed data to do unsupervised acoustic model training for example. To train the Maxent model, the Megam toolbox1 was used. 3.2

Markov Chains

In Markov Chains (MC), a sequence of observable states generates a sequence of observations. Each state has its own probability distribution, generally a mixture of Gaussians [Rabiner et al.(1986)]. MCs are very adapted to compute a probability of a sequence of temporal observations. For that reason MCs appear very attractive to detect error sequences. The MC scheme with the transition probabilities is shown in ﬁgure 1. A 2state MC is used, with one “error” state and one “correct” state. To train this MC, one would need to align the training data at state-level, which does not make sense since by deﬁnition, states are hidden. Hence, each state was trained separately as single MC and then merged together into the ﬁnal model. This approach allows to use diﬀerent numbers of Gaussian mixtures for both states according to the available amount of training data for each class. The transition matrix was designed manually. Since errors often occur in bursts, the self-loop probability to stay in the single error state (value 0.55), has been chosen larger than the transition probability between the two states (value: 0.35). Also, since there are much less errors than correct words, we applied the same transition values for the correct state. Intuitively, it is more likely 1

Available at http://www.cs.utah.edu/~ hal/megam

62

T. Pellegrini and I. Trancoso

Fig. 1. Markov chain used for error detection. One state models the error probability distribution, and the other state models the correct distribution. Self-loop probabilities, ie probabilities to stay in a same state, have been chosen larger than the transition probabilities between the two states to model the error burst phenomenom. The two smallest circles in the ﬁgure are entry and exit non-emitting states.

to have a correct word if the preceding one is correct. The HTK toolkit2 was used to train and test MCs. 3.3

Gaussian Mixture Models

Gaussian Mixture Models (GMM) are linear combinations of Gaussian probability densities, whose weights, means and variances are optimized on a training corpus [Bishop et al.(2006)]. GMMs can be seen as single-state MCs. Hence, GMM have no temporal modelling capabilities. It is interesting to compare their performance to MCs, to evaluate the need to model sequences. In this study, two GMMs were trained, one for the error class and one for the correct class. Classiﬁcation is made based on a simple comparison between the log-likelihoods estimated with the two models.

4

Corpus

The corpus used in this study were the training and evaluation corpora of the NIST HUB-4 American English transcription 1997 campaign3 . These broadcast news sets totalizes respectively about 67 hours and 3 hours of manually transcribed speech. The 70 hours were transcribed automatically with our in-house speech recognition decoder, that will be brieﬂy described in section 5.2. All the training recognized data was used to train our statistical classiﬁers. Table 1 gives the number of transcribed words for both data sets. Since transcription errors are our classiﬁcation target, errors were considered as the “positive” class. 2 3

Available at http://htk.eng.cam.ac.uk These corpus, with references LDC1998S71 and LDC2002S11, are available at www.ldc.upenn.edu

Error Detection for ASR

63

Table 1. Number of positive (errors) and negative (correct words) examples in both train and test sets Train Test Total 674,452 Total 30,014 Positives Negatives Positives Negatives 57,326 617,126 5,192 24,822

5 5.1

Experiments Evaluation

Errors are detected only with hypothesized words, thus only substitutions and insertions are addressed, and not deletions. Hence, the Word Error Rate (WER) is given by: WER =

# (Substitutions+Insertions) # (hypothesized words)

Error detection will be evaluated on a global Classiﬁcation Error Rate (CER), deﬁned as: # (Number of incorrect classiﬁcations) CER = # (hypothesized words) Nevertheless, CER depends on the relative sizes of the number of errors and correct words. Since there are hopefully many more correct words than errors, CER is not very satisfying to measure error detection performance. Hence, classiﬁers will also be characterized by statistics over true and false positives, in particular by drawing Receiver Operating Characteristics (ROC). 5.2

Automatic Transcription

Experiments were conducted with our in-house ASR system, named AUDIMUS [Meinedo et al.(2003)]. It is a hybrid Artiﬁcial Neural Networks / Hidden Markov Models system. A set of 455 context dependent diphone-like acoustic models, plus two non-speech models (one for silence and one for breath) is used to transcribe American English. More details about the context dependency modelling can be found in [Abad et al.(2008)]. Acoustic models were trained on 140 hours of manually transcribed HUB-4 speech. The language model is a 4-gram model, with Kneser-Ney modiﬁed smoothing, trained on 150 million words from HUB-4 transcripts, and about 1 billion words of newspaper and newswire texts. The 64k word vocabulary consists of all the words contained in the HUB-4 training set plus the most frequent words in the broadcast news texts and Newspapers texts. Multiple-pronunciations are allowed and totalize 70k entries. The word error rate (WER) for the test corpus was 24.0%. This value is higher than the standard WER, when a normalization on the output is used before scoring (verbal form expansion in particular). The WER can be seen as the classiﬁcation error rate of an ultra-liberal classiﬁer, that would predicts as correct all the output words.

64

T. Pellegrini and I. Trancoso

1

True positive rate

0.8

0.6

0.4 Maxent Maxent t=0.5 Maxent t=0.91 Threshold Maxent GMM MC

0.2

0

0

0.2

0.4 0.6 False alarm rate

0.8

1

Fig. 2. ROC graph. The threshold curve corresponds to the conﬁdence measure threshold used to draw the Maxent curve.

5.3

Error Detection Results

Maxent models, GMMs and MCs were trained on the same data set. All GMMs and MCs have 512 and 32 Gaussian mixtures for respectively the correct state and the error state. In mean, this gives respectively about 1200 and 1800 examples per mixture to train the two models. Larger numbers of mixtures were tried for the error model, but lead to worse results. Table 2 shows the classiﬁcation results, the error class being considered as the positive class. The table gives the Classiﬁcation Error Rate (CER), along with positive and negative detection statistics: true and false negatives and positives. To further illustrate these results, ﬁgure 2 shows the ROC graph for the diﬀerent classiﬁers. In this type of graph, points closer to the left-top corner Table 2. Results in terms of classiﬁcation error rate (CER), true and false positives (tp, fp) and negatives (tn , fn)

Maxent t=0.5 Maxent t=0.91 GMM MC

CER 16.7 25.0 17.4 16.7

tp 375 3,233 840 840

fp 189 5,554 863 660

tn 24,633 19,268 23,959 24,162

fn 4,817 1,959 4,352 4,352

Error Detection for ASR

65

correspond to the best classiﬁers. The datched line gives the decision threshold with which the Maxent curve was drawn, as a function of the false alarm rate. For example, when a 0.8 threshold is used, all conﬁdence measures estimated by Maxent that are smaller than 0.8 will give an error label for the words to be classiﬁed. The plain line corresponds to the performance of the Maxent classiﬁer, when varying the decision threshold. Two points on the Maxent curve were added to show the Maxent performance, at two operating points: one corresponding to a standard 0.5 threshold value, and one to the best threshold value of 0.91 as predicted by the curve, to be as close to the top-left corner as possible. The Maxent results can be seen as our baseline, since it is actually used in our ASR system. Performance of the GMM and the MC classiﬁers are indicated by only two single points, since they were used as binary decision tools, by simply comparing the probabilities for a word to be correct or wrong. Maxent with a 0.5 threshold value and MC gave the best CERs, with a 16.7% value. Nevertheless, only 375 errors out of 5.2k were detected by Maxent, whereas MC detected 840 errors, corresponding to 16% of the errors. Most of the probabilities given by the Maxent model were larger than the standard 0.5 threshold, even for mis-recognized words. Thus, this threshold can be chosen larger but the CER will increase, due to a larger number of false alarms. According to the ROC curve, that focus only on positive performance rates, the best working point would be the (false alarm rate=25.5%, detection rate=66.0%) point closest to the (0,1) ideal point, corresponding to a 0.91 decision threshold. At this working point, Maxent detected about 3.2k true positives, but the number of false alarms was very high, with a 5.6k value. The corresponding CER was 25.0%, which is larger than the WER. 80% of the errors detected by GMM were also detected by MC, but GMM showed a much larger number of false alarms, with an increase of about 30% relative. Both GMM and MC ROC points are much higher in the graph than the Maxent t=0.5 point, showing better results. Nevertheless, these points are still far from the ideal (0,1) point. The choice of using MC or Maxent at t=0.91 will depend on the application. If priority is given to detection, then large number of false detections may not be critical, and Maxent with a large threshold could be used. In the inverse case, MC might be better. 5.4

Result Analysis

All classiﬁers presented high false alarm rates. In particular, when increasing the decision threshold used with the Maxent model, the number of wrong error label detections (false alarms) rapidly increased. The most frequent false alarms (FA) appeared to be very short words. The ten most frequent FA were: THE, IN, I, TO, SOME, OF, A, THIS, AS, AND. The mean word length in characters of the false alarms was smaller than the mean length for the true positives: 4.9 versus 6.1. This may be due to the fact that most insertion errors of the decoder were small words. Then, the error classiﬁer was inclined to label short words as errors too easily. When using conﬁdence measures, a higher decision threshold could be used for frequent short words.

66

T. Pellegrini and I. Trancoso

700

Maxent t=0.5 GMM MC

600

# Sequences

500 400 300 200 100 0

1

2

3 # Words

4

5

Fig. 3. Number of error segments correctly labeled by Maxent, GMM and MC, as a function of their length in number of words

It is interesting to compare GMM and MC, since GMM can be seen as a single-state MC, with no temporal modelling capabilities. The better performance of MC was due to a smaller number of FAs, compared to GMM. These false detections correspond mainly to single error labels in a sequence of correct words, instead of error detections in a sequence of consecutive errors in the ASR output. This seems to conﬁrm the usefullness of the temporal modelling capabilities of MC, which give more “inertia” to the model. When the preceding word has been classiﬁed as correct, the current word has a higher probability to be also labeled as correct, and idem for the error class. Figure 3 shows the number of word sequences correctly labeled by Maxent, GMM and MC, as a function of their length in number of words. It appears that MC predicted more multi-word error sequences than the other two models. GMM predicted much more single-word errors. 5.5

Impact of the Transition Probability Matrix

One intuition that lead to test MCs to detect ASR output errors was that very often, errors appear in bursts. The self-loop probability to stay in the error state was therefore chosen larger than the transition to the other state. The transition probability matrix used so far in this study was:

Error Detection for ASR

67

⎡

⎤ 0.0 0.5 0.5 0.0 ⎢ 0.0 0.65 0.25 0.1 ⎥ ⎢ ⎥ ⎣ 0.0 0.25 0.65 0.1 ⎦ 0.0 0.0 0.0 0.0 Note that only the 2x2 central part of the matrix is of interest since the other values concern the entry and exit non-emitting states. Results for two additional MC, named hereafter MC-b and MC-c, are reported here. Respective transition matrices were the following: ⎡ ⎤ 0.0 0.5 0.5 0.0 ⎢ 0.0 0.45 0.45 0.1 ⎥ ⎢ ⎥ ⎣ 0.0 0.45 0.45 0.1 ⎦ 0.0 0.0 0.0 0.0 and

⎡

⎤ 0.0 0.5 0.5 0.0 ⎢ 0.0 0.25 0.65 0.1 ⎥ ⎢ ⎥ ⎣ 0.0 0.65 0.25 0.1 ⎦ 0.0 0.0 0.0 0.0

For MC-b, the probabilities to stay in a single state or to jump to the second state are the same (0.45). For MC-c, the probability transition between the two states is larger than the self-loop probability (0.65 opposed to 0.25). Table 3 gives the performance results for the three models MC, MC-b, MCc, all trained on the same data. Classiﬁcation error rates for MC-b and MC-c were larger than MC’s CER. The number of correct detections increased, but the number of false alarms increased much more. Self-loop probabilities larger than transition probs gave the best performance. This result seems to validate the assumption that it is more likely to stay in a single state, error or correct, i.e. errors mostly occur in bursts, rather than isolated. Table 3. Classiﬁcation error rate (CER), true and false positives (tp, fp) and negatives (tn , fn) for MC-b and MC-c that diﬀer from MC only on the transition and self-loop probabilities CER tp fp tn fn MC 16.7 840 660 24,162 4,352 MC-b 17.3 1,114 1,124 23,698 4,078 MC-c 18.7 1,443 1,877 22,945 3,749

6

Summary and Future Work

In this article, the problem of detecting errors in automatic transcriptions has been addressed with the use of various statistical tools, with the idea that recognition errors often appear in “bursts”, i.e. in sequences of several wrong hypothesized words. The Markov chains ability to model temporal sequences has been

68

T. Pellegrini and I. Trancoso

tested by comparing this approach to a Gaussian Mixture Model (GMM), and to a maximum entropy model, that is currently used in our in-house ASR system to estimate conﬁdence measures. Experiments were carried out on a large American English broadcast news speech NIST corpus. A Maxent model with a 0.5 decision threshold was able to detect only 7% of the errors correctly. With a 0.91 threshold, this percentage raised 62% but the number of false errors increased much more, to almost double the number of correct detections. The Markov chain outperformed Maxent and GMM, with a 16.7% CER and 860 errors correctly detected. The temporal modelling capabilities of this model seemed to bring a useful “inertia” that lowered the number of false alarms. The choice of using MC or Maxent with a large decision threshold will depend on the application. If priority is given to detection, then large number of false detections may not be critical, and Maxent could be used. In the inverse case, MC might be better. Result analysis showed that the Maxent and GMM models detected mainly single-error words in the decoder word output stream. The MC system was able to detect more multi-word error sequences, justifying the use of a model with temporal sequence modelling capabilities. This last assumption has been also conﬁrmed by the MC superiority over GMM. Future work will consist in comparing MCs to their somehow equivalent discriminant model, linear-chain conditional random ﬁelds, recently used in many natural language processing tasks. Finally, the ability to mark words recognized with low conﬁdence in an automatically recognized broadcast news transcript is also very relevant for our computer aided language learning system [Marujo et al.(2009)]. Learning from recent documents such as broadcast news videos with automatically produced captions is one of the features that may make the use of the system more motivating for students. The captions have a diﬀerent color for the low conﬁdence words.

Acknowledgements This work was partially supported by FCT (INESC-ID multi-annual funding) through the PIDDAC Program funds, by the FCT REAP.PT project (CMUPT/HuMach/0053/2008) and the Vidivideo European project.

References [Gillick et al.(1997)] Gillick, L., Ito, Y., Young, J.: A probabilistic approach to conﬁdence estimation and evaluation. In: Proceedings of ICASSP, Munich, pp. 879–882 (1997) [Allauzen(2007)] Allauzen, A.: Error detection in confusion. In: Proceedings of INTERSPEECH, Antwerp, pp. 1749–1752 (2007) [Weintraub et al.(1997)] Weintraub, M., Beaufays, F., Rivlin, Z., Konig, Y., Stolcke, A.: Neural - Network Based Measures of Conﬁdence for Word Recognition. In: Proceedings of ICASSP, Los Alamitos, pp. 887–890 (1997)

Error Detection for ASR

69

[Xue et al.(2006)] Xue, J., Zhao, Y.: Random forests-based conﬁdence annotation using novel features from confusion network. In: Proceedings of ICASSP, Toulouse, pp. 1149–1152 (2006) [Hillard et al.(2006)] Hillard, D., Ostendorf, M.: Compensating for Word Posterior Estimation Bias in Confusion Networks. In: Proceedings of ICASSP, Toulouse, pp. 1153–1156 (2006) [Schwartz et al.(1994)] Schwartz, R., Nguyen, L., Kubala, F., Chou, G., Zavaliagkos, G., Makhoul, J.: On Using Written Language Training Data for Spoken Language Modeling. In: Proceedings of ACL, New Jersey, pp. 94–97 (1994) [Ratnaparkhi(1996)] Ratnaparkhi, A.: A Maximum Entropy Model for Part-Of-Speech Tagging. In: Proceedings of EMLNP, Philadelphia, pp. 133–142 (1996) [Jaynes(1957)] Jaynes, E.T.: Information theory and statistical mechanics. Physical review 106(4), 620–630 (1957) [Rabiner et al.(1986)] Rabiner, L.R., Juang, B.H.: An Introduction to Hidden Markov Models. IEEE Acoustics Speech and Signal Processing Magazine ASSP-3(1), 4–16 (1986) [Bishop et al.(2006)] Bishop, C.: Pattern recognition and machine learning. Springer, Heidelberg (2006) [Meinedo et al.(2003)] Meinedo, H., Caseiro, D., Neto, J., Trancoso, I.: AUDIMUS.MEDIA: A broadcast news speech recognition system for the european portuguese language. In: Mamede, N.J., Baptista, J., Trancoso, I., Nunes, M.d.G.V. (eds.) PROPOR 2003. LNCS, vol. 2721, pp. 9–17. Springer, Heidelberg (2003) [Abad et al.(2008)] Abad, A., Neto, J.: Incorporating Acoustical Modelling of Phone Transitions in a Hybrid ANN/HMM Speech Recognizer. In: Proceedings of INTERSPEECH, Brisbane, pp. 2394–2397 (2008) [Marujo et al.(2009)] Marujo, L., Lopes, J., Mamede, N., Trancoso, I., Pino, J., Eskenazi, M., Baptista, J., Viana, C.: Porting REAP to European Portuguese. In: SLATE 2009 - Speech and Language Technology in Education, Brighton (2009)

Pronunciation and Writing Variants in an Under-Resourced Language: The Case of Luxembourgish Mobile N-Deletion Natalie D. Snoeren, Martine Adda-Decker, and Gilles Adda LIMSI-CNRS, BP 133 91403, Orsay, France http://www.limsi.fr

Abstract. The national language of the Grand-Duchy of Luxembourg, Luxembourgish, has often been characterized as one of Europe’s underdescribed and under-resourced languages. Because of a limited written production of Luxembourgish, poorly observed writing standardization (as compared to other languages such as English and French) and a large diversity of spoken varieties, the study of Luxembourgish poses many interesting challenges to automatic speech processing studies as well as to linguistic enquiries. In the present paper, we make use of large corpora to focus on typical writing and derived pronunciation variants in Luxembourgish, elicited by mobile -n deletion (hereafter shortened to MND). Using transcriptions from the House of Parliament debates and 10k words from news reports, we examine the reality of MND variants in written transcripts of speech. The goal of this study is manyfold: quantify the potential of variation due to MND in written Luxembourgish, check the mandatory status of the MND rule and discuss the arising problems for automatic spoken Luxembourgish processing. Keywords: pronunciation variants, writing variants, low e-resourced languages, Luxembourgish.

1

Introduction

The national language of the Grand-Duchy of Luxembourg, Luxembourgish or "Lëtzebuergesch", has often been characterized as one of Europe’s underdescribed and under-resourced languages. Just like the English language, Luxembourgish can be considered as a mixed language with strong Germanic and Romance inﬂuences. It is hard to estimate the precise proportion of Germanic and Romance inﬂuenced words in Luxembourgish, as these proportions are modulated by the communicative settings. For instance, although vernacular Luxembourgish is mainly inﬂuenced by Germanic stems, there are Romance words to be found as well (Merci, Äddi,"Adieu", futti, colloquial of the French "foutu" meaning "damned"). Nonetheless, more technical and administrative communication systems include a high proportion of Romance words (Konditioun, "condition"; agéieren, "to act"; abordéieren, "get into" ). Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 70–81, 2011. c Springer-Verlag Berlin Heidelberg 2011

Pronunciation and Writing Variants in Luxembourgish

71

It is estimated that about 300,000 people worldwide speak Luxembourgish. As was previously pointed out in [7] and [13], Luxembourgish should be considered as a partially under-resourced language, mainly because of the fact that written production remains relatively low. Rather surprisingly, written Luxembourgish is not systematically taught to children in primary school, German being usually the ﬁrst written language learned, immediately followed by French [21]. Although many eﬀorts have been made in the past to standardize an oﬃcial orthography of Luxembourgish, no oﬃcially recognized spelling system was being recognized until the adoption of the "OLO" (ofizjel lezebuurjer ortografi) in 1946, which aimed at producing written forms that clearly diverge from German orthography. In spite of its oﬃcial character, it never became popular in schools. A more successful standardization eventually emerged from the work of specialists charged with the task of creating a dictionary that was published between 1950 and 1977 [3]. Nonetheless, up until today, German and French are the most practiced languages for written administrative purposes and communication in Luxembourg, guaranteeing a larger dissemination, whereas Lëtzebuergesch is the main language used for oral communication purposes between native speakers of Luxembourgish. The strong inﬂuence of both German and French, among other factors, can explain the fact that Luxembourgish exhibits a large amount of both pronunciation and derived potential writing variants. For instance, it is fairly common to have several regional pronunciations for function words (e.g. the English personal determiner "our" can be written and pronounced as eis [ajs], ons [Ons], is [i:s]). Pronunciation variants may give rise to resulting variations in written Luxembourgish, as Luxembourgish orthography strives for phonetic accuracy [2]. The question then arises, in particular for oral transcripts, whether the written form reﬂects the perceived pronunciation form or whether some sort of normalization process is at work that eliminates part of the variation. The current paper addresses the study of spoken and written variants arising from Luxembourgish mobile n-deletion, according to which the phoneme -n can be deleted in speciﬁc phonemic contexts (both word-ﬁnally and word-internally). Eﬃcient automatic speech processing relies on a good deal on the availability of both written and spoken language corpora. Luxembourgish can be considered as an under-resourced language in the sense that the production of written material is very limited. One of the main reasons for this is that French and German are used as the two main written communication languages. Apart from written materials, the use of sibling resources that provide similar content in both written and auditory modalities has proven to be particularly helpful for automatic speech recognition (ASR). In Luxembourg, news broadcasts are delivered in Luxembourgish on a daily basis. Newspapers, however, remain for the most part bilingual German/French with occasional code-switching to Luxembourgish (especially for titles). In spite of the ubiquitous inﬂuence of German and French on Luxembourgish (see e.g., [8]), a lot of eﬀort has been made over the past few years to establish Luxembourgish word lists and multilingual dictionaries in electronic form ( [22] and references therein). As far as web resources are

72

N.D. Snoeren, M. Adda-Decker, and G. Adda

concerned, Luxembourgish holds rank 55 in the list of 272 oﬃcial wikipedias (cf. the Wikimedia foundation for various languages). This means that about 28000 Wikipedia articles have been created in Luxembourgish, which shows that there is a societal demand to communicate in this language.

2

The Study of Written and Spoken Variants

Over the past decades, one of the main challenges in automatic speech recognition pertained to the question as to how to handle variation [19]. Both spoken and written forms need to be modeled, as well as any resulting spoken and written variations to obtain eﬃcient ASR modeling. In the current contribution, we will address variants that result from Luxembourgish mobile-n deletion (henceforth MND). However, before turning to our study, we will ﬁrst discuss some general aspects of written and spoken variant processing in ASR studies in terms of text normalization and pronunciation variation modeling. 2.1

Text Normalization for Written Variants

In written forms, variation may arise for a number of reasons, such as permissive orthographic rules, speciﬁc writing conventions (e.g., diacritics, abbreviations etc.), formatting issues, plain errors, and last but not least, unstable writing conventions when it comes to languages that are mainly used for oral communication. Typically, written variants are being dealt with through text normalization processes. Eﬃcient text normalization should lead to reliable estimates for n-gram based language models, something that proves to be particularly challenging when developing systems for poorly resourced languages. In text normalization, two diﬀerently represented variants may refer to the same meaning, so instead of treating these as diﬀerent, one can treat them as instances or variants of the same underlying sequence. Ultimately, the goal of text normalization is to remove "noise", achieve better lexical coverage and more precise language models that are critical to the development of performing ASR systems. In text normalization, one deﬁnes the limits of what will be a word in the system. There is, however, an apparent contradiction that needs to be resolved during the optimization of both lexical coverage and language model precision. On the one hand, a minimal number of variants is required so as to reduce the number of Out Of Vocabulary (OOV) words (i.e. the words of the texts that are not part of the vocabulary). On the other hand, one needs to limit the occurrence of ambiguities in order to increase the precision of the language model. Text normalization (mainly language-dependent) is the result of the trade-oﬀ between these two conﬂicting criteria [9]. 2.2

Pronunciation Modeling of Spoken Variants

The need for modeling pronunciation variation stems from the simple fact that the words of a language are pronounced in many diﬀerent ways due to variations in speaking style, interlocutor, communicative context, accent or dialect,

Pronunciation and Writing Variants in Luxembourgish

73

socio-economic factors and so forth. Indeed, pronouncing words implies that they are strung together into connected speech [4], as opposed to the pronunciation of isolated words. As a consequence, all sorts of interactions may take place between words in connected speech, which will result in the application of many phonologically motivated variations such as assimilation, co-articulation, segment reduction, insertion, and deletion. One means to deal with variation that occurs in word pronunciation, is through the creation of speciﬁc lexica that incorporate the most commonly observed phonological variants for each word in the lexicon. However, it has previously been shown that simply adding pronunciation variants at the lexical level does not suﬃce to obtain the best recognition performances [6]. Better results are generally obtained when the probabilities of the pronunciation variants are equally taken into consideration, either in the lexicon or in the language model [19]. The commonly adopted acoustic HMM (Hidden Markov Model) structure can implicitly account for some amount of speech lengthening, especially stemming from hesitation phenomena, and for parallel variants [1]. In a common HMM structure, each sound is represented by a set of allophonic (context-dependent) acoustic models with a three state left-to-right topology. Although this modeling method works fairly well for canonical pronunciations listed in the pronunciation dictionary, pronunciations with a number of phonemes diﬀering from the one speciﬁed in the pronunciation dictionary are generally poorly dealt with [20]. Given the speciﬁcities of Luxembourg, it appears important to check the variations arising from the diﬀerent languages in contact in Luxembourg. It becomes then feasible to focus on Luxembourgish-speciﬁc phonological phenomena, such as mobile n-deletion (hereafter shortened to MND, following [13], also known as the Eifeler rule [5,22]). 2.3

Eﬀects of Luxembourgish MND on Written and Pronunciation Variants

According to the phonological rule of MND, a word-ﬁnal -n is only retained before a vowel or before one of the following phonemes: {n, d, t, ts1 , h}. Any other phonemic right context should normally cause the deletion of the ﬁnal -n. The phoneme -n can also be deleted within compound-word boundaries. That is, the ﬁrst element of compound words ending in -n generally undergoes MND. So, for instance, given a ﬁrst element of the word Fritten ("French fries"), the -n is preserved before /d/ as in Frittendëppen ("chip pan"), but generally deleted before /f/ as in Frittefett ("frying fat"). Preﬁxes ending in -n, also undergo MND. Given the preposition an ("in"), preﬁxed to the verb droen (Ger. "tragen", Eng. "to carry") results in androen ("to register"), whereas preﬁxed to a word such as Fett (Fr. "gras", Eng. "fat"), results in the verb afetten ("to grease"). In the current contribution, we propose to investigate written and pronunciation variants in Luxembourgish that are elicited due to MND, by looking into large transcribed corpora (see [7], i.e. manual transcriptions of recorded speech 1

/ts/ is pronounced [z] in Luxembourgish.

74

N.D. Snoeren, M. Adda-Decker, and G. Adda

from either the Chamber debates or web news reports). By doing so, we are in an excellent position to characterize this particular variant and to establish with what kinds of variants the Luxembourgish listener is actually confronted with.

3

The Current Study

3.1

Data Collection

Sibling resources that provide both audio and corresponding written materials are of major interest for ASR development. The most interesting resource we have come across until so far for Luxembourgish, consists of the Chamber debates (House of Parliament) and to a lesser extent news channels that are delivered by the Luxembourgish radio and television broadcast company. The Parliament debates are broadcast and made available on the oﬃcial web site2 , together with written Chamber reports, that correspond to fairly reliable manual transcripts of the oral debates. Another interesting sibling resource stems from the Luxembourgish radio and television broadcast company RTL, that produces news written in Luxembourgish on its web site3 , together with the corresponding audio data. However, it must be noted that only a very limited amount of written Luxembourgish can be found here, whereas RTL has a profuse audio/video production. Table 1 summarizes the diﬀerent text and audio resources that are currently being collected for further analysis. Table 1. Major Luxembourgish text and audio sources for ASR studies. Collected amounts are given in word numbers, adapted from [7]. written Source: Wikipedia lb.wikipedia.org Volume: 500k Years 2008

3.2

sibling: audio+written Chamber RTL www.chd.lu www.rtl.lu 12M 700k 2002-2008 2007-2008

Characterizing Potential Mobile -N Sites

As was mentioned before, MND concerns the deletion of a word-ﬁnal -n, giving rise to a variant of the same lexical item. Following the oﬃcial Luxembourgish orthography, Luxembourgish words such as wann and wa ("when") are both recognized as existing lexical items and, as such, listed in the dictionary. Because of the fact that our corpora contain items that can occur without word-ﬁnal -n, with -n, or double -n, we ﬁrst sought to know how many Luxembourgish wordﬁnal -n (or -nn) words also occur without a word-ﬁnal -n (or -nn). These items correspond to potential MND sites. To this end, an extraction tool was developed and implemented that took as input the word list derived from the word tokens 2 3

http://www.chd.lu http://www.rtl.lu

Pronunciation and Writing Variants in Luxembourgish

75

of the corpora and produced as output a compressed word list merging all the word-ﬁnal -n variants in the format of the annotation that list word-ﬁnal -n (or -nn) items that also exist without -n. A few examples are given below: [1] gezwonge#n ⇒ gezwonge; gezwongen (Eng. "forced"); [2] ausgi#nn ⇒ ausgi; ausginn (Eng. "spent"); [3] si#n#nn ⇒ si; sin; sinn (Eng. "are"). The input word list from the transcriptions includes 194k distinct word forms. The correct orthography of these words can be checked using the oﬃcial Luxembourgish spelling checker called Cortina, which was developed by the Luxembourgish Centre de Recherche Public G. Lippmann with the support of the CPLL (Conseil Permanent pour la Langue Luxembourgeoise4 ). This checking allows to list all the words that are considered to be oﬃcially admissible Luxembourgish word forms. The oﬃcially correct list is termed here the Cortina list and includes 121k words. As such, the word list can be thought of as a standardized type of dictionary including all canonical lexical forms, contrary to the word lists that are derived from the transcriptions. Since the input word list concerns high-quality transcriptions, the size diﬀerence between the input word list and the Cortina list cannot simply be attributed to transcription errors. Moreover, a lot of Luxembourgish lexical entries have been attested that are not listed in the Cortina list, such as a number of compound words (e.g., Babyjoren; Bäckermeeschter), acronyms (NATO), proper names (Fischbach), or toponyms (Guantanamo).The results of our word-ﬁnal -n variant merging are summarized in Table 2. Table 2. Word type frequencies (%) of potential mobile -n items and variants as found in the lists derived from the transcribed corpora and in the Cortina list (oﬃcial orthography). The ﬁrst line indicates the full word list sizes. Transcriptions Cortina word list size 194k 121k -n variants Word type frequencies (%) #n 30318 (15.6) 5894 (4.9) #nn 583 (0.3) 101 (0.1) #n#nn 15 (0.0) 136 (0.1)

The results of the word-ﬁnal n merging show that a relatively large number of word-ﬁnal -n items also occur without the ﬁnal -n, according to the Cortina list (4.9% of the word types). This proportion more than triples in the Transcriptions list (15.6%), which is not surprising since transcriptions generated by humans generally allow for more variation, including potential errors. Another issue might be related to the fact that the Cortina spell checker did not include all the possible variants due to MND. The large amount of additional wordﬁnal -n variants may arise from genuine variation in the produced speech due to 4

See the following website for more information, http://www.cortina.lippmann.lu/site/

76

N.D. Snoeren, M. Adda-Decker, and G. Adda

the MND process. In future studies this point will be investigated, in particular by confronting sibling written and oral modalities. Although the number of -#n#nn type items in the Cortina list is very low (136 items), it is interesting to note that this type of variants is virtually not occurring in the transcriptions. One possible explanation might perhaps be related to avoidance of redundancy when transcribing (i.e. two orthographic representations correspond to the same phonetic variant). These raw measurements provide us with some interesting clues about potential mobile -n sites in Luxembourgish. The fact that a lot of the resulting MND variants are already listed in word-lists might be helpful in explaining under what circumstances MND occurs in Luxembourgish speech. 3.3

MND in Transcriptions

The goal of a second investigation was to ﬁnd out whether the MND rule is being respected in two transcriptions from the Chamber debates and one transcription from a news channel (transcribed by professional transcribers, who are native speakers of Luxembourgish). A PERL script was implemented that allowed to count the number of lexical items containing a word-ﬁnal n in the phonemic contexts in which MND occurs. Table 3 gives a summary of the word frequency and respective type frequencies (%) of violation of the MND rule (taking into account the exceptions to the rule such as word-ﬁnal -ioun where word-ﬁnal -n is always being retained). Table 3. Word token frequencies (%) and MND violation type frequencies (%) for three transcriptions Transcription Chamber1 Chamber2 News word list size 12395 1952 2326 MND violation (%) 0.39 0.46 2.53

These numbers suggest that there are relatively few cases for which the MND rule is being violated. MND violations may include nouns followed by prepositions (Bühn fir, "stage for", which in this particular example should not be considered as an MND violation but as an MND exception). MND Violations do seem to aﬀect other syntactic categories as well (e.g., Verb-Preposition: huele#n fir "take for", kucke#n vun "watch from", Determiner-Noun: de#n Referendum, "the referendum", Adj-Noun: anere#n Länner, "other countries"). Further examples include nouns followed by verbs (Kirchen gét) which in this case is a genuine MND violation. Obviously, a more in-depth analysis is clearly called for in order to determine whether the number of potential -n sites varies as a function of syntactic and/other linguistic factors. Given these observations, the MND rule is fairly well respected and these results make even more sense in the light of the relatively large number of listed variants resulting from MND that was mentioned before. In order to verify this

Pronunciation and Writing Variants in Luxembourgish

77

hypothesis, however, the next step would be to collect more linguistic information about the type of items that undergo MND and to see whether this information correlates with the potential mobile -n words that are listed in the dictionaries and recognized as lexical items in their own right by Luxembourgish listeners. Finally, transcriptions need to be checked against oral productions to clarify whether MND is similarly respected in the oral modality. 3.4

MND and Word List Coverage

Language model development in ASR requires that the word lists that are being used achieve high lexical coverage. As was previously mentioned, text normalization processes are employed to obtain good lexical coverage. Previous research has looked into lexical coverage of Luxembourgish word lists from raw (i.e. potentially multilingual) and ﬁltered (i.e. approximating monolingual) data by using the Chamber training and development data [7]. It was found that, concerning the composition of the diﬀerent word lists, there were actually very few French and German entries in the ﬁltered Lëtzebuergesch word list, whilst the word lists from the Chamber debates contained a high proportion of Romance import verbs. Here, we sought to quantify the impact of mobile -n variants on lexical coverage in Luxembourgish. To this end, we used the Chamber corpus that consists of 12M raw words as training data to build diﬀerent size word lists (i.e. system vocabularies). A held out development set of 100k raw words was then used to measure the percentage of words covered by the diﬀerent size word lists on the new data. The complementary measure of unknown words, termed Out of Vocabulary (OOV) words, is displayed in Figure 1 as a function of word list size (varying between 10k and 150k lexical items). The corresponding curves inform about the impact of MND, that is, after ﬁltering out all word-ﬁnal -n items, on the word list’s global lexical coverage capacity. As can be seen from the Figure 1, OOV rates overall decrease as the word list size increases. More importantly, the diﬀerence between the MND ﬁltering and the standard development data is relatively important at a low word list size. However, the diﬀerence between the two curves reduces as the word list size increases (beyond 80k). In light of the observed diﬀerences between the MND ﬁltering and development data slopes, it is relevant to see how the curves for word-ﬁnal phonemes other than -n fare with respect to the development data. Figure 2 displays the curves for the 8 most frequent word-ﬁnal phonemes other than word-ﬁnal -n, with a zoom on the 20k-80k word list size range. It can be seen from these Figures that the curves closely parallel the development data slope, whereas the word-ﬁnal -n curve stands out from the rest. The lexical coverage measure thus nicely illustrates how an ASR tool can highlight linguistic phenomena that involve speciﬁc phonemes such as word-ﬁnal -n in Luxembourgish MND. 3.5

Summary and Prospects

In the present paper, we have highlighted the complex linguistic situation of Luxembourgish, a partially under-resourced and under-described language. We

78

N.D. Snoeren, M. Adda-Decker, and G. Adda

Lexical coverage dev standard dev MND-norm

6

%OOV

5 4 3 2 1 0 0

20

40

60

80

100

120

140

Word list size (k) Fig. 1. Out of Vocabulary (OOV) word rates measured as a function of word list sizes from the Chamber standard development data and after MND ﬁltering

Lexical coverage - Development data standard n-norm t-norm r-norm s-norm h-norm g-norm m-norm l-norm d-norm

5 4 3 2

dev standard dev n-norm dev t-norm dev r-norm dev s-norm dev h-norm dev g-norm dev m-norm dev l-norm dev d-norm

3 %OOV

6

%OOV

Lexical coverage 4

2

1 0

1 0

20

40

60

80

100

Word list size (k)

120

140

20

40

60

80

Word list size (k)

Fig. 2. Out of Vocabulary (OOV) word rates measured as a function of word list sizes from the Chamber standard development data and after ﬁltering out various word-ﬁnal consonants (left). Zoom on the 20-80k word list size range (right).

Pronunciation and Writing Variants in Luxembourgish

79

have focused on variants that are elicited by Luxembourgish mobile n-deletion (MND). According to the rule that underlies MND, word-ﬁnal -n should be deleted in speciﬁc phonological contexts. Thus, MND elicits variants of the same lexical item. Although there are relatively few written resources in Luxembourgish as compared to other languages such as English and German, corpus studies in Luxembourgish will substantially add to the current debate on the processing of variants in automatic and natural speech processing. An important question that is raised by the ASR community, is to know whether the variation is modeled at the lexical level or handled by the acoustic models. It has previously been shown that better recognition performances can be obtained when taking into account the probabilities of pronunciation variants, either at the lexical level or in the acoustic models [11]. This information can be readily derived from the type of large corpus-based analyses we are proposing here. Moreover, in order to assess pronunciation and their derived writing variants, it seems that representative data are needed. New methods that are based on pronunciation rules, rather than on the variants directly, can be used to generalize over variants unseen in the training data. From this respect, Luxembourgish MND provides an excellent test-case, as the variants elicited by MND occur in speciﬁc phonological contexts and are governed by a linguistic rule. Computational ASR investigations and corpus-based analyses will not only enhance the development of a more full-ﬂedged ASR system for Luxembourgish, but can also be used to highlight speciﬁc language phenomena that can make important contributions to linguistic enquiries. It was previously mentioned that Luxembourgish has undergone strong inﬂuences from both French and German (and more recently from English). This raises the question whether Luxembourgish spoken words are globally better represented by one of these languages. To address this issue, recent research conducted in our lab [8] has looked into the acoustic properties of Luxembourgish by comparing acoustic seed models for Luxembourgish with monolingual German, French, and English acoustic model sets by means of IPA symbol correspondences. Although Luxembourgish speech is frequently interspersed with French words, forced alignments on the data sets showed in fact that German acoustic models provided the best match with the Luxembourgish acoustic models, thereby underpinning the strong Germanic typology of the language. Another important issue pertains to the question as to how human listeners cope with pronunciation variants. Indeed, over the last decade an ever-growing number of studies has looked into perceptual processing mechanisms of variants in spoken word recognition, most notably assimilation of place of articulation in English (cf. [14,18]). Corpus-based studies on variants such as the ones elicited by MND are bound to generate novel predictions about the representation in the mental lexicon and the cognitive mechanisms that are assumed to underly perceptual processing of variants (see also [10]). These predictions can be readily tested in behavioural tests, (e.g., lexical decision and priming experiments or perceptual categorization tests). For instance, a critical aspect in the debate on lexical representation and their underlying phonological structure is whether

80

N.D. Snoeren, M. Adda-Decker, and G. Adda

the capacity of distinguishing variants (e.g., those elicited by n-deletion) has to do with auditory perceptual abilities or whether explicit information, i.e. information about the written forms, over the contrastive sounds may be needed to build separate lexical representations. Given the numerous implications and applications that follow from large corpus-based studies, it is hoped that this line of research on Luxembourgish will sparkle more interest for the language in researchers working in the domains of ASR, cognitive psychology, and linguistics. Acknowledgements. The production of this article has been supported by a grant from the Luxembourgish F.N.R (Fonds National de la Recherche) awarded to N.D.S. and has been partially ﬁnanced by OSEO under the Quaero program.

References 1. Adda-Decker, M., Lamel, L.: Pronunciation Variants across Systems, Languages and Speaking Styles. Speech Communication 29, 83–98 (1999) 2. Schanen, F.: Parlons Luxembourgeois. L’Harmattan (2004) 3. Linden, P.: Luxemburger Wörterbuch. P. Linden, Hofbuchdrucker (1950) 4. Kaisse, E.: Connected Speech: The Interaction of Syntax and Phonology. Academic Press, Orlando (1985) 5. Gilles, P.: Phonologie der n-Tilgung im Moselfränkischen (’Eiﬂer Regel’): Ein Beitrag sur dialektologischen Prosodieforschung. In: Moulin, C., Nubling, D. (eds.) Perspektiven einer Linguistischen Luxemburgistik - Studien zu Diachronie, pp. 29– 68 (2005) 6. Riley, E., Ljolje, A.: Automatic Generation of Detailed Pronunciation Lexicons. In: Lee, C.-H., Soong, F.K., Paliwal, K.K. (eds.) Automatic Speech and Speaker Recognition: Advanced Topics, ch. 12. Kluwer Academic Press, Dordrecht (1996) 7. Adda-Decker, M., Pellegrini, T., Bilinski, E., Adda, G.: Developments of Letzebuergesch Resources for Automatic Speech Processing and Linguistic Studies. In: Xth Proceedings of the Language Resources and Evaluation Conference, Marrakech, Morocco (2008) 8. Adda-Decker, M., Lamel, L., Snoeren, N.D.: Initializing Acoustic Phone Models of Under-Resourced Languages: A Case-Study of Luxembourgish. In: Proceedings of the 2nd SLTU Workshop, Penang, Malaysia (2010) 9. Adda, G., Adda-Decker, M., Gauvain, J.L., Lamel, L.: Text Normalization and Speech Recognition in French. In: Proceedings of the European Conference on Speech Technology, EuroSpeech, vol. 5, pp. 2711–2714 (1997) 10. Dilley, L., Pitt, M.A.: A study of regressive place assimilation in spontaneous speech and its implications for spoken word recognition. Journal of the Acoustical Society of America 122, 2340–2353 (2007) 11. Strik, H.: Pronunciation Adaptation at the Lexical Level. In: ISCA Tutorial and Research Workshop, Sophie Antipolis, France (2001) 12. Van Son, R.J.H., Pols, L.C.W.: An Acoustic Model of Communicative Eﬃciency in Consonants and Vowels Taking into Account Context Distinctiveness. In: 15th International Conference of Phonetic Sciences, Barcelona, Spain (2003) 13. Krummes, C.: Sinn si or Si si? Mobile N-Deletion in Luxembourgish. In: Papers in Linguistics from the University of Manchester: Proceedings of the 15th Postgraduate Conference in Linguistics, Manchester, England (2006)

Pronunciation and Writing Variants in Luxembourgish

81

14. Gaskell, M.G., Marlsen-Wilson, W.D.: Phonological Variation and Lexical Access. Journal of Experimental Psychology: Human Perception & Performance 22, 144– 158 (1996) 15. Goldinger, S.: Echoes of Echoes? An Episodic Theory of Lexical Access. Psychological Review 105, 251–279 (1998) 16. Adda-Decker, M., Boula de Mareuil, P., Adda, G., Lamel, L.: Investigating Syllabic Structure and their Variation in Spontaneous French. Speech Communication 46, 119–139 (2005) 17. Snoeren, N.D., Segui, J., Hallé, P.A.: A voice for the voiceless: Production and perception of voice assimilation in French. Journal of Phonetics 34, 241–268 (2006) 18. Snoeren, N.D., Gaskell, M.G., Di Betta, A.M.: The Perception of Assimilation in Newly Learned Novel Words. Journal of Experimental Psychology: Learning, Memory & Cognition 2, 542–549 (2009) 19. Strik, H., Cucchiarini, C.: Modeling Pronunciation Variation for ASR: A Survey of the Literature. Speech Communication 29, 225–246 20. Greenberg, S.: Speaking in Shorthand - A Syllabic-centric Perspective for Understanding Pronunciation Variation. Speech Communication 2, 159–176 (1999) 21. Berg, C., Weis, C.: Sociologie de l’Enseignement des Langues dans un Environnement Multilingue. Rapport National en vue de l’Elaboration du Proﬁl des Politiques Linguistiques Educatives Luxembourgeoises. Technical report (2005) 22. Schanen, F., Lulling, J.: Introduction à l’Orthographe Luxembourgeoise, http://www.cpll.lu/ortholuxs_l.html

Morpheme-Based and Factored Language Modeling for Amharic Speech Recognition Martha Yifiru Tachbelie1, Solomon Teferra Abate2, and Wolfgang Menzel1 1

Department of Informatics, University of Hamburg, Vogt-Kölln-str. 30, D-22527 Hamburg, Germany {tachbeli,menzel}@informatik.uni-hamburg.de 2 Joseph Fourier University, LIG/GETALP, F-38041 Grenoble Cedex 9 [email protected]

Abstract. This paper presents the application of morpheme-based and factored language models in an Amharic speech recognition task. Since the use of morphemes in both acoustic and language models often results in performance degradation due to a higher acoustic confusability and since it is problematic to use factored language models in standard word decoders, we applied the models in a lattice rescoring framework. Lattices of 100 best alternatives for each test sentence of the 5k development test set have been generated using a baseline speech recognizer with a word-based backoff bigram language model. The lattices have then been rescored by means of various morpheme-based and factored language models. A slight improvement in word recognition accuracy has been observed with morpheme-based language models while factored language models led to notable improvements in word recognition accuracy. Keywords: Morpheme-based language modeling, Amharic, Lattice rescoring, Factored language modeling, Speech recognition.

1 Introduction 1.1 Language Modeling Language models (LM) are fundamental to many natural language applications such as automatic speech recognition (ASR) and statistical machine translation (SMT). The most widely used kind of language models are statistical ones. They provide an estimate of the probability of a word sequence W for a given task. The probability distribution depends on the available training data and how the context has been defined [1]. Large amounts of training data are, therefore, required in statistical language modeling to ensure statistical significance [2]. Even if we have a large training corpus, there may be still many possible word sequences which will not be encountered at all, or which appear with a statistically insignificant frequency (data sparseness problem) [2]. There are even individual words that might not be encountered in the training data irrespective of its size (Out of Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 82–93, 2011. © Springer-Verlag Berlin Heidelberg 2011

Morpheme-Based and Factored Language Modeling for Amharic Speech Recognition

83

Vocabulary words problem). These problems are more serious for morphologically rich languages. Morphologically rich languages have a high vocabulary growth rate which results in a high perplexity and a large number of out of vocabulary words [3]. As a solution, sub-word units are used in language modeling [4, 5, 6, 7, 8] to improve the quality of language models and consequently the performance of the applications that use the language models. Since Amharic is a morphologically rich language, an Amharic language model suffers from the problems of data sparseness and out-of-vocabulary words. The negative effect of the Amharic morphology on language modeling has already been reported by [9], who also recommended the development of sub-word based language models for Amharic. To this end, we [10, 11] have developed various morpheme-based language models for Amharic and gained a substantial reduction in perplexity and out-of-vocabulary rate. In this regard, using sub-word units is preferable for the development of language models for Amharic. However, the Amharic sub-word language models have not been applied to any natural language application and, therefore, nothing is known whether these language models really yield an improvement in the performance of an application. In this study, we applied sub-word (morpheme) based language models to Amharic speech recognition. [12] have investigated the application of automatic word decompounding (using Harris algorithm) for automatic speech recognition of less-represented languages, specifically Amharic. In their study, the units obtained through decomposition have been used in both acoustic and language models. Word error rate reduction over the baseline word-based system has been reported using 2 hours of training data in speech recognition. However, decompounding lexical units with the same algorithm led to worse performance when more training data (35 hours) is used [13]. This can be explained by a higher acoustic confusability which is quite similar to other languages [4, 5, 6]. [13] tried to solve this problem by using other modified decompounding algorithms. Their starting algorithm is Morfessor [14] which has been modified by adding different information. They were able to achieve a word error rate reduction only when a phonetic confusion constraint was used to block the decomposition of words which would result in acoustically confusable units. Unlike [12] and [13], we used morphemes only for the language modeling component to avoid the influence of acoustic confusability on the performance of the speech recognizer. A lattice rescoring framework, as in [5] and [15], has been applied. Lattices have been generated in a single pass recognition using a bigram word-based language model since the HTK decoder, namely Hvite, does not allow to use higher order ngram models. The lattices are subsequently rescored using sub-word language models. In addition, since factored language models [15] enable us to integrate any kind of information that helps to get better probability estimates of words, we also developed factored language models that integrate various morphological information for Amharic and applied them to a speech recognition task in the same manner the sub-word language models have been applied. In our previous paper [16], factored language models that integrate either only part-of-speech or all the available features in the ngram history have been investigated. In this paper, additional factored language

84

M.Y. Tachbelie, S.T. Abate, and W. Menzel

models have been developed and applied in lattice rescoring experiment. Section 2 gives an introduction to Amharic morphology and Sect. 3 describes the baseline speech recognition system and its performance. In Sect. 4, we present the morphemebased and factored language models that we have developed and in Sect. 5 we present the lattice rescoring experiment results. Before that we give a brief introduction to factored language models. 1.2 Factored Language Modeling Factored language models (FLM) have been first introduced in [7] for incorporating various morphological information into Arabic language modeling. In an FLM a word is viewed as a bundle or vector of K parallel factors, that is, w n ≡ f n1 , f n2 ,..., f nk . The factors of a given word can be the word itself, its stem, root, pattern, or morphological class, as well as any other linguistic element into which a word can be decomposed. The goal of an FLM is, therefore, to produce a statistical model over these factors. There are two important points in the development of FLM: (1) choosing the appropriate factors which can be done based on linguistic knowledge or using a data driven technique and (2) finding the best statistical model over these factors.

Fig. 1. Possible backoff paths

Unlike normal word or morpheme-based language models, in FLM there is no obvious natural backoff order. In a trigram word based model, for instance, we backoff to a bigram if a particular trigram sequence is not observed in our corpus by dropping the most distant neighbor, and so on. However, in FLM the factors can be temporally

Morpheme-Based and Factored Language Modeling for Amharic Speech Recognition

85

equivalent and it is not obvious which factor to drop first during backoff. If we consider a quadrogram FLM and if we drop one factor at a time, we can have six possible backoff paths as it is depicted in Fig. 1 and we need to choose a path that results in a better model. Therefore, choosing a backoff path is an important decision one has to make in FLM. There are three possible strategies for deciding on a backoff path: 1) Choosing a fixed path based on linguistic or other reasonable knowledge; 2) Generalized all-child backoff where multiple backoff paths are chosen at run time; and 3) Generalized constrained-child backoff where a subset of backoff paths is chosen at run time [17]. A genetic algorithm for learning the structure of a factored language model has been developed by [18].

2 The Morphology of Amharic Amharic is one of the morphologically rich languages. It is a language spoken mainly in Ethiopia and belongs to the Semitic branch of the Afro-Asiatic super family. Amharic is related to Hebrew, Arabic and Syrian. Like other Semitic languages such as Arabic, Amharic exhibits a root-pattern morphological phenomenon. A root is a set of consonants (called radicals) which has a basic 'lexical' meaning. A pattern consists of a set of vowels which are inserted (intercalated) among the consonants of a root to form a stem. The pattern is combined with a particular prefix or suffix to create a single grammatical form [19] or another stem [20]. For example, the Amharic root sbr means 'break'. By intercalating the pattern ä_ä and attaching the suffix –ä, we get säbbärä 'he broke' which is the first form of a verb (3rd person masculine singular in past tense, as in other Semitic languages) [19]. In addition to this non-concatenative morphological feature, Amharic uses different affixes to create inflectional and derivational word forms. Some adverbs can be derived from adjectives. Nouns are derived from other basic nouns, adjectives, stems, roots, and the infinitive form of a verb by affixation and intercalation. For example, from the noun lIğğ 'child' another noun lIğnät 'childhood'; from the adjective däg 'generous' the noun dägnät 'generosity'; from the stem sInIf, the noun sInIfna 'laziness'; from root qld, the noun qäld 'joke'; and from infinitive verb mäsIbär 'to break' the noun mäsIbäriya 'an instrument used for breaking' can be derived. Case, number, definiteness, and gender marker affixes inflect nouns. Adjectives are derived from nouns, stems or verbal roots by adding a prefix or a suffix. For example, it is possible to derive dIngayama 'stony' from the noun dIngay 'stone'; zIngu 'forgetful' from the stem zIng; sänäf 'lazy' from the root snf by suffixation and intercalation. Adjectives can also be formed through compounding. For instance, hodäsäfi 'tolerant, patient', is derived by compounding the noun hod 'stomach' and the adjective säfi 'wide'. Like nouns, adjectives are inflected for gender, number, and case [20]. Unlike the other word categories such as noun and adjectives, the derivation of verbs from other parts of speech is not common. The conversion of a root to a basic verb stem requires both intercalation and affixation. For instance, from the root gdl 'kill' we obtain the perfective verb stem gäddäl- by intercalating the pattern ä_ä. From this perfective stem, it is possible to derive a passive (tägäddäl-) and a causative stem

86

M.Y. Tachbelie, S.T. Abate, and W. Menzel

(asgäddäl-) using the prefixes tä- and as-, respectively. Other verb forms are also derived from roots in a similar fashion. Verbs are inflected for person, gender, number, aspect, tense and mood [20]. Table 1 shows the inflection of perfective and imperfective verbs for person, gender, number. Other elements like negative marker and prepositions also inflect verbs in Amharic. A single Amharic word may, therefore, represent a sentence in English. For instance, amt'Iččewalähu is equivalent with the sentence 'I have brought it'. Table 1. Inflection for person, gender and number

Person 1st 1st plural 2nd masculine feminine polite plural 3rd masculine feminine polite plural

Perfective säbbär-ku/hu säbbär-n

Imperfective ?-säbr ?-n-säbr

säbbär-h/k säbbär-š säbbär-u säbbär-aččhu

t-säbr t-säbr-i t-säbr-u t-säbr-u

säbbär-ä säbbär-äčč säbbär-u säbbär-u

y-säbr t-säbr y-säbr-u y-säbr-u

From the above brief description of Amharic morphology, it can be seen that Amharic is a morphologically rich language. It is this feature that makes the OOV words and data sparseness problems more serious in Amharic language modeling.

3 The Baseline Speech Recognition System 3.1 Speech and Text Corpus The speech corpus used to develop the speech recognition system is a read speech corpus developed by [21]. It contains 20 hours of training speech collected from 100 speakers who read a total of 10,850 sentences (28,666 tokens). Compared to other speech corpora that contain hundreds of hours of speech data for training, for example, the British National Corpus (1,500 hours of speech), our models obviously suffer from a lack of training data. Moreover, the pronunciation dictionary has not been developed by linguistic experts. Encoding of the pronunciation of a corpus can range from very simple and achievable with automatic procedures to very complex and time-consuming that requires manual work with high linguistic expertise. The Amharic speech corpus has been encoded by means of a simple procedure that takes advantage of the orthographic representation which is a consonant vowel syllable. Although the corpus includes four different test sets (5k and 20k both for development and evaluation), for the purpose of the current investigation we have generated

Morpheme-Based and Factored Language Modeling for Amharic Speech Recognition

87

the lattices only for the 5k development test set, which includes 360 sentences read by 20 speakers. The text corpus used to train the baseline backoff bigram language model consists of 77,844 sentences (868,929 tokens or 108,523 types). 3.2 The Acoustic and Language Models The acoustic model is a set of intra-word triphone HMM models with 3 emitting states and 12 Gaussian mixtures that resulted in a total of 33,702 physically saved Gaussian mixtures. The states of these models are tied, using decision-tree based state-clustering that reduced the number of triphone models from 5,092 logical models to 4,099 physical ones. The baseline language model is a closed vocabulary (for 5k) backoff bigram model developed using the HTK toolkit. The absolute discounting method has been used to reserve some probabilities for unseen bigrams where the discounting factor, D, has been set to 0.5, which is the default value in the HLStats module. The perplexity of this language model on a test set that consists of 727 sentences (8,337 tokens) is 91.28. 3.3 Performance of the Baseline System We generated lattices from the 100 best alternatives for each sentence of the 5k development test set using the HTK tool and decoded the best path transcriptions for each sentence using the lattice processing tool of SRILM [22]. Word recognition accuracy of this system was 91.67% with a language model scale of 15.0 and a word insertion penalty of 6.0. The better performance (compared to the one reported by [9], 90.94%, using the same models and on the same test set) is due to the tuning of the language model and word insertion penalty scales.

4 Morpheme-Based and Factored Language Models 4.1 Morpheme-Based Language Models We have trained several sub-word based and factored language models for Amharic on the same data that has been used to develop the baseline language model. Both statistical and linguistic morphs have been considered as units in language modeling. Since there is no morphological analyzer (for Amharic) specifically designed for our purpose, we also used the language independent, unsupervised morphology learning algorithm, Morfessor [14] to obtain the statistical morphs. However, this algorithm segments a word only into a sequence of morphemes, and can not extract the root and pattern morphemes of Amharic. On the other hand, a good and complete segmentation of words into morphs leads to a better language model [11]. Therefore, we also investigated the performance of linguistic morpheme-based language models for speech recognition. The linguistic morphs are obtained according to a manually segmented collection of 72,428 word types [11]. That is, we substituted each word in the corpus with its segmentation if the word is in the manually segmented word collection. Otherwise, the word is left unsegmented. Due to the limited coverage of the collection, a substantial share of words (12.3%) in our training data could not be segmented at all.

88

M.Y. Tachbelie, S.T. Abate, and W. Menzel

We developed various sub-word language models using the statistical and linguistic morphs as units in language modeling. We tried to develop ngram language models of order two to four. In all cases we used the SRILM toolkit to train the language models. We smoothed the language models using modified Kneser-Ney smoothing which is known for its state-of-the-art performance unless it became impossible to use it because of zero count of counts. Table 2 presents the perplexity of the various morpheme-based language models on the segmented version of the test set that has been used to test the baseline bigram language model. As the number of morphs in the linguistically and statistically segmented test sets is different, we can not directly compare the perplexities of statistical and linguistic morpheme-based language models. Although the statistical morpheme-based models have high perplexities, they seem better than the linguistic morpheme-based ones if compared with respect to the probability they assign to the test set. Table 2. Perplexity of morpheme-based language models

Language models Linguistic morph bigram Linguistic morph trigram Linguistic morph quadrogram Statistical morph bigram Statistical morph trigram Statistical morph quadrogram

Perplexity 36.55 23.09 18.39 114.92 71.61 64.22

Logprob -34654 -30232 -28038 -31800 -28630 -27899

4.2 Amharic Factored Language Models The manually segmented data has also been used to obtain a factored version of the corpus. Each word is considered a bundle of features including the word itself, partof-speech (POS) tag of the word, prefix, root, pattern and suffix. Each feature in the feature vector is separated by a colon (:) and consists of a tag-value pair. In our case the tags are: W for word, POS for Part-of-Speech, PR for prefix, R for root, PA for pattern and SU for suffix. Although, in Amharic words can have more than one prefix and suffix, we considered each word as having zero or one prefix and/or suffix by concatenating a sequence of affixes into a single unit. A given tag-value pair may be missing from the feature bundle. In this case, the tag takes a special value 'null'. After representing each word as a bundle of features, the resulting corpus has then been used to train various kinds of factored language models1. All the factored language models have been tested on the factored version of the test set used to test the baseline language model. We have developed a factored language model that considered all the available factors (word, POS, prefix, root, pattern and suffix) as histories and that uses a fixed 1

This experiment adopted a similar procedure as the one reported in [16], but the data was modified. Words which are not derived from root consonants have got a null value for the R tag in [16] while, in the experiment reported here, these words assigned the stem of a word for R (root) feature. Hence, the resulting figures are different.

Morpheme-Based and Factored Language Modeling for Amharic Speech Recognition

89

backoff path by dropping suffix first, then pattern, and so on. Since it is difficult to determine which factor combination and which backoff path would result in a robust model yielding an improvement of speech recognition, we applied the genetic algorithm [18] again, to find the optimal one. The best model is the one that uses four factors (word, prefix, root and pattern) as histories and combines generalized all-child and constrained-child backoff. We applied the two best (in terms of perplexity) models that differ only in the backoff path to the speech recognition task. The perplexities and log-probabilities of the factored language models are given in Table 3. The FLM with fixed backoff is the best model compared to the others. Table 3. Perplexity of factored language models

Language models2 FLM with fixed backoff 1st Best factor combination 2nd Best factor combination

Perplexity 55.04 71.89 118.62

Logprob -15777 -16828 -18800

Other factored language models that take one word feature (besides the words) in the ngram history have been developed. The additional features used are part-ofspeech (POS), prefix (PR), root (R), pattern (PA) and suffix (SU). The models developed contain two ( w n | w n −1 X n −1 ) 3 and four ( w n | w n − 2 X n − 2 w n −1 X n − 1 ) parents where the backoff paths have been defined by dropping w n −1 and then X n −1 for the former models and by dropping X n − 2 , then w n − 2 , then w n − 1 , and finally X n −1 for the latter ones. The perplexity and log-probability of these models are presented in Table 4. The models with four parents are almost similar in perplexity and probability. Among the models with two parents, the one that takes root (R) as an additional information is the best one while the model that takes pattern is the worst. Table 4. Perplexity of other factored language models

Language models W|W1,POS1 W|W1,PR1 W|W1,R1 W|W1,PA1 W|W1,SU1 W|W2,POS2,W1,POS1 W|W2,PR2,W1,PR1 W|W2,R2,W1,R1 W|W2,PA2,W1,PA1 W|W2,SU2,W1,SU1 2

3

Perplexity 64.11 65.02 62.14 66.50 65.43 10.61 10.67 10.36 10.89 10.70

Logprob -16377 -16433 -16255 -16522 -16458 -9298 -9322 -9204 -9401 -9330

The language models are smoothed with Witten-Bell smoothing since it was not possible to use the Kneser-Ney smoothing technique due to the existence of zero count of count. X is a place holder which can represent one of the extra word feature.

90

M.Y. Tachbelie, S.T. Abate, and W. Menzel

5 Lattice Rescoring Experiment 5.1 Lattice Rescoring with Morpheme-Based Language Models The lattices generated as indicated in Section 3 have been rescored using the various morpheme-based language models and decoded to find the best path. An improvement in word recognition accuracy (WRA) has been observed (see Table 5). All morph-based models brought a slight improvement in WRA. However, the linguistic morphs contribute more to the performance improvement than the statistical morphs (an absolute 0.25% increase in accuracy with the linguistic morph trigram model). Using higher order ngram brings only a slight improvement in performance, from 91.77 to 91.82 and then to 91.85 as a result of using trigram and quadrogram language models, respectively. Table 5. WRA improvement with morpheme-based language models

Language Models Used Baseline word-based (BL) BL + Statistical morph bigram BL + Statistical morph trigram BL + Statistical morph quadrogram BL + Linguistic morph bigram BL + Linguistic morph trigram BL + Linguistic morph quadrogram

Word recognition accuracy in % 91.67 91.77 91.82 91.85 91.87 91.92 91.89

5.2 Lattice Rescoring with Factored Language Models Since it is problematic to use factored language models in standard word decoders, we substituted each word in the lattice with its factored representation. A word bigram model that is equivalent to the baseline word bigram language model has been trained on the factored data and used as a baseline for factored representations. This language model has a perplexity of 63.59. The best path transcription decoded using this language model has a WRA of 91.60%, which is slightly lower than the performance of the normal baseline speech recognition system (91.67%). This might be due to the smoothing technique applied in the development of the language models. Although absolute discounting with the same discounting factor has been applied to both bigram models, the unigram models have been discounted differently. While in the baseline word based language model the unigram models have not been discounted at all, in the equivalent factored model the unigrams have been discounted using GoodTuring discounting technique which is the default discounting technique in SRILM. The various factored language models (described in 4.2) have been used to rescore the lattices and most of them brought a considerable improvement in WRA. The first best factored language model learned by the genetic algorithm outperformed the second best model and the one with fixed backoff (see Table 6). The factored language model with fixed backoff did not contribute much to the WRA improvement although it is the best model in terms of the perplexity and the probability it assigns to the test set as it is shown in Table 3.

Morpheme-Based and Factored Language Modeling for Amharic Speech Recognition

91

Table 6. WRA improvement with factored language models

Language models used Baseline word bigram (FBL) FBL + FLM with fixed backoff FBL + 1st Best factor combination FBL + 2nd Best factor combination

Word recognition accuracy in % 91.60 91.99 92.82 92.55

All the factored language models that integrate an additional word feature in the ngram history brought an improvement in WRA as shown in Table 7. Table 7. WRA improvement with other factored language models

Language models used Baseline word bigram (FBL) FBL + W|W1,POS1 FBL + W|W1,PR1 FBL + W|W1,R1 FBL + W|W1,PA1 FBL + W|W1,SU1 FBL + W|W2,POS2,W1,POS1 FBL + W|W2,PR2,W1,PR1 FBL + W|W2,R2,W1,R1 FBL + W|W2,PA2,W1,PA1 FBL + W|W2,SU2,W1,SU1

Word recognition accuracy in % 91.60 92.87 92.85 92.75 92.77 92.58 93.60 93.82 93.65 93.68 93.53

Among the models with two parents, the one that takes POS contributed more to the performance improvement. Although the model in which the probability estimation was conditioned on the previous word and its root has a lower perplexity and assigned higher probability to the test set compared to the others, this model did not achieve the highest improvement in WRA. Models with four parents did not fare better than the ones with two parents, if the maximal ngram order to be used for transition weight assignment was set to 2. However, when trigrams are used, all the models contributed a notable improvement where the maximum WRA improvement (absolute 2.22%) has been achieved with a model that uses prefix as an additional information.

6 Conclusion Several language models (statistical and linguistic morpheme-based as well as FLMs) have been applied to an Amharic speech recognition task in a lattice rescoring framework. Lattices consisting of the 100 best alternatives for each test sentence have been generated and subsequently rescored with various language models. A considerable improvement in WRA has been observed as a result of using factored language models. The morpheme-based language models brought a slight improvement in WRA.

92

M.Y. Tachbelie, S.T. Abate, and W. Menzel

The linguistic morph-based language models contributed more to the performance improvement than the statistical morph-based ones even though a substantial share of the words have been left unsegmented. Therefore, we conclude that morpheme-based language models and factored language models are better suited for Amharic speech recognition than word only models. Moreover, integrating morphological information to word-based models is a promising direction for performance improvement when the problem of OOV words is not an issue.

References 1. Junqua, J.-C., Haton, J.-P.: Robustness in Automatic Speech Recognition: Fundamentals and Applications. Kluwer Academic, London (1996) 2. Young, S., Evermann, G., Gales, M., Hain, T., Kershaw, D., Liu, X., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev, V., Woodland, P.: The HTK Book. Cambridge University Engineering Department (2006) 3. Vergyri, D., Kirchhoff, K., Duh, K., Stolcke, A.: Morphology-Based Language Modeling for Arabic Speech Recognition. In: ICSLP 2004, pp. 2245–2248 (2004) 4. Geutner, P.: Using Morphology towards Better Large-Vocabulary Speech Recognition Systems. IEEE International on Acoustics, Speech and Signal Processing I, 445–448 (1995) 5. Whittaker, E., Woodland, P.: Particle-Based Language Modeling. In: Proceeding of International Conference on Spoken Language Processing, pp. 170–173 (2000) 6. Byrne, W., Hajič, J., Ircing, P., Jelinek, F., Khudanpur, S., Krebc, P., Psutka, J.: On Large Vocabulary Continuous Speech Recognition of Highly Inflectional Language - Czech. In: Proceeding of the European Conference on Speech Communication and Technology, pp. 487–489 (2001) 7. Kirchhoff, K., Bilmes, J., Henderson, J., Schwartz, R., Noamany, M., Schone, P., Ji, G., Das, S., Egan, M., He, F., Vergyri, D., Liu, D., Duta, N.: Novel Speech Recognition Models for Arabic. In: Johns-Hopkins University Summer Research Workshop (2002) 8. Hirsimäki, T., Creutz, M., Siivola, V., Kurimo, M.: Morphologically Motivated Language Models in Speech Recognition. In: Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning, pp. 121–126 (2005) 9. Abate, S. T.: Automatic Speech Recognition for Amharic. University of Hamburg (2006) 10. Tachbelie, M.Y., Menzel, W.: Sub-Word Based Language Modeling for Amharic. In: Proceedings of International Conference on Recent Advances in Natural Language Processing, pp. 564–571 (2007) 11. Tachbelie, M.Y., Menzel, W.: Morpheme-Based Language Modeling for Inflectional Language - Amharic. In: Nicolov, N., Angelova, G., Mitkov, R. (eds.) Recent Advances in Natural Language Processing Selected Papers from RANLP 2007, vol. V, pp. 301–310. John Benjamin’s Publishing, Amsterdam (2009) 12. Pellegrini, T., Lamel, L.: Investigating Automatic Decomposition for ASR in Less Represented Languages. In: Proceedings of INTERSPEECH 2006 (2006) 13. Pellegrini, T., Lamel, L.: Using Phonetic Features in Unsupervised Word Decompounding for ASR with Application to A Less-Represented Language. In: Proceedings of INTERSPEECH 2007, pp. 1797–1800 (2007) 14. Creutz, M., Lagus, K.: Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora Using Morfessor 1.1. A81, Neural Networks Research Center, Helsinki University of Technology (2005)

Morpheme-Based and Factored Language Modeling for Amharic Speech Recognition

93

15. Kirchhoff, K., Bilmes, J., Das, S., Duta, N., Egan, M., Ji, G., He, F., Henderson, J., Liu, D., Noamany, M., Schone, P., Schwartz, R., Vergyri, D.: Novel Approaches to Arabic Speech Recognition: Report from the 2002 Johns-Hopkins Summer Workshop. In: Proceedings of International Conference on Acoustics, Speech, and Signal Processing, vol. 1, pp. 344–347 (2003) 16. Tachbelie, M.Y., Abate, S.T., Menzel, W.: Morpheme-Based Language Modeling for Amharic Speech Recognition. In: Proceedings of the 4th Language and Technology Conference, pp. 114–118 (2009) 17. Kirchhoff, K., Bilmes, J., Duh, K.: Factored Language Models - a Tutorial. Dept. of Electrical Eng., Univ. of Washington (2008) 18. Duh, K., Kirchhoff, K.: Automatic Learning of Language Model Structure. In: Proceeding of International Conference on Computational Linguistics (2004) 19. Bender, M.L., Bowen, J.D., Cooper, R.L., Ferguson, C.A.: Languages in Ethiopia. Oxford Univ. Press, London (1976) 20. Yimam, B.: yäamarIŋa säwasäw. 2nd. ed. EMPDE, Addis Ababa (2007) 21. Abate, S.T., Menzel, W., Tafila, B.: An Amharic Speech Corpus for Large Vocabulary Continuous Speech Recognition. In: Proceedings of 9th European Conference on Speech Communication and Technology (2005) 22. Stolcke, A.: SRILM - an Extensible Language Modeling Toolkit. In: Proceedings of International Conference on Spoken Language Processing, vol. II, pp. 901–904 (2002)

The Corpus Analysis Toolkit - Analysing Multilevel Annotations Stephen Wilson and Julie Carson-Berndsen School of Computer Science and Informatics, University College Dublin {stephen.m.wilson,julie.berndsen}@ucd.ie http://muster.ucd.ie

Abstract. This paper considers a number of issues surrounding current annotation science and corpus analysis and presents a bespoke suite of software, the Corpus Analysis Toolkit, for processing and analysing multilevel annotations of time-aligned linguistic data. The toolkit provides a variety of specialised tools for performing temporal analysis of annotated linguistic data. The toolkit is feature-set and corpus independent and oﬀers support for a number of commonly used annotations formats. Keywords: Annotation Science, Speech Technology, Linguistics.

1

Introduction

Much of the progress that has been achieved by natural language processing systems in recent decades can be attributed to the widespread adoption of statistical techniques by the research community and their application to problems such as speech recognition, speech synthesis, and information retrieval. Statistical models, while successful, are resource-hungry as they require signiﬁcant amounts of training data in order to be useful. While the automatic acquisition of training resources is currently an active topic of research [19], they have been traditionally sourced from corpora of manually annotated data, (e.g. [9]). In order to maximise the eﬃcient deployment of corpus-based resources to greatest beneﬁt, tools that permit the annotated data within the corpus to be analysed with respect to each other are required. This paper presents a suite of bespoke software tools, collectively referred to as the Corpus Analysis Toolkit, that allows a wide-range of diagnostic and analytical tasks to be carried out on corpora of time-aligned linguistic data. An advantage of having a single framework that can perform the same type of analyses on diﬀerent sets of annotations is that the output of automatic transcription or recognition systems can be directly compared with each other, thereby aiding inter-system evaluation. The ability to perform analyses on diﬀerent sets of time-aligned data also has applications for the study of inter-modal synchrony and asynchrony, that is the temporal relationship between the movements of the outwardly visible articulators and the acoustic realisations of those movements. The following sections discuss the process of linguistic annotation in general terms and highlight some of the more common approaches in the ﬁeld. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 94–104, 2011. c Springer-Verlag Berlin Heidelberg 2011

The Corpus Analysis Toolkit

2

95

Linguistic Annotation

Broadly speaking linguistic annotation is the addition of some layer of information to some underlying speech data. This may take the form of labelling text with respect to syntactic categories or semantic glosses, labelling speech signal with respect to phonemic categories or pitch accents, or annotating video with respect to manual or facial gestures. Frequently linguistic annotations pertaining to speech are time-aligned, meaning that in addition to a label describing some particular linguistic phenomenon, the annotation also contains temporal information that associates the label with a speciﬁc point or portion of an underlying speech signal. For example, Figure 1 below shows a portion of a spectrogram representing an utterance of the word two, an accompanying timeline, and a corresponding annotation indicating that the interval 2.696ms to 2.819ms on the timeline corresponds to /t/, the ﬁrst phoneme of the word. Time-aligned annotations are particularly useful for training statistical models, as they directly relate signal events to labelled time-stamps, from which the models can then build probabilistic relations.

Fig. 1. Spectrogram showing time-aligned annotation

3

Multilevel Annotations

There are a number of autonomous (though related) physiological processes involved in the production of speech in humans. These include, but are not limited to, vocal cord vibration, tongue movements, relative positioning of the lips and teeth, lip-rounding etc, all of which serve to modulate the acoustic energy wave produced during speech resulting in the distinct phonemic units which are

96

S. Wilson and J. Carson-Berndsen

then perceived by listeners. The understanding that speech is composed of these parallel but autonomous processes has underpinned some of the most important shifts in phonological theory of the last number of decades, e.g. [8], [5], [17], and [2], as well as having important implications for computational phonology [7]. Processing the speech signal in terms of independent feature streams has also been adopted by portions of the automatic speech recognition community [11] and has also extended to those working with audio-visual or multimodal speech representations [20], [21],[16]. Given the prevalence of multilevel annotations across a wide number of research areas in computational linguistics and speech technology, a framework for studying and examining the relationships between annotated data on diﬀerent tiers is required. The Corpus Analysis Toolkit provides such a framework, and contains a variety of tools for analysing temporal relations between the labelled components in a corpus. The toolkit is featureset and corpus independent, and can be fully parameterised to only examine relations between those labels as speciﬁed by the user.

4

Tiers, Intervals, and Points

Considering speech to be composed of a number of parallel processes requires that any corresponding annotations to have a similar composition. Such

Fig. 2. Multilevel Annotation

The Corpus Analysis Toolkit

97

annotations are frequently referred as multilevel or multitiered annotations. A tier in this context refers to an autonomous descriptive category. Figure 2 shows an example of a multilevel annotation, with independent tiers for ﬁve aspects of visible speech production: lip-opening, spreading, rounding, tongue visibility, and dental visibility. Tiers can be composed of either points or intervals. Points are generally marks on a timeline with no duration that represent the presence of some event, with an associated label. Intervals are labelled units on a tier that have a temporal duration, and consequently have clear and distinct start and endpoints. The Corpus Analysis Toolkit can process annotations comprising both point and interval tiers, and considers points on a tier to be intervals where the start- and end-points are the same. Multilevel annotations can either be acquired semiautomatically [12] or through manual annotation [21]. Regardless of the means through which they are acquired, it is useful to have tools that can examine these annotations with respect to a number of diﬀerent analytical conditions. The following sections outline a number of these.

5

Inter-tier Analysis

The individual tiers in a multilevel annotation describe processes that are autonomous, that is, they may act independently of one another and an event on one tier does not necessarily inﬂuence events on other tiers. Although events on separate tiers are autonomous, it is useful to consider them in terms of intertier overlap relations, that is where a portion of a labelled interval on one tier temporally overlaps with a portion of another labelled interval on a separate tier. For example, in Figure 3, the property spread (highlighted) overlaps with open on the Lips tier, non-round on the Rounding tier, not-visible on the Tongue tier and visible on the Dental tier. Such inter-tier analysis is useful from many perspectives. For example, when large-scale corpora are analysed in this manner relationships and dependencies between diﬀerent properties can be studied, and potential redundancies uncovered. Additionally, the type of inter-tier analysis that can be performed with the Corpus Analysis Toolkit could be readily applied to the output of a speech recognition system based on the recognition of streams of phonological features in a signal, e.g. [1], and form part of a system evaluation and optimisation process.

6

Inter-annotation Analysis

Corpus analysis typically requires multiple annotations from a number of different speakers to be examined and investigated in accordance with a set of diagnostic criteria. This task is sometimes referred to as inter-annotation analysis and it is greatly aided by a uniﬁed workspace that permits corpus annotations to be examined as a single systematic collection. The Corpus Analysis Toolkit permits the simultaneous processing of multiple speaker annotations and can

98

S. Wilson and J. Carson-Berndsen

Fig. 3. Multilevel annotation showing temporal overlap

be parameterised to examine temporal relations between sets of user-speciﬁed values across an entire corpus, or indeed across a number of separately labelled corpora.

7 7.1

The Corpus Analysis Toolkit An Integrated Analytical Framework

The previous sections and subsections highlighted some of the issues inherent in automated corpus analysis and underlined the need for tools that could be used with diﬀerent annotations created with diﬀerent applications and represented in diﬀerent formats. The Corpus Analysis Toolkit was originally developed for use with a speciﬁc corpus of audio-visual speech data [21]. However, one of the principals underpinning the design and development of the toolkit was that it should be, in as far as is possible, independent of data-set and annotation format. As a result, the toolkit consists of a large repository of analytical and processing software that performs a number of diﬀerent tasks with respect to time-aligned corpus data and so represents an integrated framework for performing corpus analysis.

The Corpus Analysis Toolkit

7.2

99

Supported Formats

Currently the toolbox supports Praat TextGrids, [4], and the Eudico Annotation Format, [6]. Praat is the de facto standard annotation environment for performing phonetic annotation by computer, while the Eudico Annotation Format is an emerging standard used by Elan [6], an application for carrying out annotation of multimedia and video ﬁles. File support in this context means that the toolkit already possesses mechanisms that permit annotations that result from these applications to be automatically mapped to the internal representation used by the tools. In theory, any corpus of time-aligned data can be processed and analysed by the toolkit, provided it can ﬁrst be transformed to this internal format, (a trivial matter). The internal representation is discussed in the following section. 7.3

Internal Representation

The Corpus Analysis Toolkit makes use of an internal representation, into which all annotations are transformed before any processing takes place. Using a single internal representational format permits inter-annotation analysis to be carried out, mentioned above, which is particularly useful for audio-visual corpus analysis, where it is common for annotations pertaining to diﬀerent modalities but referring to the same speech data, to be annotated using separate tools, which results in quite separate formats. The internal representation is straightforward, and simply requires that each labelled interval or point in an annotation must be represented using the following rubric: startpoint endpoint label where startpoint and endpoint are points on the timeline indicating when an interval starts and ends, (these values will be the same for so-called point-valued tiers mentioned earlier). Label simply refers to value assigned to the interval or point by the person or machine performing the annotation. It should be clear, that given the simplicity of the internal representation of the toolkit, it ought to be a relatively trivial matter to transform annotations from formats not currently supported so that they can be analysed using the kit. 7.4

Toolkit Inventory

The toolkit was designed speciﬁcally for use with multitiered or multilevel annotations that are time-aligned to some signal. Many of the most useful types of analyses that can be applied to such corpora involve the examination of individual annotated events in terms of their temporal relationships to each other, speciﬁcally in terms of temporal overlap and precedence relations. Detailed examples of these types of analyses are provided in Section 4. What follows is a inventory of some of the most important tools of the Corpus Analysis Toolkit, with a short description of the function of each. The Corpus Format Module. This module transforms Praat TextGrids, Eudico Annotation Files into the internal representation used by the toolkit, as outlined in section 3.3 above.

100

S. Wilson and J. Carson-Berndsen

The Corpus Data Module. Generates corpus meta-data, including but not limited to exact numbers of user-speciﬁed labelled intervals in the corpus, total number of labelled intervals, average duration of speciﬁc intervals, etc. All output is the result of user-formulated queries. The Temporal Overlap Investigation Module. Annotated intervals temporally overlap within the corpus. The module can operate on single speakers, groups of speakers, or the entire corpus. In addition to user-speciﬁed requests, the module can also simply catalogue and output the entire number of temporal overlaps in the corpus. The Co-occurrence Identification Module. This fully parameterisable module oﬀers a complementary analysis to the module described in 3.4.3, and computes the proportion of co-occurrence between pairs of annotated units as speciﬁed by the user. The proportion of co-occurrence is a diagnostic metric introduced by Wilson [21] and deﬁned as follows: The proportion of co-occurrence between an annotated interval A occurring on a tier T1 and an interval B occurring on a separate tier T2, is deﬁned as the number of times A overlaps with B expressed as a percentage of the total number of times A overlaps with any interval on T2. The Temporal Inclusion Investigation Module. This module allows for the investigation of a special case of temporal overlap, that of temporal inclusion. An interval on one tier is said to temporally include a separate interval on a diﬀerent tier, if both intervals overlap and the duration of the second interval is completely spanned by the duration of the ﬁrst. The Multitiered Representation Extractor. This module is designed to extract the types of multilevel annotations outlined in Section 2. For a corpus annotated with respect to phonemes and phonological features, or perhaps phonemes and outwardly visible articulatory gestures, this module could be used to automatically extract a multitiered representation for each annotated phoneme in the data.

8

Analysing a Corpus

The Corpus Analysis Toolkit has been used in the analysis of a bimodal speech corpus, the results of which were reported in Wilson [21]. It is beyond the scope of this paper to present the results of the analyses in their entirety. Instead, a selection has been chosen to demonstrate the usefulness of the toolkit. 8.1

Corpus Overview

The bimodal speech corpus resulted from the manual annotation of audio-visual speech data at a level of granularity not seen before. The corpus was organised along the following three dimensions:

The Corpus Analysis Toolkit

101

1. Outwardly visible articulator gestures 2. Phonemes 3. Syllables The phoneme and syllable annotations were carried out using Praat, the video annotations using Elan. The video annotations consisted of ﬁve distinct gestural tiers pertaining to: lip-rounding, lip-spreading, opening of the lip aperture, tongue position and visibility, and visibility of the teeth. More detailed discussion of the linguistic motivation behind the choice of the gestural tiers can be found in [20] and [21]. The underlying audio-visual data for the corpus was supplied by the Clemson University Audio-visual Experiments database (CUAVE), [15]. As the annotations for both the audio and visual modalities are in diﬀerent formats (Praat TextGrid, Eudico Annotation File), the Corpus Format Module ﬁrst transforms them into the common internal representation used by the toolkit, so that inter-annotation analysis may be carried out. 8.2

Acquiring General Corpus Information

As sketched brieﬂy above the Corpus Data Module outputs meta-data about the corpus and can provide information regarding the total number of intervals of a speciﬁc label are contained within the entire collection, or else within the annotations pertaining to a speciﬁc speaker. It is important to note that the module is independent of the label-set used in the annotations. Users formulate queries upon which the module acts. Table 1 shows a subset of some of the corpus data generated for and presented in Wilson [21]. Table 1. Showing General Corpus Information #phoneme intervals #unique phonemes #instances of /o/ Avg. duration of /o/ 4821

8.3

22

283

0.201secs

Interval-Based Corpus Analysis

The Temporal Overlap Identiﬁcation Module allows corpus analysis to be carried out on an interval-wise basis. The module can be parameterised so that the user speciﬁes two labelled intervals on diﬀerent tiers. Recall that the analysis upon which this section is based was carried out on a bimodal corpus that had been annotated with respect to outwardly visible articulator gestures, including gestures pertaining to lip-rounding. By supplying round, a label on the liprounding tier, and /o/, a label on the phoneme tier, the following information is gained. This type of analysis can be performed for individual phonemes and individual gestures in the corpus, or the module can be parameterised so as to operate on sets of annotated intervals in tandem.

102

8.4

S. Wilson and J. Carson-Berndsen

Investigation of Temporal Inclusion

The ability to determine if two (or more) intervals on separate tiers overlap in time is undoubtedly useful. However, what is at times equally helpful is to investigate whether any of a set of overlapping intervals is temporally included in another. Table 2. Temporal Inclusion Analysis #tokens of /o/ that #tokens of /o/ that #tokens round that overlap round temporally include temporally include round /o/ 194

15

74

The sample analyses shown in the above table indicate that it is far more common for a round gesture as observed in the visual domain to temporally include the phoneme /o/, which gives an insight into gestural phasing relations and inter-modal asynchrony. 8.5

Extracting Multilevel Representations

The module was used to extract gesture-based multilevel representations for every annotated phoneme in the corpus, which resulted in precisely 4821 representations. The module was then parameterised to analyse each of these representation and compute the smallest set of unique multilevel representations required to describe every phoneme in the corpus, reducing the number from 4821 to 665. This led to additional analyses, the scope of which is beyond this paper, but which have been discussed fully in [21]. However, the ability to extract detailed multilevel representations from large-scale corpora and reduce these to a set of unique representational types is a functionality that may prove useful to researchers across a number of diﬀerent areas.

9

The Toolkit and Emerging Standards

One of the most interesting areas of current active research in corpus linguistics and annotation science is the question of standards as they pertain to linguistic resources and tools. Scientists are increasingly aware of the need for data, tools, and resources that have been developed or acquired independently to be able to be used by others for additional purposes. For the inter- operability of such resources to become an achievable goal, a set of standards must be agreed upon, to which resources must then adhere. The popularity of XML as a data exchange format has led to easier ways with which data can be transferred between applications and platforms. However, to be truly successful, any attempts at resource inter-operability must go beyond simple data exchange and attempt

The Corpus Analysis Toolkit

103

to standardise not only the type of data categories that are being annotated, but also the representational formats used, as well as the tools required to process them. Current work in this area includes that of [10], with other work including that of [13], and [3]. The Corpus Analysis Toolkit has a role to play in this regard, as it already oﬀers a generic and corpus independent framework for the temporal analysis of annotated data. It provides support for the current de facto standards for phonetic and multimodal annotations and can be readily extended to cater for others. As work on the inter-operability of linguistic data increases and additional standards begin to emerge, the generic and feature-independent tools of the kit may prove to be a valuable analytical resource.

10

Future Work

Areas for future work include incorporating additional support for other annotation format, including Anvil [14], another popular tool for performing multimodal annotations, and potentially formats used by the EXMARaLDA discourse analysis framework, [18]. It is also intended to increase the types of analysis that the toolkit can perform. Finally, work is ongoing on preparing the toolkit for dissemination to the wider research community, making it freely available for download under public licence.

11

Concluding Remarks

This paper has outlined some of the general issues that arise when one considers the task of automatic corpus analysis, including the problem of standardised representations for annotated data, issues with inter-annotation analysis, and the need for data-set independent tools that work across platforms and annotation formats. As an attempt to address some of these challenges, the Corpus Analysis Toolkit was presented, a generic suite of bespoke analytical software designed for use with corpora of time-aligned data. The toolkit was speciﬁcally designed to be format and data-set independent. A subset of tools was discussed and the usefulness of the kit demonstrated with examples drawn from the analysis of a bimodal speech corpus.

References 1. Aioanei, D.: YASPER: A Knowledge-based and Data-driven Speech Recognition Framework. PhD Thesis, University College Dublin (2008) 2. Bird, S., Klein, E.: Phonological Events. Journal of Linguistics 26, 33–56 (1990) 3. Bird, S., Liberman, M.: A Formal Framework for Linguistic Annotation. Speech Communication 33, 23–60 (2001) 4. Boersma, P., Weenik, D.: A System for Doing Phonetics by Computer. Glot International 5, 9–10 (2001)

104

S. Wilson and J. Carson-Berndsen

5. Browman, C., Goldstein, L.: Towards an Articulatory Phonology. Phonology Yearbook 2, 219–252 (1986) 6. Brugman, H., Russell, A., Broeder, D., Wittenburg, P.: Eudico - Annotation and Exploitation of Multimedia Corpora over the Internet (2000) 7. Carson-Berndsen, J.: Time Map Phonology 5, Text Speech and Language Technology. Kluwer, Dordrecht (1998) 8. Goldsmith, J.: Autosegmental Phonology. PhD Thesis, MIT, Boston, USA (1976) 9. Greenberg, S.: Speaking in shorthand a syllable-centric perspective for pronunciation variation. Speech Communication 29, 159–176 (1999) 10. Ide, N., Romary, L., de la Clergerie, E.: International Standard for a Linguistic Annotation Framework. CoRR abs/0707.3269 (2007) 11. Kanokphara, S., Carson-Berndsen, J.: Better HMM-Based Articulatory Feature Extraction with Context-Dependent Models. FLAIR (2005) 12. Kelly, R.: Learning Multitape Finite-state Machines from Multilevel Annotations. PhD Thesis, University College Dublin (2005) 13. Kelly, R., Neugebauer, M., Walsh, M., Wilson, S.: Annotating Syllable Corpora with Linguistic Data Categories in XML. In: Proceedings of the 4th International Conference on Linguistic Resources and Evaluation (2004) 14. Kipp, M.: Anvil A Generic Annotation Tool for Multimodal Dialogue. In: Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech), pp. 1367–1370 (2001) 15. Patterson, E.K., Gurbuz, S., Tufecki, Z., Gowdy, J.N.: CUAVE: A New Audiovisual Database for Multimodal Human Computer Interface Research. IEEE Conference on Acoustics, Speech and Signal Processing (2002) 16. Saenko, K., Livescu, K., Glass, J., Darell, T.: Visual Speech Recognition with Loosely Synchronised Feature Streams. In: Proceedings ICCV, Beijing (2005) 17. Sagey, E.: On the ill-formedness of crossing association lines. Linguistic Enquiry 19(1), 109–118 (1988) 18. Schmidt, T.: The transcription system EXMARaLDA: an application of the annotation graph formalism as the basis of a database of multilingual spoken discourse. In: Proceedings of the IRCS Workshop on Linguistic Databases (2001) 19. Van Bael, C., Boves, L., van den Heuvel, H., Strik, H.: Automatic Transcription of Large Speech Corpora. Computer Speech and Language 21(4), 652–668 (2007) 20. Walsh, M., Wilson, S.: An Agent-based Framework for Audio-visual Speech Investigation. In: Proceedings of Audio Visual Speech Processing Conference (2005) 21. Wilson, S.: Gesture-based Representations of Speech - Acquiring and Analysing Resources for Audio-visual Processing. PhD Thesis. University College Dublin (2007)

Time Durations of Phonemes in Polish Language for Speech and Speaker Recognition Bartosz Zi´olko and Mariusz Zi´ olko Department of Electronics AGH University of Science and Technology al.Mickiewicza 30, 30-059 Krak´ ow, Poland {bziolko,ziolko}@agh.edu.pl www.dsp.agh.edu.pl

Abstract. Statistical phonetic data for Polish were collected. Phonemes are of diﬀerent lengths, varying from 30 ms to 200 ms. Average phoneme durations are presented. A corpus of spoken Polish was used to collect statistic values of real language and evaluated to be applied in an automatic speech recognition and speaker identiﬁcation systems. These natural phenomena could be used in phonemes parametrisation and modelling. An additional source of information for a case of speech segmentation was obtained. The collected data are presented in the paper (average values for all available male speakers and for some chosen ones), along with comments on the corpus and the used method. The obtained data were compared with the expected values according to phonetic literature.

1

Introduction

The linguistic knowledge and statistic parameters are important part of speech technology applications. Phoneme durations could be used eﬀectively in speech modelling. There are published segmentation and acoustic modelling methods to locate phoneme boundaries in speech with unknown content [2], [19]. The segmentation combined with recognition could improve both processes, i.e. segmentation could be reset, if the recognised phoneme is of much diﬀerent duration than its expected statistical length. A phoneme duration can be seen as an additional parameter of a phoneme being recognised. The example of a word with phonemes of diﬀerent durations is presented in Fig. 1. Rate of speaking depends on individual characteristics of a person. This is why phoneme durations can be used for speaker identiﬁcation.

2

Phoneme Segmentation

Constant-time segmentation, i.e. framing for example into 23.2 ms blocks [17], is frequently used to divide the speech signal for digital processing. This method beneﬁts from simplicity of implementation and results in an easy comparison of blocks, which are of the same time duration. Anyway, the uniform segmentation Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 105–114, 2011. c Springer-Verlag Berlin Heidelberg 2011

106

B. Zi´ olko and M. Zi´ olko

DWT level 1

8 6

15

4

10

2

5

0

0

50

100

150

DWT level 3

5

DWT level 2

20

0

0

50

100

150

DWT level 4

15

4 10

3 2

5

1 0

0

50

100

150

DWT level 5

12

0

50

100

150

DWT level 6

3

10

2.5

8

2

6

1.5

4

1

2

0.5

0

0

0

50

100

150

0

0

50

100

150

Fig. 1. Discrete Wavelet Transform levels (from high to low frequencies) of word ’siedem’ (Eng. seven) with segmentation (dashed, vertical lines) noted by a phonetician

is perceptually unnatural, because the duration of phonemes varies signiﬁcantly and is clearly longer than 23.2 ms. Human phonetic categorisation is very poor for short segments [8]. Moreover, boundary eﬀects provide additional distortions (partially reduced by applying the Hamming window), and such short segments create many more boundaries than there are between phonemes in the acoustic signals. The boundary eﬀects can cause diﬃculties in speech recognition systems. Additional diﬃculties appear when two phonemes are mixed in a single frame (Fig. 2). Moreover the contextual connections between neighbouring phonemes make frequency properties of the begining and the end of phonemes extremely irregular. A smaller number of boundaries means a smaller number of errors due to the eﬀects described above. Constant segmentation therefore, while straightforward, risks losing valuable information about the phonemes due to the merging of diﬀerent sounds into a single block. Moreover, the complexity of individual phonemes cannot be represented in short frames.

Time Durations of Phonemes

a

b

a

a

a

?

107

Speech (2 phonemes)

b

b

b

uniform segmentation

phoneme duration segmentation

Fig. 2. Comparison of the frames produced by constant segmentation and phoneme segmentation

The length of a phoneme can be also used as an additional parameter in speech recognition improving the accuracy of the whole process. The phoneme durations can also help in speaker identiﬁcation, as they are speciﬁc for diﬀerent speakers. The system has to know the expected duration of all phonemes to achieve the greater eﬃciency. Models based on processing information over long time ranges have already been introduced. The RASTA (RelAtive SpecTrAl) methodology [5] is based on relative spectral analysis. The TRAPs (TempoRAl Patterns) approach [8] is based on multilayer perceptrons with the temporal trajectory of logarithmic spectral energy as the input vector. It allows to generate class posterior probability estimates. A number of approaches have been suggested [12], [3], [15], [20], [14], [19] to ﬁnd phoneme boundaries from the time-varying speech signal properties. These approaches utilise features derived from acoustic knowledge of the phonemes. For example, the solution presented in [3] analyses diﬀerent spectra subbands in the signal. DWT (Discrete Wavelet Transform) was applied for phoneme segmentation task [19]. Phoneme boundaries are extracted by comparing the fractions of signal power in diﬀerent subbands. The Toledano et al. [14] approach is based on spectral variation functions. Such methods need to be optimised for particular phoneme data and cannot be performed in isolation from phoneme recognition itself. ANN (Artiﬁcial Neural Networks) [13] have also been tested, but they require time consuming training. Segmentation can be applied by the SM (Segment Models) [9], [11] by searching paths through sequences of frames of diﬀerent lengths. Such a solution means that segmentation and recognition are conducted at the same time and there is a set of possible observation lengths. In a general SM, the segmentation is associated with a likelihood and in fact describes the likelihood of a particular segmentation of an utterance. The SM for a given label is also characterised by a family of output densities which gives information about observation sequences of diﬀerent lengths. These features of SM solution allow the location of boundaries only at several ﬁxed positions which are dependent on framing (i.e. on an integer multiple of the frame length).

108

B. Zi´ olko and M. Zi´ olko

The dynamic programming [10], [6] is a typical approach to phoneme segmentation for creating speech corpora. In speech segmentation it is used for time alignment of boundaries. The common practice is to provide a transcription done by professional phoneticians for one of the speakers in the given corpus. Then it is possible to automatically create phoneme segmentation of the same utterances for other speakers. This method is very accurate but demands transcription and hand segmentation to start with.

3

Experimental Data

The statistics were collected from CORPORA, created under supervision of Stefan Grocholewski in Institute of Computer Science, Pozna´ n University of Technology [4]. We investigated the male speakers only. Speech ﬁles in CORPORA were recorded with the sampling frequency f0 = 16 kHz equivalent to sampling period t0 = 62.5 μs. The part of the database, which we used, contains 365 utterances (33 single letters, 10 digits, 200 names, 8 simple computer commands and 114 short sentences), each spoken by 28 adult males, giving 10220 utterances totally. One set was hand segmented. The rest were segmented by a dynamic programming algorithm which was trained on hand segmented one and based on transcriptions and manually checked afterwards.

4

Statistics Collection

The phoneme duration statistics were collected from MLF (Master Label Files) attached to CORPORA. MLF is a standard solution, used for example in HTK (Hidden Markov Model Toolkit) [18]. MLF are deﬁned in [17] as index ﬁles holding pointers to the actual label ﬁles which can either be embedded in the same index ﬁle or stored anywhere else. The description starts with name of an audio ﬁlename. Phoneme transcriptions are given (starting time, end time, a phoneme description) in following lines. The format ends with a dot. A basic time unit in this standard is 100 ns. The example of a part of an MLF from CORPORA is as follows: ”*/ao1m1ada.lab” 0 50000 sil 100000 1350000 a 1400000 1900000 d 1950000 3100000 a 3150000 4150000 m 4200000 4350000 sil .

5

Results

All diﬀerences between starting and end times for all types of phonemes are summed separately. The quantities of all types of phonemes in the corpus were

Time Durations of Phonemes

109

Table 1. Average duration of Polish phonemes (in brackets divided by sum of all average durations ) with notations from CORPORA [4] and SAMPA [1], standard deviations and ratio between deviation and average CORPORA SAMPA

e a sz s si c a ci cz f zi e z rz drz o h dz u t dzi k i p n b y m d g w j l ni r l N

e j∼ o w∼ S s s’ ts a ts’ tS f z’ e z Z dz’ o x dz u t dZ k i p n b I m d g v j w n’ r l N

average [ms] (%) deviation 174 (4.51) 58 166 (4.31) 52 152 (3.94) 59 132 (3.43) 46 130 (3.37) 45 128 (3.32) 41 127 (3.30) 48 125 (3.24) 42 124 (3.22) 40 122 (3.17) 64 115 (2.98) 33 111 (2.88) 48 107 (2.78) 34 106 (2.75) 31 103 (2.67) 36 103 (2.67) 35 100 (2.59) 45 100 (2.59) 35 99 (2.57) 42 98 (2.54) 52 98 (2.54) 27 94 (2.44) 45 93 (2.41) 38 93 (2.41) 41 91 (2.36) 41 88 (2.28) 27 88 (2.28) 43 86 (2.23) 34 83 (2.15) 29 83 (2.15) 28 82 (2.13) 32 81 (2.10) 34 79 (2.05) 33 76 (1.97) 33 73 (1.89) 30 72 (1.87) 31 72 (1.87) 25

deviation average

0.33 0.31 0.39 0.35 0.35 0.32 0.38 0.34 0.32 0.52 0.29 0.43 0.32 0.29 0.35 0.34 0.45 0.35 0.42 0.53 0.28 0.48 0.41 0.44 0.45 0.31 0.49 0.40 0.35 0.34 0.39 0.42 0.42 0.43 0.41 0.43 0.35

example g¸e´s ci¸az˙ a szyk syk ´swit cyk pat ´cma czyn fan ´zle test zbir z˙ yto d˙zem pot hymn dzwo´ n puk test d´zwig kit PIT pik nasz bit typ mysz dym gen wilk jak lyk ko´ n ryk luk p¸ek

transcr. ge∼s’ ts’ow∼Za SIk sIk s‘vit tsIk pat ts’ma tSIn fan z’le test zbir ZIto dZem pot xImn dzvon’ puk test dz’vik kitk pit pik naS bit tIp mIS dIm gen vilk jak wIk kon’ rIk luk peNk

also saved. Then the average phoneme duration was calculated, as the sum of durations divided by the number of phoneme occurrences. The standard deviation was also calculated to evaluate how useful and trustworthy the data are. The statistics are presented in Tab. 1. The average durations vary from 72 ms to 174 ms.

110

B. Zi´ olko and M. Zi´ olko Table 2. Statistics of phoneme durations for some particular male speakers Speaker AO1M1

Speaker JS1M1 deviation

deviation

CORPORA average (%) deviat. average

CORPORA average(%) deviat. average

e a sz s si c a ci cz f zi e z rz drz o h dz u t dzi k i p n b y m d g w j l ni r l N

e a sz s si c a ci cz f zi e z rz drz o h dz u t dzi k i p n b y m d g w j l ni r l N

154 (4.31) 152 (4.26) 140 (3.92) 131 (3.68) 131 (3.66) 118 (3.30) 104 (2.91) 120 (3.37) 114 (3.20) 118 (3.31) 117 (3.28) 97 (2.72) 106 (2.98) 104 (2.92) 105 (2.93) 91 (2.56) 106 (2.98) 98 (2.73) 87 (2.43) 87 (2.44) 89 (2.50) 92 (2.59) 83 (2.33) 98 (2.73) 72 (2.03) 83 (2.32) 76 (2.13) 73 (2.05) 69 (1.92) 79 (2.20) 70 (1.97) 64 (1.80) 68 (1.89) 68 (1.89) 63 (1.76) 60 (1.68) 83 (2.32)

44 43 37 43 46 36 32 34 32 50 32 39 28 29 32 31 42 24 33 41 22 37 31 40 27 21 30 26 24 29 22 23 21 30 23 18 22

0.29 0.28 0.26 0.33 0.36 0.31 0.30 0.28 0.28 0.42 0.27 0.41 0.26 0.28 0.31 0.34 0.40 0.24 0.38 0.47 0.24 0.40 0.38 0.41 0.37 0.26 0.39 0.35 0.35 0.37 0.32 0.36 0.31 0.45 0.37 0.31 0.27

182 (4.30) 166 (3.94) 167 (3.95) 148 (3.50) 133 (3.16) 147 (3.48) 147 (3.48) 134 (3.17) 133 (3.14) 115 (2.73) 120 (2.84) 124 (2.94) 117 (2.77) 113 (2.68) 105 (2.48) 113 (2.69) 103 (2.45) 126 (2.99) 112 (2.64) 102 (2.42) 119 (2.82) 95 (2.26) 104 (2.47) 99 (2.35) 104 (2.47) 97 (2.30) 99 (2.34) 93 (2.20) 98 (2.33) 93 (2.20) 95 (2.25) 88 (2.09) 92 (2.18) 85 (2.01) 83 (1.95) 81 (1.91) 91 (2.15)

73 57 49 40 37 44 53 38 35 44 29 49 29 24 31 35 37 36 41 47 28 44 38 42 43 24 41 28 31 26 26 32 38 29 33 34 16

0.40 0.34 0.29 0.27 0.28 0.30 0.36 0.29 0.27 0.38 0.24 0.39 0.25 0.21 0.30 0.31 0.36 0.28 0.37 0.46 0.23 0.46 0.37 0.43 0.41 0.25 0.41 0.30 0.31 0.28 0.28 0.36 0.41 0.35 0.40 0.43 0.18

CORPORA transcriptions are based on SAMPA notation with 37 symbols. Letters ¸e and a¸ are phonetically transcribed as e and a in CORPORA. However, each of these letters should be actually represented by two phonemes: ¸e should be e j∼ and a¸ should be o w∼. We are not able to detect these extra boundaries precisely enough. This is why we decided to keep them together as they are in the corpus. This is why e and a are the longest in Tab. 1. Each of them represents two phonemes, actually.

Time Durations of Phonemes

111

Data like the presented in Tab. 1 are till now not available for Polish. However, we have found some information regarding this topic. According to [16], duration of phonemes is changeable and depends on speech ratio, type of utterance, localisation in a syllable and accents. A ratio between durations of diﬀerent phonemes is quite constant. The longest ones are e and a . Then a, o and e are a bit shorter. Phonemes i, y, u follow them. Next, n and m are average ones with r a bit shorter. Phonemes l and l are even shorter and j is the shortest one. It corresponds a bit to our results but not completly. Phonemes e and a are indeed the longest ones in both descriptions. We found that a and e are long, as described in [16] but o is average. We realised that i, u are expected to be quite long [16] but in CORPORA they are of average duration. Phoneme y is even shorter, however, [16] claims it should be long. Phonemes n and m are quite average as stated in [16]. Phoneme r was found by us as a short phoneme what is in a contrast with [16]. The experiment supports the opinion from [16] that l, l and j are short. According to [16] there is a general rule that a duration is bigger for phones, for which a bigger number of parts of vocal tract are necessary to be used. The standard deviations of our results are generally high. The ratio between standard deviation and average duration vary and is between 0.27 and 0.53. Phonemes f, t, y, k, n, h, p, ni, e, l (CORPORA notation) have relativelly high standard deviations. It is probably a result of diﬀerent ways of pronouncing these phonemes by diﬀerent people. The ratio of standard deviation to average value is lowest for phonemes dzi, zi, rz, a , z, c, cz. There are some similar data in [7]. However, it does not present a complete list of phonemes durations. It gives some examples like that: a transient can be up to 50 ms, t around 100 ms and r usually 20 ms. Again, some of these values corresponds to our results, like transient to short pause and phoneme t, but not all of them, like r, which is one of the shortest in our list, but its duration is 73 ms rather than 20 ms. Let us denote the average values and the standard deviations, presented in Table 1, by m ∈ 37 and d ∈ 37 , respectively. For ith speaker, where 1 ≤ i ≤ 28, let mi be the average value and di let be the standard deviation (see Table 2). For one of the speakers no phoneme N was detected. In order to avoid biased results, average value of all speakers was used in following calculations. For the collected data the correlation matrix C=

28

(m − mi )(m − mi )T

(1)

i=1

was computed. The results in graphical form are presented in Fig. 3. In keeping with expectations, the vast majority of durations of phonemes are correlated positively. It means that almost all phonemes for each speaker are simultaneously greater or smaller then their average values. It depends on speech velocity. The greatest correlation is between phonemes sz-f, s-f, c-f, ci-f, cz-f, h-f, sz-s, sz-si, sz-cz, s-c. The smallest correlation have the following pairs of phonemes: e -N, a -N, si-a, si-e, si-o, si-u, si-y, a-cz, a-zi, a-drz, a-h, a-dzi, a-p, a-ni, a-N,

112

B. Zi´ olko and M. Zi´ olko

e_ a_ sz s si c a ci cz f zi e z rz drz o h dz u t dzi k i p n b y m d g w j l_ ni r l N e_ a_ sz s si c a ci cz f zi e z rz drz o h dz u t dzi k i p n b y m d g w j l_ ni r

l N

Fig. 3. The graphical presentation of values of correlation matrix (1)

cz-o, cz-u, cz-y, f-y, zi-e, zi-o, zi-u, zi-y, e-ni, e-p, e-N, e-drz, e-dzi, rz-o, rz-u, rz-y, y-h, u-p, u-N, t-y, o-dzi, o-p, o-ni, o-N, drz-y, drz-o, drz-y, drz-i, dzi-y, k-y, i-p, i-N, p-y, y-ni, y-N. Generally small correlations are between vowels and some of consonants (N, si, cz, zi, drz, dzi, p, f, rz, h, ni).

6

Conclusions

The average values of phoneme durations for Polish vary from 72 ms to 174 ms. The standard deviations are high and usually are around one third of the average values. It is caused by considerably high diversities of speech rate of diﬀerent people. The obtained data can be used eﬃciently in speech modelling and in automatic speech recognition systems. Average duration of phonemes strongly depends on a speaker and can be used for speaker identiﬁcation. In Table 2 there are presented

Time Durations of Phonemes

113

values for two diﬀerent speakers i = 1 and i = 9. The properties of ith speaker are speciﬁed by vector mi , and its distance to mean value vector m can be computed as 1 m(j) − mi (j) 1 mi (j) = 1− . 37 j m(j) 37 j=1 m(j) 37

δi =

37

(2)

For speaker AO1M1 δ1 = 0.0759 was obtained, while for the JS1M1 δ9 = −0.1026. It is possible to compare relative values (i.e. % presented in brackets) of each speaker with average values. For this case we can use measure deﬁned in the following way δ%i =

37 37 1 m% (j) − mi% (j) αi mi (j) =1− , 37 j m% (j) 37 j−1 m(j)

where

37

αi =

(3)

m(j)

j=1 37 j−1

mi (j)

.

(4)

For AO1M1 δ1% = 0.0021 was obtained, while for JS1M1 δ9% = −0.0069. It is easy to show that δ%i = 1−αi (1−δi ). These examples show that values obtained from (2) are higher and more strongly dependent on speakers. The relative values used in (3) are rather independent of speakers and can be used to support the phonemes recognition.

Acknowledgements This work was supported by MNISW grant OR00001905.

References 1. Demenko, G., Wypych, M., Baranowska, E.: Implementation of grapheme-tophoneme rules and extended SAMPA alphabet in Polish text-to-speech synthesis. In: Speech and Language Technology, PTFon, Pozna´ n, vol. 7(17) (2003) 2. Glass, J.: A probabilistic Framework for Segment-Based Speech Recognition. Computer Speech and Language 17, 137–152 (2003) 3. Grayden, D.B., Scordilis, M.S.: Phonemic Segmentation of Fluent Speech. In: Proceedings of ICASSP, Adelaide, pp. 73–76 (1994) 4. Grocholewski, S.: Zalo˙zenia akustycznej bazy danych dla j¸ezyka polskiego na no´sniku cd rom (eng. Assumptions of acoustic database for Polish language). Mat. I KK: Glosowa komunikacja czlowiek-komputer, Wroclaw 5. Hermansky, H., Morgan, N.: RASTA processing of speech. IEEE Transactions on Speech and Audio Processing 2(4), 578–589 (1994) 6. Holmes, J.N.: Speech Synthesis and Recognition (2001)

114

B. Zi´ olko and M. Zi´ olko

7. Jassem, W.: Podstawy fonetyki akustycznej (Eng. Rudiments of acoustic phonetics). Pa´ nstwowe Wydawnictwo Naukowe, Warszawa (1973) 8. Morgan, N., Zhu, Q., Stolcke, A., Sonmez, K., Sivadas, S., Shinozaki, T., Ostendorf, M., Jain, P., Hermansky, H., Ellis, D., Doddington, G., Chen, B., Cretin, O., Bourlard, H., Athineos, M.: Pushing the envelope - aside. IEEE Signal Processing Magazine 22(5), 81–88 9. Ostendorf, M., Digalakis, V.V., Kimball, O.A.: From HMM’s to segment models: A uniﬁed view of stochastic modeling for speech recognition. IEEE Transactions on Speech and Audio Processing 4, 360–378 10. Rabiner, L., Juang, B.-H.: Fundamentals of speech recognition. PTR Prentice-Hall, Inc., New Jersey (1993) 11. Russell, M., Jackson, P.J.B.: A multiple-level linear/linear segmental HMM with a formant-based intermediate layer. Computer Speech and Language 19, 205–225 12. St¨ ober, K., Hess, W.: Additional use of phoneme duration hypotheses in automatic speech segmentation. In: Proceedings of ICSLP, Sydney, pp. 1595–1598 (1998) 13. Suh, Y., Lee, Y.: Phoneme segmentation of continuous speech using multi-layer perceptron. In: Proceedings of ICSLP, Philadelphia, pp. 1297–1300 (1996) 14. Toledano, D.T., G´ omez, L.A.H., Grande, L.V.: Automatic phonetic segmentation. IEEE Transactions on Speech and Audio Processing 11(6), 617–625 (2003) 15. Weinstein, C.J., McCandless, S.S., Mondshein, L.F., Zue, V.W.: A system for acoustic-phonetic analysis of continuous speech. IEEE Transactions on Acoustics, Speech and Signal Processing 23, 54–67 16. Wierzchowska, B.: Fonetyka i fonologia j¸ezyka polskiego (Eng. Fonetics and phonology of Polish). Zaklad Narodowy im. Ossoli´ nskich, Wroclaw (1980) 17. Young, S.: Large vocabulary continuous speech recognition: a review. IEEE Signal Processing Magazine 13(5), 45–57 18. Young, S., Evermann, G., Gales, M., Hain, T., Kershaw, D., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev, V., Woodland, P.: HTK Book. Cambridge University Engineering Department, UK 19. Zi´ olko, B., Manandhar, S., Wilson, R.C., Zi´ olko, M.: Wavelet method of speech segmentation. In: Proceedings of 14th European Signal Processing Conference EUSIPCO, Florence (2006) 20. Zue, V.W.: The use of speech knowledge in automatic speech recognition. Proceedings of the IEEE 73, 1602–1615 (1998)

Polysemous Verb Classification Using Subcategorization Acquisition and Graph-Based Clustering Fumiyo Fukumoto, Yoshimi Suzuki, and Kazuyuki Yamashita∗ Interdisciplinary Graduate School of Medicine and Engineering, Faculty of Education Human Sciences , Univ. of Yamanashi, Kofu 400-8511, Japan {fukumoto,ysuzuki,kazuyuki}@yamanashi.ac.jp

Abstract. This paper presents a method for classifying Japanese polysemous verbs. We used a graph-based unsupervised clustering algorithm, which detects the spin configuration that minimizes the energy of the material. Comparing global and local minima of an energy function allows for the detection of spins (nodes) with more than one cluster. We applied the algorithm to cluster polysemies. Moreover, we used link analysis to detect subcategorization frames, which are used to calculate distributional similarity between verbs. Evaluation are made on a set collected from Japanese dictionary, and the results suggest that polysemy, rather than being an obstacle to word sense discovery and identification, may actually be of benefit. Keywords: Polysemies, Verb Classification, Soft Clustering Algorithm, Markov Random Walk Model.

1 Introduction Semantic classification of words is not an end task in itself, but supports many NLP tasks, such as word sense disambiguation, and automatic construction of thesauri. A number of methodologies have been developed for word classification. One such attempt is to apply clustering techniques to classification[16,23,14]. However, two main difficulties arise in the use of clustering algorithm. The first is that we do not know how many classes there are in a given input verbs. The second is that the algorithm should allow each data point (word) to belong to more than one cluster because of the existence of polysemies. It is often the case that a word which is polysemous in general is not polysemous within a restricted subject domain. However, even in texts from a restricted subject domain such as Wall Street Journal, one encounters quite a large number of polysemous words. In particular, unlike nouns, verbs are often polysemous even in a restricted subject domain. In this article, we focused on Japanese polysemous verbs, and present a method for classifying them into semantic classes of IPAL Japanese dictionary. We used a graph-based clustering method which allows nodes (verbs) to belong to multiple clusters (senses). The essence of this approach is to define an energy function, called the Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 115–126, 2011. c Springer-Verlag Berlin Heidelberg 2011

116

F. Fukumoto, Y. Suzuki, and K. Yamashita

Hamiltonian which achieves minimal energy when there is high within-cluster connectivity and low between-cluster connectivity. The energy minimum is obtained by simulated annealing. In this context, two verbs are “connected” if they share many of the same subcategorization frames. We used a large Japanese newspaper corpus to obtain verbs and their subcategorization frames. One problem is that each verb is extremely high dimensional representations for which it would not be convenient to use clustering techniques directly. In response to this insight, the present work applies link analysis called the Markov Random Walk (MRW) model on the verb subcategorization frames. The MRW model is a ranking algorithm that has been successfully used in Web-link analysis, social networks [25], and more recently in text processing applications. The approach decides the importance of a vertex within a graph based on global information drawn recursively from the entire graph [3]. We applied the algorithm to detect verb subcategorization frames. The rest of the article is organized as follows. The next section provides an overview of existing techniques. Section 3 explains Japanese verb description including selection of verb frame patterns and distributional similarity measures to compute semantic similarities between pairs of verbs. Section 4 explains a clustering algorithm. Finally, we report some experiments using IPAL Japanese dictionary, and end with a discussion of evaluation.

2 Related Work With the growth of readily available fine-grained and large-scale semantic knowledge like WordNet, COMLEX, EDR dictionary, research activities for classifying words into semantic classes by using corpus-based statistics, and evaluate their methods using semantic knowledge are very popular. For example, Levin’s taxonomy of verbs and their classes is widely used to evaluate classification, as it is the largest syntactic-semantic verb classification in English providing a classification of 3,024 verbs into 192 finegrained classes [10]. Schulte [20] attempted to classify English verbs using two algorithms: iterative clustering based on a definition by [5], and unsupervised latent class analysis as described by [19], based on the expectation maximization algorithm. Stevenson and Joanis [22] compared their supervised method for English verb classification with semi-supervised and unsupervised techniques. Brew et al. focused on dimensionality reduction on the verb frame patterns, and applied a spectral clustering technique [14] to the unsupervised clustering of German verbs to Levin’s English classes[1]. They reported that the results by a spectral clustering outperformed the standard k-means against all the evaluation measures including “F-measure” and all the distance measures including “skew divergence.” In the context of graph-based clustering of words, Widdows and Dorow used a graph model for unsupervised lexical acquisition [23]. The graph structure is built by linking pairs of words that participate in particular syntactic relationships. An incremental cluster-building algorithm using the graph structure achieved 82% accuracy at a lexical acquisition task, evaluated against WordNet 10 classes, and each class consists of 20 words. Matsuo et al. proposed a method of word clustering based on a word similarity measure by Web counts [12]. They used Newman clustering, and reported that

Polysemous Verb Classification

117

the results obtained with the algorithm were better than those obtained by average-link agglomerative clustering using 90 Japanese noun words. However, all these methods relied on hard-clustering models, and thus have largely ignored the issue of polysemy by assuming that words belong to only one cluster. In contrast to hard-clustering algorithms, soft clustering allows that words to belong to more than one cluster. Much of the previous work on soft clustering is based on the EM algorithms. The earliest work in this direction is that of Pereira et al. who described a hierarchical soft clustering method that clusters noun words [16]. The clustering result was a hierarchy of noun clusters, where every noun belongs to every cluster with a membership probability. The initial data for the clustering process were frequencies of verb-noun pairs in a direct object relationship, as extracted from conditional verb-noun probabilities, the similarity of the distributions was determined by the KL divergence. The EM algorithm was used to learn the hidden cluster membership probabilities, and deterministic annealing performed the divisive hierarchical clustering. Schulte et al. proposed a method for semantic verb classification that relies on selectional preferences as verb properties [21]. The model was implemented as a soft clustering approach to capture the polysemous verbs. The training procedure used the EM algorithm to iteratively improve the probabilistic parameters of the model, and applied the MDL principle to induce WordNet-based selectional preferences for arguments within subcategorization frames. The results showed the effectiveness of the method, while they did not report in detail whether the clusters captured polysemic verbs. Our work is similar to their method in the used of verb frames with selectional preferences. Korhonen et al. used verb-frame pairs to cluster verbs into Levin-style semantic classes [7]. They introduced a new evaluation scheme against a polysemic gold standard to allow for sense variation because their method interpreted polysemy as multiple-hard assignments. Our work is similar to their method in the use of clustering method, and focused especially on verbal polysemy. The difference is, again, that the method presented in this article allows that verbs belong to more than one cluster. This enables us to handle truly polysemies, each of which has different numbers of different senses.

3 Japanese Verb Description 3.1 Selectional Preferences A typical word clustering task is to cluster words into classes based on their distributional similarity. A Vector Space Model (VSM) is often used to calculate distributional similarity of two words[4,11,2]. Like previous work on verb classification, we used subcategorization frame distributions with selectional preferences to calculate similarity between verbs[21]. We used EDR dictionary of selectional preferences consisting of 5,269 basic Japanese verbs and EDR sense dictionary. The dictionary has each sense of a verb, the group of possible co-occurrence surface-level case particles, the types of sense relation label that correspond to the surface-level case as well as the range of possible senses that may fill the deep-level case. Table 1 illustrates an example of a verb “ashirau (arrange)“. “Sentence pattern” refers to the co-occurrence pattern between a verb and a noun with a case marker. “Sense relation” expresses the deep-level case, while “Case particle” shows the surface-level case. “Sense identifier” refers to the

118

F. Fukumoto, Y. Suzuki, and K. Yamashita Table 1. An example of a verb “ashirau (arrange)“

[Sentence pattern] [Sense relation] [Case particle] [Sense identifier]

ga agent ga (nominative) 30f6b0 (human); 30f746 (organization)

wo object wo (accusative) 30f6ae (entity)

< word3 > ni ashirau (arrange) goal ni (dative) 30f6ae (entity)

range of possible senses for the case filler. We note that all of the arguments in Table 1 are obligatory which are illustrated by the following examples. (1) Taro ga flowers wo Hanako ni ashirau. ”Taro arranged the flowers for Hanako.” (2) Taro ga flowers wo φ ashirau ”Taro arranged the flowers.” (3) Taro ga φ Hanako ni ashirau ”Taro arranged (it) for Hanako (4) φ flowers wo Hanako ni ashirau ”(Someone) arranged the flowers for Hanako.” We selected four subcategorization frame pattern from the examples shown in (1), (2), (3) and (4): these are instances of the different subcategorization frame. The subcategorization frame pattern of (1), for example, consists of three arguments with selectional preferences and is given below: ashirau 30f6b0;30f746 ga 30f6ae wo 30f6ae ni arrange human;org. nom entity obj entity obj where x of the argument “x y” refers to sense identifier and y denotes case particle. 3.2 Link Analysis The Markov Random Walk (MRW) model is a method of deciding the importance of a vertex within a graph based on global information recursively drawn from the entire graph [3]. The basic idea is that of “voting” or “recommendation” between the vertices. A link between two vertices is considered as a vote cast from one vertex to the other vertex. The score associated with a vertex is determined by the votes that are cast for it, and the score of the vertices casting these votes. We applied MRW to detect important frames for each verb. Given a set of verbs V , G = (V F , E) is a graph reflecting the relationships between frames in the verb set. V F is a set of vertices and each vertex vfi in V F is a verb frame pattern in the verb set. E is the set of edges, which is a subset of V F × V F . Each edge eij in E is associated with an affinity weight f (i → j) between two verb frame patterns vfi and vfj (i = j). The weight is computed using the standard cosine measure between the two frame patterns. f (i → j) =

vfi · vfj . | vfi | × | vfj |

(1)

Polysemous Verb Classification

119

where vfi and f vj are the corresponding vectors of vfi and vfj . The vertices are connected if their affinity weight is larger than 0 and we let f (i → i)= 0 to avoid self transition. The transition probability from vfi to vfj is then defined as follows: ⎧ f (i→j) ⎪ , if Σf = 0 ⎪ ⎪ F| ⎨ |V f (i→k) p(i → j) = ⎪ ⎪ k=1 ⎪ ⎩ 0 , otherwise.

(2)

We used the row-normalized matrix Uij = (Uij )|V F |×|V F | to describe G with each entry corresponding to the transition probability, where Uij = p(i → j). In order to make U be a stochastic matrix, the rows with all zero elements are replaced by a smoothing vector with all elements set to |V1F | . The matrix form of the saliency score Score(vfi ) can be formulated in a recursive form as in the MRW model. λ = μU T λ +

(1 − μ) e. |VF |

(3)

where λ = [Score(vfi )]|V F |×1 is the vector of saliency scores for the frame patterns. e is a column vector with all elements equaling to 1. μ is the damping factor. We set μ to 0.85, as in the PageRank algorithm [15]. The final transition matrix in the Markov chain is given by formula (4), and each score of the frame pattern is obtained by the principle eigenvector of the new transition matrix M . M = μT +

(1 − μ) T ee . |VF |

(4)

After the saliency scores of frame patterns have been obtained, choose a certain number of patterns according to the rank score into frames for each verb. 3.3 Distributional Similarity We concentrated on eight more commonly used measures, (1) the cosine measure (Cos), (2) the cosine measure based on probability of relative frequencies (rfCos), (3) the Dice coefficient (Dice), (4) L1 Norm (L1 ), (5) Jaccard’s coefficient (Jacc), (6) KullbackLeibler (KL), (7) α-skew divergence (αdiv.) and (8) the Jensen-Shannon (JS). Kullback-Leibler is not defined in case vi = 0, where vi refers to the number of times verb v occurred in frame i. Therefore, the probability distributions must be smoothed. We tested two smoothing methods, i.e., Add-one smoothing and Witten and Bell smoothing [24]. We report Add-one smoothing results in the evaluation, as it was better than Witten and Bell smoothing. Similarly, we used α = 0.9 of α-skew divergence, as Lee reported the best results with the value [9]. All measures except Cos, rfCos, Dice, and Jacc showed that smaller values indicate a closer relation between two verbs. Thus, we used inverse of each value.

120

F. Fukumoto, Y. Suzuki, and K. Yamashita

4 Clustering Method We used a graph-based unsupervised clustering technique reported by [17]; we call this the RB algorithm. This algorithm detects the node configuration that minimizes the energy of the material. The energy function, called the Hamiltonian, for assignment of nodes into communities clusters together those that are linked, and keeps separate those that are not by rewarding internal edges between different clusters. Here, “community” or “cluster” have in common that they are groups of densely interconnected nodes that are only sparsely connected with the rest of the network. Only local information is used to update the nodes which makes parallelization of the algorithm straightforward and allows the application to very large networks. Moreover, comparing global and local minima of the energy function allows the detection of overlapping nodes. Reichardt et al. evaluated their method by applying several data including a large protein folding network, and reported that the algorithm successfully detected overlapping nodes[18]. We thus used the algorithm to cluster verbs. Let vi (1 ≤ i ≤ n) be a verb in the input, and σi be a label assigned to the cluster in which vi is placed. The Hamiltonian H is defined as: H({σi }) = − (Aij (θ) − γpij )δσi σj . (5) i 0 Aij (θ) = (6) 0 otherwise. We calculated sim(vi , vj ), i.e., similarity between vi and vj using one of the measures mentioned in Section 3.3. The matrix pij in Eq. (5) denotes the probability that a link exists between vi and vj , and is defined as: pij =

i Lexicon-Grammar > View.

Using Lexicon-Grammar Tables for French Verbs in a Large-Coverage Parser

185

Our conversion process starts from lglex verbal entries and turns them into entries in the Alexina format, i.e., the same as the format of the syntactic lexicon Lefff .

3 The Lefff Syntactic Lexicon and the Alexina Format The Lefff (Lexique des formes fléchies du français — Lexicon of French inflected form) is a large-coverage syntactic lexicon for French [10,11]4 . It relies on the Alexina framework for the acquisition and modeling of morphological and syntactic lexicons. To represent lexical information, an Alexina lexicon relies on a two-level architecture: – the intensional lexicon associates (among others) an inflection table and a canonical sub-categorization frame with each entry and lists all possible redistributions from this frame; – the compilation of the intensional lexicon into an extensional lexicon builds different entries for each inflected form of the lemma and every possible redistribution. For example, consider the following (simplified) intensional entry: clarifier1 Lemma;v;; %ppp_employé_comme_adj,%actif,%passif, %se_moyen_impersonnel,%passif_impersonnel

It describes an entry of the verbal lemma clarifier (clarify) which is transitive (two arguments canonically realized by the syntactic functions Suj and Obj listed between brackets), and which allows for the functional redistributions past participle used as an adjective, active (the default distribution), impersonal middle-voice “se” construction, impersonal passive, and passive. The different syntactic functions are defined in the Lefff by criteria close to that used by the authors of the verb valency lexicon D ICOVALENCE [15], i.e., they rely for a large part on cliticization and other pronominal features. The Lefff uses the following syntactic functions: Suj (subject), Obj (direct object), Objà (indirect object canonically introduced by preposition “à”), Objde (indirect object canonically introduced by preposition “de”), Loc (locative), Dloc (delocative), Att (attribute), Obl or Obl2 (other oblique arguments). Defining criteria for these functions are described in [11]. Each syntactic function can be realized by three types of realizations: clitic pronouns, direct phrases (nominal phrase (sn), adjectival phrase (sa), infinitive phrase (sinf), completive (scompl), indirect interrogative (qcompl)) and prepositional phrases (direct phrases preceded by a preposition, such as de-sn, à-sinf or pour-sa).5 Finally, a function whose realization is not optional has its realizations list between brackets. Complementary syntactic information (control, mood for completives, etc.) are represented by macros (@CtrlSujObj, @ComplSubj, etc.) whose formal interpretation varies according to the context of use. An LFG modeling of these macros is provided with the Lefff . 4

5

On-line distribution under the LGPL-LR license at http://gforge.inria.fr/ projects/alexina/ à-scompl and de-scompl represent realizations of the form à/de ce que P.

186

E. Tolone and B. Sagot

4 Conversion of the Verbal Lexicon lglex into a Lexicon in the Alexina Format 4.1 Sketch of the Conversion Process Each entry in lglex is associated with a set of constructions that can be classified into several types: 1. the “base” construction(s), defining feature for the originating class of the entry; 2. “extended base” constructions, obtained by adding extra arguments to the base construction; in practice, these constructions are all intermediate constructions between the base construction and a construction called “maximal extended base” construction, or MEBC; 3. constructions that are variants of the base construction, obtained by deleting one or several arguments, or by changing the realization type (e.g., Qu P can become Vi inf W, as for the direct object of savoir — to know —, that can be a finite phrase but also an infinitive phrase); 4. constructions that are in fact redistributions, such as [passif de], that denotes the possibility of having a passive with an agent introduced by de (cf. Pierre est aimé de Marie — Pierre is loved by Marie) or N1 est Vpp de ce Qu P (cf. Marie est étonnée de ce que Pierre soit là — Marie is surprised that Pierre is here) ; 5. constructions that should seemingly have led to distinct entries, called “secondary entries”, such as neutral constructions of transformations like N1 se V de ce Qu P (cf. Luc se félicite d’avoir réussi à séduire Léa vs. Max félicite Luc qu’il ait réussi à séduire Léa — Luc is very pleased he succeeded in seducing Léa vs. Max congratulates Luc for having succeeded in seducing Léa). We developed a method for aligning two constructions, i.e., for building correspondences between arguments despite their surface differences6 and their possible deletion. This method allows us to identify and align the MEBC and its variants, which we put together in a single entry of the final lexicon, called canonical entry. Among the other constructions, those that correspond to standard redistributions ([passif par], [extrap]. . . ) lead to the inclusion of the corresponding redistribution in the canonical entry.7 Other constructions lead to the creation of extra entries, because they correspond to secondary entries (5th type in the enumeration above) or because they involve redistributions that have not yet been identified in the Alexina format. Once the entries to be produced are identified, we build sub-categorization frames. First, we build the frame corresponding to the maximal construction for each entry (the 6 7

For example, Qu P vs. N1, or à N1 vs. Prép N1 if in addition it is known that Prép can be à. In the table of classes, the feature [passif par] (the standard passivability) is not yet correctly described, even for transitive classes. Considering this lack of information as a negative information (non-passivable), as done for other features, leads to a really incomplete lexicon. Therefore, we decided to add the corresponding %passif redistribution to all entries that have an argument whose syntactic function is Obj (direct object). Note that direct complements of the entries of table 32NM do not receive the function Obj (see below). Therefore, our heuristics is valid, apart from rare cases of non-passivability such as regarder (often to look at, but also to concern) in the sense of concerner (to concern).

Using Lexicon-Grammar Tables for French Verbs in a Large-Coverage Parser

187

MEBC for the canonical entry, and their unique construction for secondary entries). The syntactic function of each argument is obtained by the following heuristics. First, the first argument always receives the function Suj (subject). The first post-verbal argument, if it is direct, receives the function Obj, apart from entries of table 32NM. Then, an argument introduced by à (resp. de) receives the syntactic function Objà (resp. Objde), except if an additional indicator contradicts this choice (e.g., for an N1 argument introduced by à, the feature à N1 = Ppv =: le shows it must receive the syntactic function Obj, as in Il apprend à conduire / Il l’apprend — He is learning how to drive / He is learning it). Arguments introduced by Loc have the syntactic function Loc, except those of the form Loc Ni source or for which Loc Ni =: de Ni source is a valid feature, which receive the syntactic function Dloc. Finally, other arguments are considered as Att if they are direct, and as Obl if they are introduced by a preposition (Obl2 if an Obl already exists). The realizations of these syntactic functions are built in two steps. First, the kind of phrase (nominal, infinitive, etc.) is determined. Then, possible introducers are extracted from the set of corresponding prepositions and other introducers (e.g., et — and). For the canonical entry, all variants of the MEBC lead to modifications of the resulting sub-categorization frame, by adding realizations and making some arguments optional. Other types of information are then added so as to finalize the entry, such as the originating table and the corresponding row number, as well as a frequency information extracted from the DELA. Finally, syntactic macros concerning the auxiliary, the mood of completive arguments, idiomatic clitics (se, en, ne, etc.) and control phenomena are extracted and added to the final entry. 4.2 Resulting Lexicon The resulting verbal lexicon contains 16,903 entries for 5,694 distinct verb lemmas (on average, 2.96 entries per lemma). As a comparison, the Lefff only contains 7,072 verbal entries for 6,818 distinct verb lemmas (on average, 1.04 entries per lemma). The resulting lexicon extracted from lglex, despite the fact that it describes fewer verbal lemmas, has a larger coverage in terms of syntactic constructions and therefore is much more ambiguous. At the extensional level, the Lefff has 361,268 entries whereas the lexicon extracted from lglex has 763,555 entries. The construction of this lexicon from lglex according to the process described in this section is achieved by a perl script that contains less than 1,000 lines. The conversion in itself, i.e., the execution of the script of the whole lglex, takes less than a minute.8 Therefore, if a new version of the Lexicon-Grammar French verb tables or of the corresponding table of classes is released, building the new corresponding Alexina-format lexicon is a matter of seconds, and does not require any new development.

5 Integration in the FRMG Parser The main goal of this work is to allow the use of the linguistic data coded in LexiconGrammar tables for French to be used as a lexical database for a French parser. Among 8

On a 2.4 GHz machine using Ubuntu Linux.

188

E. Tolone and B. Sagot

the various parsers that rely on a syntactic lexicon in the Alexina format, we chose the FRMG parser [14]. It relies on a compact factorized Tree Adjoining Grammar (TAG) generated from a meta-grammar, and on the Lefff . The compilation and execution of the parser is performed by the DYAL OG system [3]. The result of the parsing itself is a derivation shared forest, that undergoes a symbolic (weight-based) disambiguation process so as to output only one parse. In case a sentence is not covered by the grammar and the lexicon, FRMG outputs several partial parses that cover disjoint parts of the sentence (however, no attempt is made to reassemble these partial parses into a global parse). FRMG takes as its input the result of the presyntactic processing chain S XPipe [13], which converts a raw text into a lattice of forms known by the lexicon (namely, the Lefff ).9 The integration of the Alexina-format lexicon extracted from lglex in the FRMG parser is straightforward: in its standard configuration FRMG ’s lexer performs calls to a lexical database built from the Lefff . We shall call this standard parser FRMG Lefff . What is required to use the lexical information from Lexicon-Grammar verb tables is to replace verbal entries in the Lefff by those of the lexicon built from lglex while keeping other Lefff entries, to build the corresponding lexical database, and to tell FRMG to use it rather than the default Lefff -only one. However, several verbal entries which are not covered by lglex had to be added as well: entries for auxiliaries and semi-auxiliaries, some raising verbs, impersonal verbs and light verbs. The result is a variant of the FRMG parser, that we shall call FRMGlglex , to distinguish it from the standard FRMG Lefff .

6 Evaluation and Discussion We evaluated both FRMG Lefff and FRMG lglex by parsing the manually annotated part of the EASy corpus [7], i.e., 4,306 sentences of diverse genres (journalistic, medical, oral, questions, literature, and others). We used the metrics defined and used during the first French parsing evaluation campaign EASy, which took place at the end of 2005 [8]. These metrics rely on both (nonrecursive) « chunks » and « relations » (dependencies between full words), for which the standard measures (precision, recall, f-measure) are applied. In this paper, we simply provide f-measures. Before discussing the results of these experiments, some precautions must be taken: – the conversion process described in this paper and its still preliminary implementation certainly contain errors, and we evaluate a variant of FRMG that relies on converted entries extracted from Lexicon-Grammar tables, not directly on LexiconGrammar entries from the tables; – the Lefff was developed in parallel with EASy campaigns, unlike Lexicon-Grammar tables; some choices in the EASy annotation guide may have influenced choices made during the development of the Lefff , whereas it is obviously not the case for Lexicon-Grammar tables; 9

S XPipe includes, among others, modules for (deterministic) sentence splitting and tokenization, as well as non-deterministic spelling error correction, named entity detection and identification of compound forms.

Using Lexicon-Grammar Tables for French Verbs in a Large-Coverage Parser

189

Table 1. EASy results for FRMGLefff and FRMGlglex , expressed in terms of f-measure. For reasons of space, figures are given for the whole EASy corpus and for only a sample of sub-corpora. Chunks Sub-corpus general_lemonde general_senat litteraire_2 medical_2 oral_delic_8 questions_amaryllis EASy corpus overall

Relations

FRMG Lefff FRMG lglex FRMG Lefff FRMG lglex

86.8% 82.7% 84.7% 85.4% 74.1% 90.5% 84.4%

82.8% 83.1% 81.5% 89.2% 73.6% 90.6% 82.3%

59.8% 56.7% 59.2% 62.4% 47.2% 65.6% 59.9%

56.9% 54.9% 56.3% 58.6% 48.5% 63.2% 56.6%

– as mentioned in the previous section, lglex had to be completed by various lexical entries from the Lefff , but other entries may still need to be added. Comparative results for both parsers are shown on Table 1, with detailed results for some illustrative sub-corpora. As can be seen, results are for now a bit better for FRMGLefff . We do not think that this result questions the relevance of using LexiconGrammar tables in a parser, especially given the above-mentioned precautions. In particular, we remain convinced that using as rich a lexical resource as possible is an efficient way to improve the quality of a parser, as has been shown for example by the results of the work described in [12]. However, parsing times are more than twice as high with FRMGlglex as with FRMGLefff (median average time per sentence: 0.62 s vs. 0.26 s), which is certainly a consequence of the higher average number of entries per lemma, which is three times higher in the lexicon extracted from lglex than in the Lefff (see above). In fact, these higher parsing times necessarily lead to a higher ratio of parsing failures because of reaching the timeout, which leads to the construction of partial parses whose quality can only be lower. Nevertheless, on several sub-corpora, FRMGlglex performs better in terms of chunk fmeasure; but results on relations are better with FRMG Lefff , apart from two sub-corpora. A careful study of the results shows the following interesting facts: –

FRMGlglex performs better than FRMGLefff on several relations, such as “adjective modifier” and “adverb modifier”, and also on two relations for which results are anyway quite low (“preposition modifier” and “apposition”); – the relation “(subject of object) attribute” is that for which the difference in terms of recall is the highest (34.0% vs. 58.5%); – the high number of verb arguments listed in lglex’s sub-categorization frames tends to fool the usual disambiguation heuristics according to which “arguments are preferred to modifiers”: any phrase that can be parsed as a verbal argument tends to be done in this way. For example, in a sentence such as [...] on estime que cette décision [ferait] dérailler le processus de paix (it is estimated that this decision would derail the peace process), FRMG lglex considers de paix (peacegenitive) as an argument of estimer (estimer qqch de qqch/qqn — to estimate something about something/somebody), whereas FRMG Lefff gets the correct parse.

190

E. Tolone and B. Sagot

In the short term, the following statement can be made. Many sentences get a full parse from FRMG lglex but not from FRMGLefff , and vice versa. For example, on the general_lemonde sub-corpus, 177 sentences are fully parsed by both parsers, 85 only by FRMGLefff , 76 only by FRMGlglex , and 111 by neither of them. Since experience shows that partial parses lead to worse results (approx. 10 points lower in terms of f-measure on EASy relations), an interesting experiment would be to couple both parsers in such a way that if only one of them builds a full parse for a given sentence, this parse is kept (what should be done in other cases remains an open question). The results of such a “meta-parser” should be better than those of both parsers. In the long term, it is important to benefit from this complementarity between both resources. It will be interesting to study the differences between errors made by both parsers, in particular thanks to techniques such as those described in [12]. This could lead to an improvement for both resources, and in particular the lexicon converted from lglex. Perhaps we will realize that most errors come from the conversion process; but some errors may come from errors in Lexicon-Grammar tables, and may therefore allow us to improve them.

7 Conclusion and Future Work In this paper, we introduced a methodology and a tool for converting the textual version of Lexicon-Grammar tables into an NLP lexicon based on the Alexina framework, i.e., in the same format as the Lefff syntactic lexicon for French, which is used by the FRMG parser. The relevance of the resulting lexicon is confirmed by its use for parsing the evaluation corpus of the French parsing evaluation campaign EASy. The first step described here has allowed us to identify several problems in the input data (tables and tables of classes), but also several simplifications and approximations in the conversion process. Therefore, there is space for significant improvements, that could eventually lead to the construction of a syntactic lexicon for French based on Lexicon-Grammar tables. Such a lexicon would improve the quality of existing tools and resources, e.g., by fusion with other lexical resources and by integration in a largecoverage parser. However, as mentioned in the introduction, we intend to enlarge the scope of our approach by applying the same approach on French to Lexicon-Grammar tables for other categories, but also on tables for other languages, as soon as the corresponding tables of classes become available. The next step, which should be taken soon, will deal with French predicative nouns, verbal idiomatic expressions and adverbs.

References 1. Boons, J.P., Guillet, A., Leclére, C.: La structure des phrases simples en français : Constructions intransitives, Droz, Geneva, Switzerland (1976) 2. Constant, M., Tolone, E.: A generic tool to generate a lexicon for NLP from LexiconGrammar tables. In: Proceedings of the 27th Lexis and Grammar Conference, L’Aquila, Italy (2008)

Using Lexicon-Grammar Tables for French Verbs in a Large-Coverage Parser

191

3. de La Clergerie, É.: DyALog: a tabular logic programming based environment for NLP. In: Proceedings of 2nd International Workshop on Constraint Solving and Language Processing (CSLP 2005), Barcelona, Spain (October 2005), ftp://ftp.inria.fr/INRIA/ Projects/Atoll/Eric.Clergerie/CSLP05.pdf 4. Gardent, C., Guillaume, B., Perrier, G., Falk, I.: Maurice Gross’ Grammar Lexicon and Natural Language Processing. In: Proceedings of the 2nd Language and Technology Conference (LTC 2005), Pozna´n, Poland (2005) 5. Gross, M.: Méthodes en syntaxe: Régimes des constructions complétives. Hermann, Paris (1975) 6. Guillet, A., Leclère, C.: La structure des phrases simples en français : Les constructions transitives locatives, Droz, Geneva, Switzerland (1992) 7. Paroubek, P., Pouillot, L.G., Robba, I., Vilnat, A.: EASy : campagne d’évaluation des analyseurs syntaxiques. In: Proceedings of the EASy workshop of TALN 2005, Dourdan, France (2005) 8. Paroubek, P., Robba, I., Vilnat, A., Ayache, C.: Data, Annotations and Measures in EASy, the Evaluation Campaign for Parsers of French. In: Proceedings of the 5th Language Resources and Evaluation Conference (LREC 2006), Genoa, Italy (2006) 9. Paumier, S.: De la reconnaissance de formes linguistiques à l’analyse syntaxique. Ph.D. thesis, Université Paris-Est Marne-la-Vallée, France (2003) 10. Sagot, B., Clément, L., de La Clergerie, É., Boullier, P.: The Lefff 2 syntactic lexicon for French: architecture, acquisition, use. In: Proceedings of the 5th Language Resources and Evaluation Conference (LREC 2006), Genoa, Italy (2006), http://atoll.inria.fr/ ~sagot/pub/LREC06b.pdf 11. Sagot, B., Danlos, L.: Améliorer un lexique syntaxique à l’aide des tables du LexiqueGrammaire : Constructions impersonnelles, Cahiers du Cental (2007) 12. Sagot, B., de La Clergerie, É.: Error mining in parsing results. In: Proceedings of ACL/COLING 2006, pp. 329–336. Association for Computational Linguistics, Sydney (2006), http://www.aclweb.org/anthology/P/P06/P06-1042 13. Sagot, B., Boullier, P.: S XPipe 2 : architecture pour le traitement présyntaxique de corpus bruts. Traitement Automatique des Langues (T.A.L.) 49(2) (2008) (to appear) 14. Thomasset, F., de La Clergerie, É.: Comment obtenir plus des méta-grammaires. In: Proceedings of TALN 2005, Dourdan, France (June 2005), ftp://ftp.inria.fr/INRIA/ Projects/Atoll/Eric.Clergerie/mg05.pdf 15. van den Eynde, K., Mertens, P.: Le dictionnaire de valence D ICOVALENCE : manuel d’utilisation (2006), http://bach.arts.kuleuven.be/dicovalence/ manuel_061117.pdf

Eﬀect of Overt Pronoun Resolution in Topic Tracking Fumiyo Fukumoto and Yoshimi Suzuki Interdisciplinary Graduate School of Medicine and Engineering, Univ. of Yamanashi, Kofu 400-8510, Japan {fukumoto,ysuzuki}@yamanashi.ac.jp

Abstract. This article focuses on overt pronouns which are related to a topic and an event in news stories, and studies issues on the eﬀect of their resolution in topic tracking. The antecedent of the pronoun is identiﬁed by using three linguistic features, morphological, syntactic and semantic knowledge. The morphological cues are part-of-speech information including named entities. Syntactic and semantic information is verbs and their subcategorization frames with selectional preferences. They are derived from the WordNet and VerbNet. The results on the TDT3 English show the usefulness of the overt pronoun resolution, especially for a small number of positive training data.

1

Introduction

With the exponential growth of information on the Internet, it is becoming increasingly diﬃcult to ﬁnd and organize relevant materials. Topic tracking, i.e., it starts from a few sample stories and ﬁnds all subsequent stories that discuss the target topic, is a research to attack the problem [7]. One of the major problems in the tracking task is how to make a clear distinction between a topic and an event. Here, a topic is something that occurs at a speciﬁc place and time associated with some speciﬁc actions, and it becomes the background among documents. An event refers to the subject of a story itself, i.e., a writer wants to express, in other words, notions of who, what, where, when, why and how in the story. Therefore, an event drifts, but a topic does not. For example, in the stories of “SwissAir111 crash” from the TDT3 corpus, the event includes the crash and ensuing investigation, plans to compensate the victims’ families, any safety measures proposed or adopted as a direct result of this crash, while the topic is SwissAir Flight 111 crashes. A wide range of statistical and machine learning (ML) techniques have been applied to topic tracking, including k-Nearest Neighbor classiﬁcation, Decision Tree induction [10], relevance feedback method of IR [4,16], unsupervised and supervised clustering [17], and a variety of Language Modeling [15]. The common framework of these techniques is to represent training and test stories as a set of weighted content words. There are quite a lot of work on term weighting methods applied to topic tracking tasks, such as frequency-based weighting, tf∗idf method and Information Gain method. Allan et al presented a method to extract terms Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 192–202, 2011. c Springer-Verlag Berlin Heidelberg 2011

Eﬀect of Overt Pronoun Resolution in Topic Tracking

193

called “surprising” features that have unusual occurrence characteristics [8], and implemented an adaptive version of the tracking system that may rebuild th query after it “tracks” a news story on a given event. They showed how adaptive tracking based on surprising features is a useful method for capturing those features in story sequences about some events such as disaster or crime events, and for reducing the number of training stories needed. However, one encounters quite a large number of referring expressions including overt pronouns instead of using the same nouns repeatedly. Moreover, they sometimes show topics related to a speciﬁc time and place, and events referred to notions of who (person), where (place), when (time) including what, why and how in a story. In this article, we focus on overt pronouns and investigate the eﬀect of their resolution in the topic tracking task on broadcast news stories. The antecedent of a pronoun is identiﬁed by using three linguistic features, morphological, syntactic and semantic knowledge. The morphological cue is part-of-speech information including named entities. Syntactic and semantic information is verbs and their sub-categorization frames with selectional preferences extracted from the WordNet1 and VerbNet2 . In the tracking task, we applied term weighting and adaptation to ﬁnd stories that discuss the target topic from incoming stories. The rest of the article is organized as follows. The next section provides an overview of existing techniques, which are accommodated the fact that an event and a topic are diﬀerent from each other. After describing pronoun resolution, we explain a tracking method. Finally, we report some experiments using the TDT3, and end with a discussion of evaluation.

2

Related Work

Topic tracking is similar to information ﬁltering task [20] of Information Retrieval (IR), since both handle shift, i.e., topic tracking handles subject shift and information ﬁltering handles a change of user’s interest. The earliest work on information ﬁltering which focused on shift is the work of Allan [6]. He proposed a method of incremental relevance feedback for information ﬁltering which is based on weighting stories. The result of the method which was tested on the data from the TREC routing experiments [2] has shown satisfactory performance. However, TREC data provides a large amount of training data with queries and relevance judgments. For the TDT conditions, on the other hand, the number of labeled positive training examples which are allowed is very small (one to four). More recently, Elsayed [22], Connell [16], and Allan’s group explored on-line adaptive ﬁltering approaches based on the threshold strategy. The basic idea behind their work is that stories closer together in the stream are more likely to discuss related topics than stories further apart. The method presented by Allan’s group at UMass is based on adaptation, the traditional vector space model 1 2

http://wordnet.princeton.edu/ http://verbs.colorado.edu/~mpalmer/projects.verbnet.html

194

F. Fukumoto and Y. Suzuki

with tf∗idf weighting, and relevance models [16]. Adaptation allows the addition of the incoming story to the topic representation and recomputes the topic centroid. Adaptation uses an additional threshold which determines whether a new story is similar enough to the centroid to be added to the topic. A new centroid is computed from the story vectors each time a new story is added to the topic. The method proposed in this article is similar to Allan’s group which uses term weighting and adaptation, while these process are conducted for the stories, where the antecedent of each pronoun in the stories was disambiguated. In the context of pronoun resolution, most work on anaphora resolution has focused on pronominal anaphora, often achieving good accuracy. Kennedy and Boguraev [1], Mitkov [21], and Strube, Rapp, and Mueller [19], for example, report accuracies of 75.0%, 89,7%, and an F-measure of 82.8% for personal pronouns, respectively. Markert and Nissim studied the issues on nominal anaphora with full lexical heads [14]. They compare an algorithm that relies on links encoded in the manually created lexical hierarchy WordNet and an algorithm that mines corpora by means of shallow lexico-semantic patterns. They concluded that the Web-based method alleviated the lexical knowledge gap often encountered in anaphora resolution and handled examples with context-dependent relations between anaphora and antecedent, while the knowledge encoded in WordNet is often insuﬃcient, especially for anaphor-antecedent relations that exploit subjective or context-dependent knowledge. Like much previous work on anaphora resolution, we focus on overt pronoun resolution, and especially investigate the inﬂuence of the results to topic tracking.

3 3.1

Pronoun Resolution Pre-processing

A prerequisite for pronoun resolution is to obtain possible cues in an input story. To determine the cues, a pipeline of NLP modules is used. It consists of part-ofspeech tagging including noun phrase identiﬁcation, named entity recognition, syntactic analysis, and semantic class determination. We used a part-of-speech tagger developed by Schmid [5]. Our named entity recognition module uses Stanform NER [13]. For both noun phrases determined by part-of-speech tagging and named entities, they are merged in such a way that if the noun phrase overlaps with a named entity, the noun phrase boundaries will be adjusted to subsume the named entity. Moreover, the gender of a cue is determined for designators and pronouns such as “Mr” and “he”. Next, we used a syntactic parser [3] to extract verbs and their subcategorization frames from each sentence in a story. Finally, we assigned selectional preferences to the extracted verbs and their subcategorization frames. To do this, we used VerbNet, and the WordNet sense dictionary. We assumed that the semantic class of each noun phrases (noun) is the ﬁrst sense of the head noun. Since WordNet orders the senses of a noun by their frequency, this is equivalent to choosing the most frequent sense as the semantic class for each noun.

Eﬀect of Overt Pronoun Resolution in Topic Tracking

3.2

195

Identiﬁcation of Antecedent

For each overt pronoun in test data, we extracted features, i.e., number, gender, named entity and semantic class. The semantic class of each overt pronoun is determined by its main verb and subcategorization frame with selectional preferences. The number and gender of the overt pronoun “he”, for example, is “singular” and “male”, respectively. The named entity recognition assigned “person” to the pronoun “he”. For each overt pronoun, if its preceding noun matches features, then propose this noun as antecedent. Consider the following three sentences from TDT3 corpus including three overt pronouns, “he”, “his”, and “he”: (1) Prime Minister Tony Blair/person plans a ﬁve-day visit next week to Hong Kong and China, where he/singular/male/+animate, +organization is expected to address the continued Chinese occupation of Tibet. (2) Blair/person is scheduled to arrive in Beijing on Tuesday. (3) During his/singular/male visit, he/singular/male/+animate, +organization plans to meet with Premier Zhu Rongji to address to Asian ﬁnancial crisis, human rights, Hong Kong and International terrorism, the prime minister’s oﬃce said. “he” in sentence (1) has a singular and male features. It is also tagged +animate, +organization by using VerbNet information of the verb “expert”. The semantic class of the preceding noun “tony Blair” is assigned “person” by using the WordNet. Then the anteceding of “he” is regarded as “Tony Blair”. Similarly, the anteceding of “he” in sentence (3) is “Blair” in sentence(2). We note that the semantic class of “his” in sentence(3) is not assigned. Because there is no main verb of this pronoun. In this case, the anteceding of the pronoun is searched by using only the number and gender features.

4

Tracking Based on Term Weighting and Adaptation

Allan et al. at UMass proposed a method using adaptation, the traditional vector space model, and relevance models [9]. Adaptation allows addition of the incoming story to the topic representation and recomputes the topic centroid. Adaptation uses an additional threshold that determines whether a new story is suﬃciently similar to the centroid to be added to the topic. A new centroid is computed from the story vectors each time a new story is added to the topic. Their method is an enhancement based on several years of TDT tracking research, while there is other research work on topic tracking, such as the approaches proposed by CMU and ICT in 2004 [23,18]. The method proposed in this article is similar to Allan et al.’s method, while our method for collecting relevant stories is based on the results of anaphora resolution. Each story is represented as a vector of terms with tf∗idf weights in an ndimensional space, where n is the number of terms in the collection [8]. Whether each test story is positive is judged using the distance (measured by cosine similarity) between a vector representation of the test story and each centroid g of the clusters. The centroid g for each cluster is calculated as follows:

196

F. Fukumoto and Y. Suzuki p

p

i=1

i=1

1 1 g = (g1 , · · · , gn ) = ( xi1 , · · · , xin ) p p

(1)

where xij (1 ≤ j ≤ n) is the tf∗idf weighted value of term j in the story xi . The test story is declared positive if the cosine similarity between the test story and the centroid with positive stories has the largest value. This procedure is repeated until the last test story is judged. The test story judged to be positive is added to the initial positive training stories, and tf∗idf weights are recomputed. This procedure is repeated until the last test story is judged. Similar to Allan’s group, the number of stories for adaptation is less than 100.

5

Experiments

We chose the TDT3 English corpora as our gold standard corpora, and set the evaluation conditions used in the TDT benchmark evaluation. The stories of TDT3 were collected from 8 sources (34,600 stories) including the ABC News for the period of Oct. through Dec. 1998. A set of 60 target topics were manually identiﬁed as topics (2000 Evaluation Topics)3 . 5.1

Anaphora Resolution

Experiments Set Up We used 10 stories from every 8 sources, and conducted an anaphora resolution experiment by applying our method to these data. The test stories are shown in Table 1. As shown in Table 1, we tested 1,082 pronouns in all. We used a part-of-speech tagger developed by Schmid[5]. We also used Stanford NER[13] to identify named entity. We used a syntactic parser[3] to extract verbs and their subcategorization frames from each sentence in a story. We deﬁned “success rate” as the number of pronouns successfully resolved, divided by the total number of resolved pronouns, and used it as an evaluation measure. Results Table 2 shows the results. “# of correct pro.” refers to the number of correct pronouns. The evaluation was made by three humans. The classiﬁcation is correct if the majority of three human judges agrees. The overall results obtained by our method attained a success rate for pronoun resolution of 62.8%. We can see from Table 2 that the accuracy depends on the source type. The overall results using broadcast TV and radio sources were worse than those of newswire. One reason behind this lies that broadcast TV and radio sources often consists of interviews, which include many cataphora and formal 3

http://projects.ldc.upenn.edu/TDT3/

Eﬀect of Overt Pronoun Resolution in Topic Tracking

197

Table 1. Test Data Source type Sources Pronoun Newswire Associated Press Worldstream (APW) 78 New York Times (NYT) 396 Broadcast PRI The World (PRI) 30 Radio VOA English (VOA) 60 Broadcast CNN Headline News (CNN) 39 TV ABC World News Tonight (ABC) 131 NBC Nightly News (NBC) 137 MSNBC News (MNB) 214 Total 1,082 Table 2. Results of Anaphora Resolution Sources # of pro. # of correct pro. Success rate APW 78 59 0.756 NYT 396 288 0.727 PRI 30 18 0.600 VOA 60 32 0.533 CNN 39 18 0.462 ABC 131 79 0.603 NBC 137 72 0.526 MNB 214 131 0.612 Total 1.082 679 0.628

subject. the remainder of the errors were due to syntactic parsing with respect to the identiﬁcation of complex noun phrases (causing an error rate of around 2%). Moreover, we found resource scarcity problem, i.e., words which did not appear in the WordNet resource causing high error rate, around 22%. therefore, we should be able to obtain further advantages in eﬃcacy in our method by incorporating a method to estimate the semantic class of the words which were not appeared in the WordNet. 5.2

Topic Tracking

Experiments Set Up We applied anaphora resolution to TDT3 consisting of 34,600 stories, and used the results to test topic tracking. We set the evaluation measures used in the TDT benchmark evaluations. “Precision” stands for the ratio of correct assignments by the system divided by the total number of system’s assignments. “F” (pooled average) is a measure that balances recall and precision, where recall denotes the ratio of correct assignments by the system divided by the total number of correct assignments. “Miss” denotes Miss rate, which is the ratio of the stories that were judged as YES but were not evaluated as such for the run in question. “F/A” shows false alarm rate, which is the ratio of the stories judged as NO

198

F. Fukumoto and Y. Suzuki Table 3. Basic Results Nt 1 2 4

Miss 67% 46% 38%

TDT3 (60 topics) F/A Recall Precision F M IN 0.04% 33% 66% 0.44% 0.134 0.05% 54% 58% 0.56 0.086 0.05% 62% 55% 0.59 0.086

but were evaluated as YES. The DET curve plots misses and false alarms, and better performance is indicated by curves more to the lower left of the graph. The detection cost function (CDet ) is deﬁned by Eq (2). CDet = (CMiss ∗ PMiss ∗ PT arget + CF a ∗ PF a ∗ (1 − PT arget )) PMiss = #M isses/#T argets PF a = #F alseAlarms/#N onT argets

(2)

CMiss , CF a , and PT arget are the costs of a missed detection, false alarm, and priori probability of ﬁnding a target, respectively. CMiss , CF a , and PT arget are usually set to 10, 1, and 0.02, respectively. The normalized cost function is deﬁned by Eq (3), and lower cost scores indicate better performance. More information about these measures can be found in [12]. (CDet )N orm = CDet /M IN (CMiss ∗ PT arget , CF a ∗ (1 − PT arget ))

(3)

Basic Results Table 3 summaries the tracking results, and Fig. 1 shows DET curves varying the number of initial positive training stories. The threshold of cosine similarity is set to 0.3. “Nt” denotes the number of initial positive training stories. “Miss” denotes Miss rate, and “F/A” shows false alarm rate. “Prec” stands for precision, and “F” (pooled avg) is a measure that balances recall and precision. MIN denotes M IN (CDet )N orm which is the value of (CDet )N orm at the best possible threshold. We can see from Table 3 and Fig. 1 that the method correctly extracted stories related to the target topic even for a small number of positive training stories, as the value of M IN (CDet )N orm is at most 0.134. The Eﬃcacy of Anaphora Resolution The contribution of anaphora resolution is best explained by looking at two results: the results with and without anaphora resolution. In each method, the top 100 terms with tf∗idf are used as a vector representation of each story, and cosine similarity between training and each test story is calculated. If the value of cosine similarity between them is larger than a certain threshold value, the test story is regarded as positive, and is added to the initial positive training stories. Fig. 2 illustrates DET curves obtained by two methods. To make some comparison possible, only the Nt = 1 is given for each. The threshold value of

Eﬀect of Overt Pronoun Resolution in Topic Tracking

199

90 random performance Nt=1 Evaluated at Nt=2 Evaluated at Nt=4 Evaluated at

80

Miss Probability (in %)

60

40

20

10 5

2 1

.01 .02 .05 0.1 0.2

0.5

1

2

5 10 20 False Alarm Probability (in %)

40

60

80

90

Fig. 1. Tracking Result again # of Nt Values

cosine similarity was 0.3. We found small advantages in the DET curves for our method compared with a baseline result without anaphora resolution. Table 4 shows examples of terms (top ﬁve) extracted by our method and the method without anaphora resolution for some of the topics (Nt = 4). “ID” denotes topic number deﬁned by the TDT. The term is shown along with the number of times it occurs in the Nt = 4 training set, the number of those 4 stories it occurs in . Bold font indicates terms which are not appeared in the result without anaphora resolution but appeared in the result obtained by our method. As can be clearly seen from Table 4, Some terms such as “Cambodian”, “Cambodia” and “Arafat” that are related to the contents of each topic were not appeared in the result without anaphora resolution. Similarly, the extracted terms by our method with higher frequency than the method without anaphora resolution, e.g., “Israel” in ID30010, and “Blair” in ID30018, are related to the content of each topic, i.e., “Topic name” in Table 4 that is provided by the TDT project. The frequency of “Blair” obtained by our method was 36, while that of the method without anaphora resolution was 25. These results support our claim on the usefulness of anaphora resolution in topic tracking task. Comparison with Related Work Table 5 illustrates the performance of our method compared to the other research sites which tested on the 2001 training topics, i.e., 60 topics in all, and 30 each drawn from the 1999 and 2000 evaluation sets (withheld topics). Table 5 shows the result when Nt is set to 1. Each site in Table 5 was reported by Fiscus [11]4 . 4

http://www.nist.gov/speech/tests/tdt/tdt2001/PaperPres/Nistpres/ NIST-presentation-v6_files/frame.htm

200

F. Fukumoto and Y. Suzuki 90 random performance Without anaphora resolution Evaluated at Anaphora resolution Evaluated at

80

Miss Probability (in %)

60

40

20

10 5

2 1

.01 .02 .05 0.1 0.2

0.5

1

2

5 10 20 False Alarm Probability (in %)

40

60

80

90

Fig. 2. The Eﬃcacy of Anaphora Resolution Table 4. Examples of Terms (Top 5) ID

Topic name

30001 Cambodian government coalition

30010 Car bomb in Jerusalem

30013 Dalai Lama visits U.S.

30014 Nigerian gas line ﬁre

30018 Tony Blair visits China in Oct. 30021 Thai airbus crash

with Anaphora Resolution (term,tf,df) (Hum,9,4) (Sen,9,4) (opposition,15,3) (Cambodian,5,4) (Cambodia,19,4) (cabinet,13,4) (Palestinian,16,3) (Israel,17,4) (Arafat,5,4) (radio,3,1) (Chinese,16,4) (Lama,22,4) (Dalai,14,4) (China,7,2) (Tibet,14,4) (pipeline,7,2) (oil,8,2) (oﬃcial,9,3) (sabotage,5,3) (petroleum,4,1) (Blair,36,4) (Hong,10,4) (Kong,10,4) (British,12,4) (visit,14,4) (crash,9,4) (Thani,6,2) (Surat,6,2) (airport,6,3) (Thai,9,4)

Without Anaphora Resolution (term,tf,df) (Hum,9,4) (Sen,8,4) (be,27,4) (opposition,13,3) (Sam,7,1) (cabinet,13,4) (Palestinian,15,3) (Israel,15,4) (radio,3,1) (agreement,9,4) (Chinese,16,4) (Lama,14,4) (Dalai,14,4) (China,7,2) (Tibet,13,4) (pipeline,7,2) (oil,7,2) (oﬃcial,9,3) (petroleum,3,1) (sabotage,5,3) (Blair,25,4) (Hong,10,4) (Kong,10,4) (British,12,4) (visit,12,4) (crash,9,4) (Thani,4,2) (Surat,4,2) (airport,6,3) (Thai9,4)

Eﬀect of Overt Pronoun Resolution in Topic Tracking

201

Table 5. Results by our method and other sites (Nt =1) TDT3 (30 topics, Nt =1) Site M IN Site M IN LIMSI 0.1203 RMIT3 0.3636 Ulowa1 0.1857 UMASSI 0.1468 Ana (adaptation) 0.1389

The threshold of cosine similarity used in our method is 0.3 which was learned from 2000 evaluation topics. We can see that the performance by our method is competitive to other sites, since the top site is LIMSI (0.1203), and the result by our method (0.1389) is very similar to UMASSI (0.1468).

6

Conclusion

We presented a method for topic tracking by using the results of overt pronouns resolution. The main contribution of this article is to investigate the eﬀect of pronoun resolution in the topic tracking task. The empirical results are very encouraging and show that the method is eﬀective even for a small number of positive training stories. Future work includes (i) extending the pronoun resolution method to estimate the semantic class of the words which were not appeared in the WordNet, (ii) resolving other anaphors such as nominal anaphora, and (iii) applying the method to the TDT4 corpora for quantitative evaluation.

References 1. Kennedy, C., Branimir, N.: Anaphora for Everyone: Pronominal Anaphora Resolution without a Parser. In: Proc. of the 16th International Conference on Computational Linguistics, pp. 113–118 (1996) 2. Harman, D.: Overview of the forth Text REtrieval Conference (TREC4). In: Proc. of the 4th Text REtrieval Conference (1996) 3. Lin, D.: Automatic Retrieval and Clustering of Similar Words. In: Proc. of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, pp. 768–774 (1998) 4. Oard, D.W.: Topic Tracking with the PRISE Information Retrieval System. In: Proc. of DARPA Workshop (1999) 5. Schmid, H.: Improvements in Part-of-Speech Tagging with an Application to German. In: Proc. of the EACL SIGDAT Workshop (1995) 6. Allan, J.: Incremental Relevance Feedback for Information Filtering. In: Proc. of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 11–22 (1996) 7. Allan, J., Carbonell, J., Doddington, G., Yamron, J., Yang, Y.: Topic Detection and Tracking Pilot Study Final Report. In: Proc. of DARPA Workshop (1998) 8. Allan, J., Papka, R., Lavrenko, V.: On-line New Event Detection and Tracking. In: Proc. of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 37–45 (1998)

202

F. Fukumoto and Y. Suzuki

9. Allan, J., Lavrenko, V., Connell, M.E.: A Month to Topic Detection and Tracking in Hindi. ACM Trans. on Asian Language Information Processing (TALIP) 2(2), 85–100 (2003) 10. Carbonell, J., Yang, Y., Laﬀerty, J., Brown, R.D., Pierce, T., Liu, X.: CMU Report on TDT-2: Segmentation, Detection and Tracking. In: Proc. of DARPA Workshop (1999) 11. Fiscus, J.: Overview of the TDT 2001 Evaluation and Results. In: Workshop in TDT 2001 (2001) 12. Fiscus, J.G., Doddington, G.R.: Topic Detection and Tracking Evaluation Overview. In: Allan, J. (ed.) Topic Detection and Tracking. Kluwer Academic Publisher, Dordrecht (2002) 13. Finkel, J.R., Grenager, T., Manning, C.: Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In: Proc. of the 43nd Annual Meeting of the Association for Computational Linguistics, pp. 363–370 (2005) 14. Markert, K., Nissim, M.: Comparing Knowledge Sources for Nominal Anaphora Resolution. Computational Linguistics 31(3), 367–401 (2005) 15. Larkey, L.S., Feng, F., Connell, M., Lavernko, V.: Language-speciﬁc Model in Multilingual Topic Tracking. In: Proc. of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 402–409 (2004) 16. Connell, M., Feng, A., Kumaran, G., Raghavan, H., Shah, C., Allan, J.: UMASS at TDT 2004. In: Proc. of DARPA Workshop (2004) 17. Franz, M., McCarley, J.S.: Unsupervised and Supervised Clustering for Topic Tracking. In: Proc. of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 310–317 (2001) 18. Yu, M.Q., Luo, W.H., Zhou, Z.T., Bai, S.: ICT’s Approaches to HTD and Tracking at TDT2004. In: Proc. of the Topic Detection and Tracking Workshop (2004) 19. Strube, M., Stefan, R., Christoph, M.: The Inﬂuence of Minimum Edit Distance on Reference Resolution. In: Proc. of the 2002 Conference on Empirical Methods in Natural Language Processing, pp. 312–319 (2002) 20. Belkin, N.J., Croft, W.B.: Information ﬁltering and Information Retrieval: Two sides of the same coin? Communications of the ACM 35(2), 29–38 (1992) 21. Mitkov, R.: Robust Pronoun Resolution with Limited Knowledge. In: Proc. of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, pp. 869–875 (1998) 22. Elsayed, T., Oard, D.W., Doermann, D.: TDT-2004: Adaptive Topic Tracking at Maryland. In: Proc. of DARPA Workshop (2004) 23. Zhang, Y., Callan, J.: CMU DIR Supervised Tracking Report. In: Proc. of DARPA Workshop (2004)

Sentiment Intensity: Is It a Good Summary Indicator? Mijail Kabadjov1 , Alexandra Balahur2 , and Ester Boldrini2 1

2

Joint Research Centre, European Commission, Via E. Fermi 2749, Ispra (VA), Italy [email protected] Departamento de Lenguajes y Sistemas Inform´ aticos Universidad de Alicante Apartado de Correos 99, E-03080, Alicante, Spain {abalahur,eboldrini}@dlsi.ua.es

Abstract. In this paper we address the question of whether “very positive” or “very negative” sentences from the perspective of sentiment analysis are “good” summary sentences from the perspective of text summarisation. We operationalise the concepts of very positive and very negative sentences by using the output of a sentiment analyser and evaluate how good a sentence is for summarisation by making use of standard text summarisation metrics and a corpus annotated for both salience and sentiment. In addition, we design and execute a statistical test to evaluate the aforementioned hypothesis. We conclude that the hypothesis does not hold, at least not based on our corpus data, and argue that summarising sentiment and summarising text are two diﬀerent tasks which should be treated separately.

1

Introduction

Recent years have marked the birth and the expansion of the Social Web, the web of interaction and communication. Within this context, people express opinion on a high variety of topics – from economics and politics to science and technology to cooking and gardening – that are of interest to a large community of users. People propose and debate these topics over forums, review sites and blogs, a series of newly emerging textual genres that are growing in volume and number of topics addressed daily. Moreover, such texts can oﬀer a genuine image of the opinion expressed on diﬀerent interesting topics by all the social categories, age groups, in all regions of the world. This can be useful to conduct market research, reputation tracing, product assessment, company prospects, research on public image, social media analysis or can simply satisfy the curiosity of ﬁnding out what people think of a certain issue. However, the large quantities of such data that is available cannot be read entirely. Therefore, automatic systems that are able to detect opinion, classify it and summarise it have to be built. Such a system would, for example, given a blog thread (the sequence of texts containing the post on a subject and the subsequent comments on it made by diﬀerent “bloggers”), Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 203–212, 2011. c Springer-Verlag Berlin Heidelberg 2011

204

M. Kabadjov, A. Balahur, and E. Boldrini

analyse its content as far as opinion is concerned and subsequently summarise the classes of opinions expressed (i.e., arguments pro and against the topic). While opinion mining and text summarisation have been intensely researched on separately, up until now there has been little research done at the intersection of both, namely, opinion summarisation. Therefore, the aim of our work presented herein is to study the manner in which opinion can be summarised. In this paper we address the question of whether very positive or very negative sentences from the perspective of sentiment analysis are good summary sentences from the perspective of text summarisation. We test our diﬀerent hypotheses on a corpus of blog threads from diﬀerent domains and discuss our ﬁndings. The reminder of the paper is organised as follows: in section §2 we summarise the state of the art on sentiment analysis and summarisation; in section §3 we describe our approach of gauging the usefulness of sentiment intensity for summarisation; next, in section §4, we discuss our experimental results; and ﬁnally, we draw the conclusions from this work and give pointers to future work.

2

Related Work

Whilst there is abundant literature on text summarisation [16,14,11] and sentiment analysis [3,20,10], there is still limited work at the intersection of these two areas [23], in particular, at studying the relationship between sentiment intensity and summarisation prominence. For the ﬁrst time in 2008 there was a Summarisation Opinion Pilot track at the Text Analysis Conference organised by the US National Institute of Standards and Technology (NIST). Most approaches to the problem were underpinned by already existing summarisation systems; some added new features (sentiment, pos/neg sentiment, pos/neg opinion) to account for positive or negative opinions – CLASSY [7], CCNU [13], LIPN [4] and IIITSum08 [26] – some proposed eﬃcient polarity-based methods focusing on the retrieval and ﬁltering stage – DLSIUAES [1] – or on separating information rich clauses – italica [8]. Overall, previous work on opinion mining can be broadly classiﬁed in two categories: sentiment classiﬁcation at the document level and at the sentence level. Research done in document-level sentiment classiﬁcation used supervised [6] and unsupervised methods [25], using rating scales [20] and scoring of features [9]. Work on sentence-level sentiment classiﬁcation used bootstrapping techniques [21], ﬁnding strength of opinions [27], summing up orientations of opinion words in a sentence [17] and identifying opinion holders [23]. Finally, ﬁner-grained, feature-based opinion summarisation was put forward in [15].

3

Sentiment Intensity and Summarisation

Our approach follows a simple intuition: when people express very negative or very positive sentiment, for example, in blogs, they might be also conveying important and valuable information that is somewhat more salient than other comments. The sub-area of Natural Language Processing concerned with identifying salient information in text documents is Text Summarisation, hence, we

Sentiment Intensity: Is It a Good Summary Indicator?

205

decided to formalise our intuition in the context of text summarisation and make use of standard methodology from that area. In addition, we cast the above intuition as a statistical hypothesis test where the null hypothesis we seek to reject is the opposite of our intuition, that is, the sentiment intensity of salient blog comments is no diﬀerent from the sentiment intensity of non-salient comments. In order to carry out experiments to study in a quantitative manner whether sentiment intensity is a useful summary indicator, three things are needed: a sentiment analysis system capable of producing a sentiment intensity score for a given blog comment, a summarisation algorithm exploiting this sentiment intensity score and a reference corpus annotated for both sentiment and salience (i.e., gold standard data). Next, we describe each of those components and the design of the hypothesis test. 3.1

Sentiment Annotated Corpus

Annotation Process. The corpus we employed in this study is a collection of 51 blogs extracted from the Web, a limited dataset which allows for a preliminary study in the ﬁeld. The blog posts are written in English and have the same structure, which is the following: there is an initial post created by the author, containing a piece of news and their opinion on it; subsequently, bloggers reply expressing their opinions about the topic. In most of the cases, commenting posts are the most subjective texts even if the authors, in their initil intervention may also express their point of view on the topic of the post. Blogs can also contain multi-modal information. However, the aim of our study is to summarize opinions expressed in texts and therefore we only annotated the textual content of the blogs. The annotation we performed on this blog corpus contains several elements: ﬁrst, we indicated the URL from which the thread was extracted, we then included the initial annotated piece of news and the labeled user comments with an opinion annotation scheme that is proper for blogs - EmotiBlog. In order to delimitate our work, we selected only ﬁve major topics; we gave priority to the most relevant threads, that contained a large amount of posts in order to have a considerable amount of data. We chose some of the topics that we considered relevant: economy, science and technology, cooking, society and sport. Due to space constraints, speciﬁc details on the corpus are omitted here (see [2] for more details). 3.2

Sentiment Analysis

The ﬁrst step we took in our approach (see ﬁgure 1) was to determine the opinionated sentences, assign each of them a polarity (positive or negative) and a numerical value corresponding to the polarity strength (the higher the negative score, the more negative the sentence and vice versa). Given that we are faced with the task of classifying opinion in a general context, we employed a simple, yet eﬃcient approach, presented in [3]. At the present moment, there are different lexicons for aﬀect detection and opinion mining. In order to have a more

206

M. Kabadjov, A. Balahur, and E. Boldrini

Fig. 1. The Sentiment Analysis Process

extensive database of aﬀect-related terms, in the following experiments we used WordNet Aﬀect [24], SentiWordNet [12], MicroWNOp [5]. Each of the employed resources were mapped to four categories, which were given diﬀerent scores: positive (1), negative (-1), high positive (4) and high negative (-4). As shown in [3], these values performed better than the usual assignment of only positive (1) and negative (-1) values. First, the score of each of the blog posts was computed as sum of the values of the words identiﬁed; a positive score leads to the classiﬁcation of the post as positive, whereas a ﬁnal negative score leads to the system classifying the post as negative. Subsequently, we performed sentence splitting using Lingpipe and classiﬁed the obtained sentences according to their polarity, by adding the individual scores of the aﬀective words identiﬁed. 3.3

Summarisation Algorithm Based on Sentiment Intensity

A straightforward summarisation algorithm that exploits sentiment intensity can be deﬁned as follows: 1. Rank all comments according to their intensity for a given polarity 2. Select highest n comments It is important to point out here that positive and negative polarity comments are treated separately, that is, we produce one summary for all positive comments and one for all negative comments for a given blog thread (see ﬁgure 2).

Fig. 2. The Summarisation Process

Sentiment Intensity: Is It a Good Summary Indicator?

207

We ran this algorithm at two commonly used compression rates: 15% and 30%. That is, we produce two summaries for each polarity for each thread, one choosing the top 15% and one the top 30% of all comments. The results are presented and discussed in the next section, §4. 3.4

Hypothesis Test for Sentiment Intensity Usefulness

In addition to a standard summarisation evaluation we evaluate the hypothesis that very positive or very negative comments are good choices to be included in a summary by casting the problem as a statistical hypothesis test. Student’s t-test. We deﬁne the following setting in order to execute an independent two-sample one-tailed t-test of unequal sample sizes and equal variance: ¯1 − X ¯2 = 0 1. Null hypothesis, H0 : X ¯1 > X ¯2 Alternative hypothesis, H1 : X 2. Level of significance: α = 0.05 3. t statistic: ¯1 − X ¯2 X t= (1) SX1 X2 · n11 + n12

where

2 + (n − 1)S 2 (n1 − 1)SX 2 X2 1 (2) n1 + n2 − 2 4. Criterion: Reject the null hypothesis in favour of the alternative hypothesis if t > tν,α

SX1 X2 =

where ν = n1 + n2 − 2 (degrees of freedom) and t∞,0.05 = 1.645 In equations 1 and 2, n is the number of sample points, 1 is group one and 2 is group two. More speciﬁcally, in our case group one is composed of all the comments annotated as salient in our corpus (i.e., gold summary comments) and group two is composed of all the comments that were not annotated (i.e., gold non-summary comments). Furthermore, we further slice the data upon polarity (as produced by the sentiment analysis tool), so we have two samples (i.e., group one and group two) for the case of positive comments and two samples for the case of negative comments. For example, out of all the comments that were assigned a positive score by the sentiment analysis tool there are those that were also annotated as positive by the annotators – these constitute group one for the positive polarity case – and those that were not annotated at all – these constitute group two for the positive polarity case.3 The same thinking applies for the negative polarity case. 3

Certainly, in order to use gold polarity alongside the score produced by the sentiment analysis tool as we do, we had to ﬁrstly automatically align all the automatically identiﬁed sentences with the annotated comments. The criterion for alignment we used was that at least 70% of the words in an automatically identiﬁed sentence are contained in an annotated comment for it to inherit the gold polarity of that comment (and by virtue of that to be considered a gold summary sentence).

208

4

M. Kabadjov, A. Balahur, and E. Boldrini

Experimental Results

We ﬁrst discuss the performance of the sentiment recognition system followed by the sentiment-intensity-based summarisation performance. 4.1

Sentiment Analysis

Performance results of the sentiment analysis are shown in table 1. Table 1. Sentiment analysis performance System Precision Recall F1 Sentneg 0.98 0.54 0.69 Sentpos 0.07 0.69 0.12

The ﬁrst thing to note in table 1 is that the sentiment analysis tool is doing a much better job at identifying negative comments (F 1 = 0.69) than positive ones (F 1 = 0.12), the main problem with the latter being a very low precision (P = 0.07). One possible reason for this is an insuﬃcient number of annotated positive examples (there were much more negative examples than positive ones in the corpus). We discuss in the next section whether this substantial diﬀerence in performance between the negative and positive cases has an impact on the subsequent analysis. 4.2

Summarisation

Performance results of the summariser are shown in table 3. We used the standard ROUGE evaluation [19] also used for the Text Analysis Conferences. We include the usual ROUGE metrics: R1 is the maximum number of co-occurring unigrams, R2 is the maximum number of co-occurring bigrams, RSU4 , is the skip bigram measure with the addition of unigrams as counting unit, and ﬁnally, RL is the longest common subsequence measure [19]. In all cases we present the average F1 score for the given metric. There are ﬁve rows in table 3: the ﬁrst (SISummneg at 15%) is the performance of the sentiment-intensity-based summariser (SISumm) on the negative posts at 15% compression rate; the second (SISummpos at 15%) presents the performance of SISumm on the positive posts at 15% compression rate; the third (SISummneg at 30%) is the performance of the SISumm on the negative posts at 30% compression rate; the fourth (SISummpos at 30%) presents the performance of SISumm on the positive posts at 30% compression rate; and ﬁnally, the Table 2. Ingredients for a two-sample t-test; unequal sample sizes, equal variance 2 2 ¯1 ¯2 Polarity X X n1 n2 SX SX t statistic 1 2 Negative −3.95 −4.04 1092 1381 10.13 10.5 0.021 Positive 4.37 4.26 48 1268 9.3 28.03 0.036

Sentiment Intensity: Is It a Good Summary Indicator?

209

Table 3. Summarisation performance in terms of ROUGE scores System SISummneg at 15% SISummpos at 15% SISummneg at 30% SISummpos at 30% T opSummT AC08 BottomSummT AC08

R1 0.07 0.22 0.17 0.19 – –

R2 0.03 0.03 0.06 0.03 0.111 0.069

RSU4 0.03 0.03 0.06 0.03 0.142 0.081

RL 0.07 0.19 0.16 0.17 – –

ﬁfth and the sixth rows correspond to the oﬃcial scores of the top and bottom performing summarisers at the 2008 Text Analysis Conference Summarisation track (TAC08), respectively. The last scores are included to provide some context for the other results.4 From table 3 it is evident that the ROUGE scores obtained are low (at least in the context of TAC08). This suggests that sentiment intensity alone is not a suﬃciently representative feature of the importance of comments for summarisation purposes. Thus, using it in combination with other features that have proven useful for summarisation, such as entities mentioned in a given comment [16], certain cue phrases and surface features [18], or features capturing the relevance of blog posts to the main topic5 , is likely to yield better results. In particular, incorporating topic detection features would be crucial, since at the moment oﬀ-topic, but very negative or very positive, comments are clearly bad choices for a summary, and currently we employ no means for ﬁltering these out. There is also an alternative interpretation of the attained results. These results were obtained by using a methodology used in text summarisation research, so it is feasible that the method is not particularly well-suited for the task at hand, that of producing sentiment-rich summaries. Hence, the reason for the low results may be that we addressed the problem in the context of a slightly diﬀerent task, suggesting that the task of producing content-based summaries and that of producing sentiment-based summaries are two distinct tasks which require diﬀerent treatment. In addition to the above results, we perform the statistical hypothesis test explained in section §3.4. The necessary ingredients and the resulting t-statistic values are shown in table 2. In both cases, negative and positive polarity, the t values obtained are not large enough for us to reject the null hypothesis in favour of the alternative hypothesis. That is, we do not have empirical evidence to reject the null hypothesis that the sentiment intensity of salient blog comments is no diﬀerent from the sentiment intensity of non-salient comments in favour of our alternative hypothesis that, indeed, sentiment intensity in summary blog comments is diﬀerent from that of non-summary blog comments. 4 5

We note, however, that the results on our corpus are not directly comparable with those of TAC08, since the data sets are diﬀerent. Blog posts in our corpus were annotated as important with respect to the main topic of the respective blog threads.

210

M. Kabadjov, A. Balahur, and E. Boldrini

We conclude that, based on our annotated corpus, the hypothesis that very positive or very negative sentences are also good summary sentences does not hold. But, once again, we point out that these results are meaningful in the context of text summarisation, that is, the task of producing content-based summaries. Hence, the observation we made above that producing content-based summaries is diﬀerent from producing sentiment-based summaries and as such these tasks should be treated diﬀerently applies also in this case.

5

Conclusions

In this paper we addressed the question of whether very positive or very negative blog comments from the perspective of sentiment analysis are good summary sentences from the perspective of text summarisation. We used a sentiment analysis tool capable of producing a score in the range [−10, +10] for every sentence, to process a corpus of blog threads which has been annotated for salience and sentiment. We proposed a simple summarisation algorithm that sorts all sentences of the same polarity according to their sentiment score in decreasing order and chooses the top n (e.g., the top 15% or 30%) as a resulting summary. All such summaries were evaluated against the model summaries produced according to the annotation using four ROUGE metrics commonly employed in text summarisation evaluations: R1 , R2 , RSU4 and RL (see section §4.2 for details). In addition, we designed and carried out a hypothesis test to evaluate statistically the hypothesis that very positive or very negative blog comments are good summary sentences. In the light of the low summarisation results attained and the results from the statistical test, we concluded that the aforementioned hypothesis does not hold, at least not based on our corpus data, and argued that summarising sentiment and summarising text are two diﬀerent tasks which should be treated separately. In future work we intend to explore in more depth the problem of summarising opinions expressed in blogs adopting an approach to summarisation combining statistical information with information about persons and organisations such as the one proposed in [16], since in blogs it is very common that a thread is elaborated around an important person or organisation.

References 1. Balahur, A., Lloret, E., Ferr´ andez, O., Montoyo, A., Palomar, M., Mu˜ noz, R.: The DLSIUAES team’s participation in the TAC 2008 tracks. In: National Institute of Standards and Technology [22] 2. Balahur, A., Lloret, E., Boldrini, E., Montoyo, A., Palomar, M., Mart´ınez-Barco, P.: Summarizing threads in blogs using opinion polarity. In: Proceeding of the Workshop on Events in Emerging Text Types at RANLP, Borovetz, Bulgaria (September 2009) 3. Balahur, A., Steinberger, R., van der Goot, E., Pouliquen, B.: Opinion mining from newspaper quotations. In: Proceedings of the Workshop on Intelligent Analysis and Processing of Web News Content at the IEEE / WIC / ACM International Conferences on Web Intelligence and Intelligent Agent Technology (WI-IAT) (2009)

Sentiment Intensity: Is It a Good Summary Indicator?

211

4. Bossard, A., G´en´ereux, M., Poibeau, T.: Description of the LIPN systems at TAC 2008: Summarizing information and opinions. In: National Institute of Standards and Technology [22] 5. Cerini, S., Compagnoni, V., Demontis, A., Formentelli, M., Gandini, G.: MicroWNOp: A gold standard for the evaluation of automatically compiled lexical resources for opinion mining. In: Sans` o, A. (ed.) Language Resources and Linguistic Theory: Typology, Second Language Acquisition, English Linguistics, Franco Angeli, Milano, IT (2007) 6. Chaovalit, P., Zhou, L.: Movie review mining: a comparison between supervised and unsupervised classiﬁcation approaches. In: Proceeding of HICSS 2005, the 38th Hawaii International Conference on System Sciences (2005) 7. Conroy, J., Schlesinger, S.: Classy at TAC 2008 metrics. In: National Institute of Standards and Technology [22] 8. Cruz, F., Troyani, J., Ortega, J., Enr´ıquez, F.: The Italica system at TAC 2008 opinion summarization task. In: National Institute of Standards and Technology [22] 9. Dave, K., Lawrence, S., Pennock, D.: Mining the peanut gallery: Opinion extraction and semantic classiﬁcation of product reviews. In: Proceeding of the World Wide Web Conference (2003) 10. Riloﬀ, E., Wiebe, J., Phillips, W.: Exploiting subjectivity classiﬁcation to improve information extraction. In: Proceedings of the 20th National Conference on Artiﬁcial Intelligence (AAAI) (2005) 11. Erkan, G., Radev, D.R.: LexRank: Graph-based centrality as salience in text summarization. Journal of Artiﬁcial Intelligence Research, JAIR (2004) 12. Esuli, A., Sebastiani, F.: SentiWordNet: A publicly available resource for opinion mining. In: Proceeding of the 6th International Conference on Language Resources and Evaluation, Italy (May 2006) 13. He, T., Chen, J., Gui, Z., Li, F.: CCNU at TAC 2008: Proceeding on using semantic method for automated summarization yield. In: National Institute of Standards and Technology [22] 14. Hovy, E.H.: Automated text summarization. In: Mitkov, R. (ed.) The Oxford Handbook of Computational Linguistics, pp. 583–598. Oxford University Press, Oxford (2005) 15. Hu, M., Liu, B.: Mining opinion features in customer reviews. In: Proceeding of the National Conference on Artiﬁcial Intelligence (AAAI) (2004) 16. Kabadjov, M.A., Steinberger, J., Pouliquen, B., Steinberger, R., Poesio, M.: Multilingual statistical news summarisation: Preliminary experiments with english. In: Proceedings of the Workshop on Intelligent Analysis and Processing of Web News Content at the IEEE / WIC / ACM International Conferences on Web Intelligence and Intelligent Agent Technology (WI-IAT) (2009) 17. Kim, S., Hovy, E.: Determining the sentiment of opinions. In: Proceedings of the International Conference on Computational Linguistics (COLING) (2004) 18. Kupiec, J., Pedersen, J., Chen, F.: A trainable document summarizer. In: Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, Washington, pp. 68–73 (1995) 19. Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Proceedings of the Workshop on Text Summarization Branches Out, Barcelona, Spain (2004) 20. Pang, B., Lee, L.: Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2(1-2), 1–135 (2008)

212

M. Kabadjov, A. Balahur, and E. Boldrini

21. Riloﬀ, E., Wiebe, J.: Learning extraction patterns for subjective expressions. In: Proceeding of the Conference on Empirical Methods in Natural Language Processing (2003) 22. National Institute of Standards and Technology (eds.): Proceeding of the Text Analysis Conference. Gaithersburg, MD (November 2008) 23. Stoyanov, V., Cardie, C.: Toward opinion summarization: Linking the sources. In: Proceedings of the COLING-ACL Workshop on Sentiment and Subjectivity in Text.Association for Computational Linguistics, Sydney (July 2006) 24. Strapparava, C., Valitutti, A.: WordNet-Aﬀect: an aﬀective extension of wordnet. In: Proceeding of the 4th International Conference on Language Resources and Evaluation, Lisbon, Portugal, pp. 1083–1086 (May 2004) 25. Turney, P.: Thumbs up or thumbs down? semantic orientation applied to unsupervised classiﬁcation of reviews. In: Proceeding of the Annual Meeting of the Association for Computational Linguistics (ACL) (2002) 26. Varma, V., Pingali, P., Katragadda, R., Krisha, S., Ganesh, S., Sarvabhotla, K., Garapati, H., Gopisetty, H., Reddy, V., Bysani, P., Bharadwaj, R.: IIT Hyderabad at TAC 2008. In: National Institute of Standards and Technology [22] 27. Wilson, T., Wiebe, J., Hwa, R.: Just how mad are you? ﬁnding strong and weak opinion clauses. In: Proceeding of the National Conference on Artiﬁcial Intelligence (AAAI) (2004)

Syntactic Tree Kernels for Event-Time Temporal Relation Learning Seyed Abolghasem Mirroshandel, Mahdy Khayyamian, and Gholamreza Ghassem-Sani Department of Computer Engineering, Sharif University of Technology Azadi Ave., 11155-9517 Tehran, Iran {mirroshandel,khayyamian}@ce.sharif.edu, [email protected]

Abstract. Temporal relation classification is one of the contemporary demanding tasks in natural language processing. This task can be used in various applications such as question answering, summarization, and language specific information retrieval. In this paper, we propose an improved algorithm for classifying temporal relations between events and times, using support vector machines (SVM). Along with gold-standard corpus features, the proposed method aims at exploiting useful syntactic features, which are automatically generated, to improve accuracy of the classification. Accordingly, a number of novel kernel functions are introduced and evaluated for temporal relation classification. The result of experiments clearly shows that adding syntactic features results in a notable performance improvement over the state of the art method, which merely employs gold-standard features. Keywords: Temporal Relations between Event and Time, Information Retrieval, Text Mining, Classification.

1 Introduction In recent years, researches on natural language processing (NLP) have substantially made progress. Combining statistical and symbolic methods is one of the major reasons for these advances. Part-of-speech tagging, morphological analysis, parsing, and named entity recognition are some of the tasks which have been addressed with satisfactory results [1]. Problems that require deeper semantic analysis, such as temporal information processing, are yet to be addressed. Usually, textual description of some events is a great source of temporal information. Recently, the growing attention on practical NLP applications such as question answering (Q&A), summarization, and information extraction have resulted in an increasing demand for temporal information processing [2]. In Q&A, questions such as "when an event occurred", or "what is the chronological order between some desired events" must be answered. In text summarization, especially in multi-document type, knowing the order of events is a useful source for merging related information correctly. In this paper, we propose an improved technique of classifying temporal relations between events and times, using machine learning approach. Support vector classification using effective kernel functions are specifically applied to two types of Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 213–223, 2011. © Springer-Verlag Berlin Heidelberg 2011

214

S.A. Mirroshandel, M. Khayyamian, and G. Ghassem-Sani

features: corpus gold-standard event-time features and underlying syntactic features of the contextual sentence. To capture each of these two feature types, we apply an event-time kernel to the gold-standard features, and a convolution tree-kernel to syntactic features. The event-time kernel has been implemented according to [1], and convolution tree kernel [3] has been employed as our syntactic tree kernel. Experimental results on Timebank validate the proposed method by showing 5% improvement over the state of the art method that merely uses gold-standard features. The remainder of the paper is organized as follows: section 2 is about previous approaches to temporal relation classification. Section 3 explains our proposed method. Section 4 briefly presents characteristic of the corpus that we have used. Section 5 demonstrates evaluation of the proposed algorithm. Finally, section 6 includes our conclusions.

2 Previous Works There have not been a lot of researches on event-time relation classification, but on general temporal relation classification, there are numerous ongoing works which can be applied to event-time relation extraction as well. These efforts can be divided into three categories: 1) Pattern based; 2) Rule based, and 3) Anchor based. These categories are discussed in the next three subsections. 2.1 Pattern Based Methods Pattern based methods try to extract some generic lexico-syntactic patterns for eventevent or event-time co-occurrence. Extracting these patterns can be done manually or automatically. Perhaps the simplest pattern based method is the one that was developed using a knowledge resource called VerbOcean [4]. VerbOcean has a small number of manually selected generic patterns. After manually generating these patterns, the method can obtain some of existing semantic relations. Like other manual methods, a major weakness of this method is its tendency for having a high recall but a low precision. One way to overcome this flaw is to create more specific patterns; however, it is clear that this would be very hard and time consuming. Many researchers have tried to resolve the low precision problem by using an additional component for pruning extracted relations. This component can be some heuristics [4]. On the other hand, automatic methods use machine learning techniques to extract patterns. They try to learn a classifier from an annotated corpus, and attempt to improve classification accuracy by feature engineering. The most important algorithm of this group is the MaxEnt classifier [1]. This classifier uses perfect, hand-tagged, features including tense, aspect, modality, polarity, and event class. In addition to these features, it relies on two other features including pairwise agreement of tense and aspect. In this paper, we propose a new method to improve this particular method. There have been some other methods that have used machine learning techniques for acquisition of different relation types [5, 6, 7].

Syntactic Tree Kernels for Event-Time Temporal Relation Learning

215

2.2 Rule Based Methods The common idea behind rule based methods is to find some rules for classifying temporal relations. In most existing works, these rules are determined manually and are based on Allen's interval algebra [8]. In a study, rules of temporal transitivity were applied to increase the training set by a factor of 10. Next, the MaxEnt classifier was trained on this enlarged corpus. The test accuracy on this enlarged corpus was very encouraging. There was nearly 12% progress in accuracy [1]. Reasoning with determined rules is another point of view of the rules' usage. In [2], a rich set of rules (axioms) was created. Then by using a first order logic based theorem prover, they tried to find a proof of each temporal relation by refutation. 2.3 Anchor Based Methods Anchor based methods use information of argument fillers (i.e., anchors) of every event expression as a valuable clue for recognizing temporal relations. They rely on the distributional hypothesis [9], and by looking at a set of event expressions whose argument fillers have a similar distribution, they try to recognize synonymous event expressions. Algorithms such as DIRT [10] and TE/ASE [11] are some examples of anchor based methods. It has been shown that one can gain more accuracy by merging some of these three different methods. For example, pattern and rule based methods were merged [1], and the new system showed to be more efficient than each of the base methods. In other examples, pattern and anchor based methods were combined to form a successful classifier [4, 5].

3 Syntactic Tree Kernels in SVM Syntactic features are a great source of information in various NLP and text mining applications, as it has been shown in relation extraction, semantic role labeling, and coreference resolution. Existing works in temporal relation classification have not sufficiently utilized such features. Here, we aim at taking advantage of syntactic features. Due to promising results of Support Vector Machines (SVM) [12, 13], it has been chosen as our classification algorithm. To incorporate syntactic features into SVM, convolution tree kernels are applied in combination with a simple event-time kernel. 3.1 Simple Event-Time Kernel This is a linear kernel that exclusively uses gold-standard features of events and times. For each event, there are five temporal attributes which have been tagged in Timebank: 1) tense; 2) grammatical aspect; 3) modality; 4) polarity, and 5) event class. Tense and grammatical aspect define temporal location and event structure; thus, they are necessary in any method of temporal relation classification. Modality and polarity specify non-occurring (or hypothetical) situations. The event class shows the type of event. For each time expression, this kernel uses five attributes: 1) Type, 2) Value, 3) Freq, 4) Mod, and 5) Quant. These attributes are simple attributes which have been annotated in the corpus.

216

S.A. Mirroshandel, M. Khayyamian, and G. Ghassem-Sani

In addition to these attributes, it uses part of speech tags of event and time expression as an extra feature. This kernel can be defined as follows:

KTR (TR1 , TR2 ) = K E (TR1.E , TR2 . E ) + K T (TR1.T , TR2 .T )

(1)

where TR1 and TR2 stand for two temporal relation instances, E is the event, and T is the time expression in temporal relation instance. KE and KT are simple kernel functions over the features of event and time instances:

K E (E1 , E2 ) = ∑ C (E1 . f i , E2 . f i )

(2)

K T (T1 , T2 ) =

(3)

i

∑ C (T . f , T . f ) 1

i

2

i

i

where fi means the ith event or time feature; function C returns 1 if the two feature values are identical, and returns 0, otherwise. KE and KT return the number of feature values that are common in both events and times respectively. 3.2 Tree Kernels A generalized version of convolution tree kernel [14] was presented in [3]. In this paper, we apply some customized versions of this kernel to capture syntactic features in temporal relation classification. These kernels will be described in the next subsections. 3.2.1 A Generalized Convolution Tree Kernel To explain the generalized syntactic tree kernel [3], first a feature vector characterizing a parse tree is introduced in equation (4). The ith feature of this vector equals the sum of weighted number of instances of sub-tree type ith in the tree. Function Isubtree i (n) indicates whether subtreei occurs at node n or not by returning 1 and 0, respectively. subtreei(n) is the sub-tree instance of type ith rooted at node n. According to equation (5), function tw(T) (which stands for "tree weight") associates a weight to tree T. This weight is the product of all the weights of its nodes. in(T) and en(T) are respectively the sets of internal and external nodes of tree T.

H (T ) =

( ∑ [I subtree (n ) × tw(subtree1 (n ))], . . . , 1

n ∈T

∑ [I

subtree i

∑ [I

subtree m

n ∈T

n ∈T

tw(T ) =

(n ) × tw(subtree i (n ))], . . . ,

(4)

(n ) × tw(subtree m (n ))], . . . , )

∏

inw(n ) ×

n ∈ in (T )

∏

enw(n )

n ∈ en (T )

(5)

Syntactic Tree Kernels for Event-Time Temporal Relation Learning

217

K (T1 , T2 ) = H (T1 ), H (T2 ) ⎛ ⎞ = ⎜⎜ ∑ ∑ I subtreei (n1 ) × tw(subtree i (n1 ))⎟⎟ × ⎝ i n1 ∈ T1 ⎠ ⎛ ⎞ ⎜ ∑ ∑ I subtree (n 2 ) × tw(subtree i (n 2 ))⎟ i ⎜ i n ∈T ⎟ ⎝ ⎠ 2 2 = ∑ ∑ C (n1 , n 2 )

(6)

n1 ∈ T1 n 2 ∈ T2

Since each node of the parse tree can either be an internal or an external node of a supposed sub-tree (assuming that it exists in the sub-tree), two types of weights are respectively assigned to its nodes by functions inw(n) and enw(n) (which respectively stand for the internal and the external node weights). For instance, in Fig. 1, the node labeled PP is an external node of sub-trees (1) and (7), while it is considered to be an internal node of sub-trees (3) and (4). As clarified by equation (6), a technique analogous to that of [14] is utilized to create a kernel for the calculation of dot products of H(T) vectors. The kernel calculation finally leads to the summation of C(n1, n2) values over all tree node pairs T1 and T2. Function C(n1, n2) is the weighted sum of common sub-trees of n1 and n2 that can be recursively computed [14]:

Fig. 1. Samples of sub-trees used in convolution tree kernel calculation for "Mark at the airport"

218

S.A. Mirroshandel, M. Khayyamian, and G. Ghassem-Sani

1) If n1 and n2 represent different production rules, then

C (n1 , n2 ) = 0

(7)

2) Else if n1 and n2 are identical pre-terminals (identical part of speeches), then

C (n1 , n2 ) = inw(n1 )× enw(child (n1 )) × inw(n2 )× enw(child (n2 ))

(8)

3) Else if both n1 and n2 represent identical production rules, then

C (n1 , n 2 ) = inw (n1 )× inw (n 2 ) ×

⎡enw(child i (n1 )) × enw(child i (n 2 ))⎤ ⎥ i i 1 2 ⎦

∏ ⎢+ C (child (n ), child (n )) ⎣ i

(9)

3.2.2 Customizing Tree Kernel Here, we can produce four kernels from the proposed generalized kernel by changing weighting function. These kernels can be easily utilized for temporal relation classification. Since the entire parse tree of the sentence that holds the event-time arguments may contain numerous misleading features, Path-enclosed Tree (PT) is selected as our tree portion for applying tree kernels as in [15]. PT is a portion of the parse encircled by the shortest path between the two arguments. The Original Collins and Duffy Kernel By defining inw (n ) = α = λ and enw(n)=1 for every node n of the tree, the generalized kernel can be reduced to the original kernel proposed in [14]. In their paper, parameter 0 < λ ≤ 1 is a decaying parameter used to prevent the kernel values from diverging, without which the values of the kernel over identical trees would dominate those of different trees, slowing down SVM convergence. By different definitions of the weighting functions, several customized kernels can be made out of the generalized kernel. Weighting functions enable the kernel to differentiate among sub-trees depending on their interaction with event-time arguments. Argument Ancestor Path Distance Kernel (AAPD) Weighting functions of this kernel are given as follows. Parameter 0 < α ≤ 1 is a decaying parameter similar to λ .

inw(n ) = enw(n ) = α

min ( AAPDist ( n , arg 1 ), AAPDist ( n , arg 2 )) MAXDIST

(10)

Function AAPDist(n, arg) returns the distance of node n from ancestor path of argument (time or event) arg on the parse tree as shown in Fig. 2. MAXDIST, which is the

Syntactic Tree Kernels for Event-Time Temporal Relation Learning

219

Fig. 2. A syntactic parse tree with AAPDist and ArgDist example. There is a temporal relation between Tuesday (as time) and gained (as event) in this sentence.

maximum value of AAPDist in the whole tree, is used here for normalization. According to this weighting approach, the closer a node is to one of the arguments of the ancestor path, the less it is decayed by the weighting function. Argument Distance Kernel (AD) Weighting functions of this kernel, which have identical definitions, are defined as follows. Their definitions are similar to the previous kernel functions, though they use a different distance function that measures the distance of a node from an argument rather than its ancestor path (see Fig. 2).

inw (n ) = enw (n ) = α

min ( ArgDist ( n , arg 1 ), ArgDist ( n , arg 2 )) MAXDIST

(11)

Threshold Sensitive Argument Ancestor Path Distance Kernel (TSAAPD) This kernel is intuitively similar to AAPD kernel, though instead of decaying smoothly, it uses a rough threshold based technique. Weighting functions are defined as follows:

⎧1 inw (n ) = enw (n ) = ⎨ ⎩α

AAPDist (n ) ≤ Threshold

AAPDist (n ) > Threshold

(12)

3.3 Composite Kernels

In this section, two types of composition are explained: linear composition and polynomial composition [15]. 3.3.1 Linear Composite Kernel

K l (TR1 , TR 2 ) = α Kˆ 1 (TR1 , TR 2 ) + (1 − α ) Kˆ 2 (TR1 , TR 2 )

(13)

220

S.A. Mirroshandel, M. Khayyamian, and G. Ghassem-Sani

where Kˆ 1 is the normalized form of one of the customized convolution tree kernels. A kernel K(X, Y) can be normalized by dividing it by K ( X , X ). K (Y , Y ) . Kˆ 2 is a simple normalized event-time kernel. α is the composition coefficient. Four composite kernels are constructed in this way using the four previously introduced tree kernels. 3.3.2 Polynomial Composite Kernel P K P (TR1 , TR 2 ) = α Kˆ 1 (TR1 , TR 2 ) + (1 − α ) Kˆ 2 (TR1 , TR 2 )

(14)

where Kˆ 1 , Kˆ 2 and α have the same definition as in linear composite kernel. Kˆ 2 P is the polynomial expansion of Kˆ 2 with degree d (in this work, d is set to 2) defined in the following way:

(

P Kˆ 2 = 1 + Kˆ 2

)

d

(15)

In this case, four different polynomial composite kernels are created, too.

4 Corpus Description We used Timebank (v 1.2) and Opinion Corpus [1] in our experiments. Timebank has 183 newswire documents and 64077 words, and Opinion Corpus has 73 documents with 38709 words. These two datasets have been annotated based on TimeML [16]. There are 14 temporal relations in TLink (Event-Event and Event-Time relations) class of TimeML. Similar to [1, 2, 17], we used a normalized version of these 14 temporal relations that contains 6 temporal relations RelTypes = {SIMULTANEOUS, IBEFORE, BEFORE, BEGINS, ENDS, INCLUDES}. For converting 14 relations to 6, the inverse relations were omitted, and IDENTITY and SIMULTAENOUS, as well as IS_INCLUDED and DURING, were collapsed. Table 1. The normalized TLink distribution in the Timebank and OTC Relation IBEFORE BEGINS ENDS SIMULTANEOUS INCLUDES BEFORE Total

Timebank 7 52 124 70 1634 (58.4) 910 2797

OTC 15 112 159 77 3001 (65.3%) 1229 4593

In our experiments, like some previous works, we merged two datasets to generate a single corpus called OTC. Table 1 shows the normalized TLink class distribution over Timebank and OTC.

Syntactic Tree Kernels for Event-Time Temporal Relation Learning

221

As it is shown in table 1, relation "INCLUDES" is the most frequent relation; thus it forms the majority class, and has been used as the baseline of experiments.

5 Experiments We have used LIBSVM [18] java source for the SVM classification (one-versus-one multi class strategy), and Stanford NLP package [19] for tokenization, sentence segmentation, and parsing. Since tree kernels can be more appropriately applied to the event-time pairs that exist in the same sentence, the corpus data have been accordingly split into two intrasentential and inter-sentential parts. The proposed kernels have been evaluated on the intra-sentential instances, while the simple event-time kernel has been exclusively used for the inter-sentential instances. The results reported for the whole corpus have been produced by combining those two results. All the results are the outcome of a 5fold cross validation. In order to find the appropriate value for α , λ , and C, 1000 event pairs have been randomly chosen as the development set. Experiments on this development set have shown that the appropriate value for λ , linear kernel's α , polynomial kernel's α , and C are 0.4, 0.38, 0.26, and 3, respectively. Table 2 shows the accuracy results of employing different tree kernels. In this evaluation, baseline is the majority class (INCLUDES relation) of the evaluated corpus. The Mani's method is currently regarded as the state of the art method, which exclusively uses gold-standard features [1]. The other methods were described in subsection 3.2.2. Table 2. The accuracy of tree kernels Method Baseline Mani CollinsDuffy AAPD AD TSAAPD

Timebank Corpus 58.4 75.42 76.72 78.36 77.91 78.05

OTC Corpus 65.3 76.13 76.81 77.59 77.38 77.95

As it is shown in table 2, using syntactic structure of sentences is an effective clue for temporal relation classification. On Timebank, AAPD kernel has achieved the best results, it has showed 3% improvement compared to Mani algorithm. On the other hand, TSAAPD is the most effective kernel over OTC corpus. Comparing with Mani method, it has gained near 2% progress on accuracy. The other tree kernels showed satisfactory results, too. The results of applying linear composite kernels (subsection 3.3.1) to Timebank and OTC are shown in table 3. As it is demonstrated in this table, the effective exploitation of syntactic and simple event features in the linear composite kernels resulted in a notable accuracy improvement. Here, AD linear composite kernel was

222

S.A. Mirroshandel, M. Khayyamian, and G. Ghassem-Sani Table 3. The accuracy of linear composite kernels

Method CollinsDuffy (L) AAPD (L) AD (L) TSAAPD (L)

Timebank Corpus 78.84 79.53 79.65 79.22

OTC Corpus 77.87 78.46 78.21 78.15

the most successful kernel on Timebank, which achieved over 4% improvement, and in the case of OTC, AAPD composite kernel was the best kernel with 2.3% progress. Table 4 shows the accuracy results of applying four polynomial composite kernels (subsection 3.3.2) to Timebank and OTC. Table 4. The accuracy of polynomial composite kernels Method CollinsDuffy (P) AAPD (P) AD (P) TSAAPD (P)

Timebank Corpus 79.24 80.33 80.19 79.85

OTC Corpus 77.93 79.06 78.72 78.54

The results of applying these kernels demonstrate that these methods work more successful than their linear counterparts. Here, AAPD linear composite kernel was the most successful kernel, which gained over 5% improvement when applied to Timebank, and 3% progress in accuracy in the case of OTC. Unfortunately there are not a lot of researches on event-time relation classification, and we have to compare our work only with Mani algorithm. Regarding the hardness of the problem, it can be said, that the improvement is considerable.

6 Conclusion In this paper, we have addressed the classification problem of temporal relations between events and times, which is a hot topic in natural language processing. Although syntactic features have shown acceptable results in various text classification tasks, they have not yet been efficiently exploited in temporal relation classification. We have tried to use these features as a strong clue to enhance classification performance. Support Vector Machines (SVM) have been chosen as our classification algorithm, because of its excellent results in related works. Using SVM, two types of composite kernels have been proposed by combining convolution tree kernels and a simple event-time kernel. The results of experiments, without using any extra annotated data, show a significant progress over related works in the area of pattern based methods (including the state of the art method) in terms of accuracy. It seems that using other source of information like WordNet or creating more effective kernels for SVM might even further improve the accuracy of system.

Syntactic Tree Kernels for Event-Time Temporal Relation Learning

223

References 1. Mani, I., Marc, V., Wellner, B., Lee, C.M., Pustejovsky, J.: Machine Learning of Temporal Relations. In: ACL, vol. 44, pp. 753–760 (2006) 2. Tatu, M., Srikanth, M.: Experiments with Reasoning for Temporal Relations between Events. In: Coling 2008, pp. 857–864 (2008) 3. Khayyamian, M., Mirroshandel, S.A., Abolhassani, H.: Syntactic Tree-based Relation Extraction Using a Generalization of Collins and Duffy Convolution Tree Kernel. In: HLT/NAACL 2009, pp. 66–71 (2009) 4. Chklovski, T., Pantel, P.: Global path-based refinement of noisy graphs applied to verb semantics. In: Dale, R., Wong, K.-F., Su, J., Kwong, O.Y. (eds.) IJCNLP 2005. LNCS (LNAI), vol. 3651, pp. 792–803. Springer, Heidelberg (2005) 5. Abe, S., Inui, K., Matsumoto, Y.: Two-Phased Event Relation Acquisition Coupling the Relation-Oriented and Argument-Oriented Approaches. In: Coling 2008, pp. 1–8 (2008) 6. Chambers, N., Wang, S., Jurafsky, D.: Classifying Temporal Relations between Events. In: ACL, vol. 45, pp. 173–176 (2007) 7. Lapata, M., Lascarides, A.: Learning Sentence-Internal Temporal Relations. Journal of Artificial Intelligence Research 27, 85–117 (2006) 8. Allen, J.F.: Towards a General Theory of Action and Time. Artificial Intelligence 23, 123–154 (1984) 9. Harris, Z.: Mathematical Structure of Language. John Wiley Sons, New York (1968) 10. Lin, D., Pantel, P.: Dirt - Discovery of Inference Rules From Text. In: The 7th ACM SIGKDD, pp. 323–328 (2001) 11. Szpektor, I., Tanev, H., Dagan, I.: Scaling Web-based Acquisition of Entailment Relations. In: EMNLP 2004, pp. 41–48 (2004) 12. Boser, B.E., Guyon, I.M., Vapnik, V.N.: A Training Algorithm for Optimal Margin Classifiers. In: COLT 1992, pp. 144–152. ACM, New York (1992) 13. Cortes, C., Vapnik, V.: Support-Vector Networks. Machine Learning, 273–297 (1995) 14. Collins, M., Duffy, N.: Convolution Kernels for Natural Language. In: Advances in Neural Information Processing Systems, vol. 14, pp. 625–632. MIT Press, Cambridge (2001) 15. Zhang, M., Zhang, J., Su, J., Zhou, G.D.: A Composite Kernel to Extract Relations between Entities with Both Flat and Structured Features. In: ACL, vol. 44, pp. 825–832 (2006) 16. Pustejovsky, J., Hanks, P., Sauri, R., See, A., Gaizauskas, R., Setzer, A., Radev, D., Sundheim, B., Day, D., Ferro, L., Lazo, M.: The TIMEBANK Corpus. In: Corpus Linguistics 2003, pp. 647–656 (2003) 17. Mani, I., Wellner, B., Verhagen, M., Pustejovsky, J.: Three Approaches to Learning Tlinks in TimeML. In Technical Report CS-07-268. Brandeis University, Waltham, USA (2007) 18. Chang, C. C., Lin, C.J.: Libsvm: a Library For Support Vector Machines (2001), software available at http://www.csie.ntu.edu.tw/cjlin/libsvm 19. Stanford NLP Package, http://nlp.stanford.edu/software/index.shtml

The WSD Development Environment Rafał Młodzki1 and Adam Przepiórkowski1,2 1

2

Institute of Computer Science PAS, ul. Ordona 21, 01-237 Warszawa, Poland [email protected], [email protected] University of Warsaw, Krakowskie Przedmieście 26/28, 00-927 Warszawa, Poland

Abstract. In this paper we present the Word Sense Disambiguation Development Environment (WSDDE), a platform for testing various Word Sense Disambiguation (WSD) technologies, as well as the results of ﬁrst experiments in applying the platform to WSD in Polish. The current development version of the environment facilitates the construction and evaluation of WSD methods in the supervised Machine Learning (ML) paradigm using various knowledge sources. Experiments were conducted on a small manually sense-tagged corpus of 13 Polish words. The usual groups of features were implemented including bag-of-words, parts-ofspeech, words with their positions, etc. (with diﬀerent settings), in connection with popular ML algorithms (including Naive Bayes, Decision Trees and Support Vector Machines). The aim was to test to what extent standard approaches to the English WSD task may be adopted to free word order and rich inﬂection languages such as Polish. In accordance with earlier results in the literature, the initial experiments suggest that these standard approaches are relatively well-suited for Polish. On the other hand, contrary to earlier ﬁndings, the experiments also show that adding of some features beyond bag-of-words increases the average accuracy of the results. Keywords: word sense disambiguation, machine learning, feature selection, Polish.

1

Introduction

The Word Sense Disambiguation (WSD) task consists of choosing the most appropriate sense of a word from all its senses in a given context. For example in the context: (K1) Chodzi mi o to, aby język giętki Powiedział wszystko, co pomyśli głowa. Juliusz Słowacki1 1

Roughly: Let my nimble tongue (język) represent the Mind. Juliusz Słowacki

Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 224–233, 2011. c Springer-Verlag Berlin Heidelberg 2011

The WSD Development Environment

225

the word “język” is used in the meaning “tongue”, and in the context: (K2) W języku słowackim da się powiedzieć mniej więcej tyle, co w języku polskim.2 the most appropriate sense for język is “language”. Accurate WSD could be of great importance for numerous tasks in Natural Language Processing (NLP), such as text categorization, information retrieval or machine translation. The adjective “accurate” is essential ([2]). So far the best accuracy has been obtained (English) within the supervised Machine Learning (ML) paradigm. Despite the above and the fact that WSD in English dates back to the 1950s, hardly any work on this subject has been reported for Polish (one exception is the preliminary analysis in [3]). Moreover, there is currently no publicly available sense-tagged corpus of Polish of a decent size. Both these issues are addressed in the National Corpus of Polish project (Pol. Narodowy Korpus Języka Polskiego; NKJP; http://nkjp.pl/; [10,11]), where the necessary resources and tools are created. The remainder of this paper presents the ﬁrst step in this direction: the WSD Development Environment (WSDDE) and initial experiments in applying this platform to WSD in Polish, for the time being using a small sense-tagged corpus from [3].

2

WSD Development Environment

The environment is a bundle of a number of resources, applications, procedures and scripts which facilitate the development and evaluation of WSD methods. The overall scheme of the environment is depicted in Figure 1. There are several possible scenarios of how to use the environment. In the simplest scenario the user creates a single WSD method by specifying various settings to be described in detail below (e.g., the source and number of training examples, the size of the bag-of-words window, the particular Machine Learning algorithm, feature ﬁltering methods, etc.) and runs the application. On the basis of these settings, an appropriate WSD method will be prepared, trained and evaluated. The results can be saved in the database. The settings are divided into parameters of feature generators (e.g., the size of the bag-ofwords window) and those that are not directly connected with the generation of features (e.g., an ML algorithm). Another scenario is to create many WSD methods at once. One way to achieve this is to underspecify some settings in the conﬁguration ﬁle. For example, the user can set {1, 5, 10, 20, 40} as the size of the bag-of-words window. The application will generate all the possible methods combined from all the settings (i.e., one method where this size is 1, another method where the size is 5, etc.). If the number of possible methods is too large, then a random sample is created. Another way to do this is to choose some WSD methods (e.g., the best 5% among 2

In the Slovak language (język) one can express as much as, more or less, in Polish.

226

R. Młodzki and A. Przepiórkowski

Fig. 1. The scheme of the WSDDE

the ones evaluated so far and stored in the database) and generate other WSD methods that diﬀer from those in one or two parameters. All the results saved in the database can be analysed from diﬀerent points of view with prepared SQL queries (part of WSDDE). The results reported in § 3 have been obtained with these queries. 2.1

Corpora

This group of settings informs the corespondent WSD manager which corpus is to be used to train and test the WSD method. The WSDDE contains a tool for importing sense-tagged corpora in text format. Since there are not many such corpora, it also contains a tool for creating sense-tagged-like corpora from the IPI PAN Corpus (http://korpus.pl/; [7]) using the technique of pseudowords ([4,14]).3 The user can also specify the number of examples within the training and test sets. There is a possibility to choose the cross-validation method instead of evaluating WSD methods on a test set. 3

The idea of this technique is as follows. Two monosemous words A and B are fused into one artiﬁcial bisemous (psuedo)word AB with senses A and B, and all occurrences of words A and B in the corpus are replaced with the pseudoword AB. By knowing which original word occurred in which AB position, we know the senses of all occurrences of AB.

The WSD Development Environment

2.2

227

Feature Generation

The feature generator uses user settings and contexts from the training set to generate a group of features speciﬁc to it and then is used to compute feature values for all the contexts. Due to the open architecture of WSDDE, it is easy to implement one’s own feature generators or override the existing ones (by changing the number of parameters or their semantics). In the current version of the system, the user can choose from the following built-in feature generators: thematic feature generator, structural feature generator I/II and keyword generator. Thematic Feature Generator (TFG). This generator is the source of features which could characterise the domain or general topic of a given context. This is achieved by checking whether certain words are present in the wide context (e.g., up to 50 positions to the left or the right from the disambiguated word). It seems that the occurrence of a word in such a distant position cannot indicate anything but the general topic of a context, hence the name of the generator. The behaviour of the generator is currently controlled by three basic parameters: 1. The size of the bag-of-words window: This parameter determines the size of the text window the words are taken from. Its value is an integer in the range of 0–100, where values within the range 20–50 are most common. 2. Lemmatisation indicates whether it is the orthographic word form (0) or the base form (lemma; 1) that is considered. The lemmatisation is provided by the TaKIPI tagger ([6])4 . 3. Binary indicates whether feature values are binary (presence or absence; 0) or continuous (frequencies; 1). Depending on the settings of these parameters, the generated features indicate the presence or the frequency of particular word forms or lemmata in the text window of a given size. For example, if the training set consists only of two contexts K1 and K2 given in § 1, and the only generator is the thematic feature generator with parameters set respectively to (10,1,1) (i.e., window size 10, lemma frequencies as features), then a fragment of the feature vector for język could be depicted as in Table 1. Table 1. Some features and feature vectors for the topical feature generator and training set {K1, K2}

K1 K2

4

See also [5,1].

giętki głowa słowacki powiedzieć polski 1 1 1 1 0 0 0 1 1 1

228

R. Młodzki and A. Przepiórkowski

This generator, like any other in WSDDE, can be easily modiﬁed; for example, lemmatisation could be replaced by stemming or asymmetrical text windows could considered for bag-of-words. Structural Feature Generator I (SFG1). This generator produces features which can describe some structural properties of the disambiguated word, namely, the presence of particular words on a particular position in the close proximity to the disambiguated word. A small context (window up to the size of 5) is considered. The behaviour of the generator is controlled by three parameters: 1. The size of the window: this parameter determines the size of the text window considered and, hence, the size of the feature vector. Its value is an integer in the range of 0–5, where values 0–3 are most common. 2. Lemmatisation indicates whether orthographic or base forms are considered. 3. Binary indicates whether feature values are binary or continuous. For example, if the training set consists only of two contexts K1 and K2, and the only generator is the structural feature generator I with parameters set to (2,1,1), then the part of the feature vector could be depicted as in Table 2. Table 2. Some features and feature vectors for the structural feature generator I and training set {K1, K2}

K1 K2

w -1 słowackim+1 da+2 ,-2 aby-1 giętki+1 0 0 0 1 1 1 1 1 1 0 0 0

Structural Feature Generator II (SFG2). This generator also generates features describing structural properties, but it works at the part of speech (POS) level: it checks which parts of speech occur in which position in the close proximity of the disambiguated word. The behaviour of the generator is controlled by three parameters: 1. The size of the window: This parameter determines the size of the window the parts of speech are taken from. Its value is an integer in the range of 0–5 where values around 0–3 are most common. 2. Tagset indicates which tagset is used. One can choose between the original ﬁne-grained IPI PAN Tagset ([12,13])5 and a coarse-grained tagset consisting of 5 main parts of speech. 3. Binary indicates whether feature values are binary or continuous. 5

See also [9].

The WSD Development Environment

229

For example, if the training set consists only of two contexts K1 and K2 and the only generator is the structural feature generator II with parameters set to (2,basic,1), then the feature vector could be depicted as in Table 3. Table 3. Some features and feature vectors for structural feature generator II and training set {K1, K2}

K1 K2

prep-1 adj+1 verb+2 interp-2 0 1 1 1 1 1 1 0

Keyword Feature Generator (KFG). The keyword feature generator creates features related to the disambiguated word itself. In the current version, it generates the following features: the particular orthographic form of the word, its part of speech, and whether it starts with a capital letter. Table 4. Features and feature vectors for the keyword feature generator and training set {K1, K2}

K1 K2

język języku loc nom sg capitalized 1 0 0 1 1 0 0 1 1 0 1 0

Other Feature Generators. Apart from the above basic feature generators, some more experimental generators are available, including thematic feature generator II based on the WordNet hierarchy and structural feature generator III based on a naive attempt to recognise some grammatical relations (e.g., the closest noun is treated as the subject, etc.). 2.3

Feature Selection

These settings determine how to select features. One can use all the feature selection algorithms from the WEKA package (or one’s own, if they are compatible with WEKA interfaces). Generally, feature selection based on feature ranking works much faster than feature selection based on subsets. It is possible to use the second ﬁlter after the ﬁrst one, e.g., at ﬁrst one makes feature selection with feature ranking, e.g., 200 features remain, and then one makes feature selection on these remaining 200 features based on feature subsets. 2.4

Machine Learning Algorithms

This setting indicates which ML algorithms should be used. All classiﬁers from the WEKA package ([15]) are available. By default they are run with default parameters. It is also possible to add additional classiﬁers.

230

R. Młodzki and A. Przepiórkowski Table 5. The results of the experiment bag-of-words extra features accuracy size 20 74 20 SFG2(2,basic,1), 76 SFG1(2,1,1), KFG() 30 77 30 SFG2(2,basic,1), 80 SFG1(2,1,1), KFG() 40 78 40 SFG2(2,basic,1), 82 SFG1(2,1,1), KFG() 50 78 50 SFG2(2,basic,1), 81 SFG1(2,1,1), KFG()

2.5

Runtime

This setting informs the controller about the maximum time and memory size for learning and evaluation of the given method. 2.6

Reports

These settings concern the whole experiment (not a single WSD method) and indicate what kind of reports — based on prepared SQL queries — should be produced.

3

Experiments

In order to compare the results with the literature, we based the experiments on the toy corpus used in [3] — the only previous article about Polish WSD known to us. As a starting point we used the settings of the best WSD method from the cited article: NaiveBayes as the ML algorithm, the size of a bag-of-word window equal to 20, lemmatisation turned on both for bag-of-word and word-with-itsposition (WWIP) features. We also used — as in [3] — leave-one-out crossvalidation as the evaluation method. On the other hand, feature selection was always based only on the current training set (without the left-one-out example). This more standard evaluation procedure resulted in a smaller accuracy score: on the average about 10% (sic!). Also the algorithm of feature selection was diﬀerent — ﬁrst we took 200 features6 with the highest information gain (InfoGain in 6

This number is a compromise between quality and eﬃciency, established during previous experiments.

The WSD Development Environment

231

WEKA), and then we used subset feature selection (CfsSubsetEval in WEKA) on this set in order to ﬁlter out features which were mutually correlated. Feature selection was carried out for all the features (not only for bag-of-words features, as in the cited article). The WSD method described above (NaiveBayes, bag-of-words 20, no POSfeatures, no WWIP-features) achieved a 74% accuracy on that corpus (unweighted average over all WSD tasks). Next we carried out an experiment which aim was to evaluate WSD methods with settings diﬀering slightly from what was mentioned above (the baseline). They diﬀer in the use of structural feature generator I and II (POSs and WWIPs), the keyword feature generator and in the parameters of the thematic feature generator (the size of the window). We obtained the following results (see Table 5): by adding extra features (generated by SFG1(2,1,1), SFG2(2,basic,1) and KFG()) accuracy was improved by about 2% (up to 76%). By resizing the bag-of-words window up to 30 (using only features from the thematic feature generator) it was possible to obtain a 77% accuracy (+3% compared to the baseline). Adding extra features (SFG1(2,1,1), SFG2(2,basic,1), KFG()) to that resized window resulted in more than 80% accuracy, which is about 6% more than the baseline. A further resizing of the bag-of-words window yielded 78% for bag-of-words features and 82% for bag-ofwords features with additional features. A bigger size of bag-of-words does not lead to signiﬁcant improvements. ML algorithms other than NaiveBayes had a much lower accuracy (DecisionTrees) or worked much much slower (SVM). The share of features generated by particular generators in the feature vector is shown in Table 6. Table 6. Constitution of feature vector in relation to feature generators (average over all cases when all build-in feature generators were used) Feature Generator percent Thematic 81 Structural I 7 Structural II 6 Keyword 6

4

Conclusions

The ﬁrst obvious conclusion is that the bigger a bag-of-word window is used, the more considerable level of accuracy is achieved. As far as the extra features and their impact on the results are concerned, it is possible to explain the diﬀerence (a few percent) in comparison to the cited article by diﬀerences in feature selection (described above) and the use of the keyword feature generator. In the present article we use feature selection for all the features, whereas in the cited article it was only used for the bag-of-words features and all the other features were simply added without ﬁltration, which could have been a source of noise. The use of the keyword feature generator

232

R. Młodzki and A. Przepiórkowski

resulted in the improvement in the WSD task for powód (‘reason’ or ‘plaintiﬀ’) from 88% to 96%: the genitive of powód ‘reason’ is powodu, whereas the genitive of powód as ‘plaintiﬀ’ is powoda.7 Also plural forms of plaintiﬀ (e.g., powodzi) are rather uncommon. Features generated by the keyword feature generator are rather useless in WSD of English, but for Polish with its rich inﬂection it may have some importance. Table 7. The size of training sets, accuracy and time of computations for corpus of 2-pseudowords generated in WSDDE number of examples accuracy time 25 54,3 148,21 50 61,29 142,74 100 64,41 208,14 200 76,76 303,85 300 80,39 441,95 400 81,33 682 500 82,45 996,81 600 83,23 1171,64 700 83,83 1600,9 800 84,36 1885,36 900 84,42 2614,07 1000 84,92 2919,66 1100 84,53 3066,28 1200 84,99 4124,69 1300 85,37 4429,55 1400 85,07 5549,63 1500 84,54 6230,39 1600 84,91 7008,87 1700 85,77 7927,07 1800 85,56 9229,4 1900 85,92 9986,21 2000 85,8 10293,14

The above conﬁrms that the standard approach which was used for WSD in English also works quite well for Polish. Accuracy at the level of about 80% is relatively high, given such a basic approach and considering the small number of examples in training sets (less than 100 per sense); previous experiments on pseudowords (Table 7) have shown that accuracy grows signiﬁcantly with the growth of the number of examples up to 500 examples per sense. Of course, results obtained on the basis of such a small corpus are not very reliable. We deliberately do not present more detailed results, nor do we use more advanced features, in order to avoid presenting results which might be dubious. We wait until a large high-quality sense-tagged corpus for Polish is available within the framework of NKJP. 7

Note that genetivus seems to be the most frequent case in Polish ([8]).

The WSD Development Environment

233

References 1. Acedański, S., Przepiókowski, A.: Towards the adequate evaluation of morphosyntactic taggers. In: Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010); Poster Session, Beijing, pp. 1–8 (2010) 2. Agirre, E., Edmonds, P. (eds.): Word Sense Disambiguation: Algorithms and Applications, Text, Speech and Language Technology, vol. 33. Springer, Dordrecht (2006) 3. Baś, D., Broda, B., Piasecki, M.: Towards Word Sense Disambiguation of Polish. In: Proceedings of the International Multiconference on Computer Science and Information Technology (IMCSIT 2008): Computational Linguistics – Applications (CLA 2008), pp. 73–78. PTI, Wisła (2008) 4. Gale, W.A., Church, K.W., Yarowsky, D.: Work on statistical methods for word sense disambiguation. In: AAAI Fall Symposium on Probabilistic Approaches to Natural Language, Cambridge, pp. 54–60 (1992) 5. Karwańska, D., Przepiórkowski, A.: On the evaluation of two Polish taggers. In: Goźdź-Roszkowski, S. (ed.) The Proceedings of Practical Applications in Language and Computers PALC 2009, Peter Lang, Frankfurt am Main (2009) 6. Piasecki, M.: Polish tagger TaKIPI: Rule based construction and optimisation. Task Quarterly 11(1–2), 151–167 (2007) 7. Przepiórkowski, A.: The IPI PAN Corpus: Preliminary version. Institute of Computer Science, Polish Academy of Sciences, Warsaw (2004) 8. Przepiórkowski, A.: The IPI PAN Corpus in numbers. In: Vetulani, Z. (ed.) Proceedings of the 2nd Language & Technology Conference, Poznań, Poland, pp. 27–31 (2005) 9. Przepiórkowski, A.: A comparison of two morphosyntactic tagsets of Polish. In: Koseska-Toszewa, V., Dimitrova, L., Roszko, R. (eds.) Representing Semantics in Digital Lexicography: Proceedings of MONDILEX Fourth Open Workshop, Warsaw, pp. 138–144 (2009) 10. Przepiórkowski, A., Górski, R.L., Lewandowska-Tomaszczyk, B., Łazinski, M.: Towards the National Corpus of Polish. In: Proceedings of the Sixth International Conference on Language Resources and Evaluation, LREC 2008. ELRA, Marrakech (2008) 11. Przepiókowski, A., Górski, R.L., Łazinski, M., Pęzik, P.: Recent developments in the National Corpus of Polish. In: Levická, J., Garabík, R. (eds.) Proceedings of the Fifth International Conference on NLP, Corpus Linguistics, Corpus Based Grammar Research, Smolenice, Slovakia, November 25-27, pp. 302–309. Tribun, Brno (2009) 12. Przepiórkowski, A., Woliński, M.: A ﬂexemic tagset for Polish. In: Proceedings of Morphological Processing of Slavic Languages, EACL 2003, Budapest, pp. 33–40 (2003) 13. Przepiórkowski, A., Woliński, M.: The unbearable lightness of tagging: A case study in morphosyntactic tagging of Polish. In: Proceedings of the 4th International Workshop on Linguistically Interpreted Corpora (LINC 2003), EACL 2003, pp. 109–116 (2003) 14. Schütze, H.: Context space. In: AAAI Fall Symposium on Probabilistic Approaches to Natural Language, Cambridge, pp. 113–120 (1992) 15. Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005), http://www.cs.waikato.ac.nz/ml/weka/

Semantic Analyzer in the Thetos-3 System Julia Romaniuk1 , Nina Suszcza´ nska2 , and Przemyslaw Szmal2 1

Institute of Linguistics NAS of Ukraine, Grushevskogo 4, Kyiv, 01001, Ukraine [email protected] 2 Silesian University of Technology, Institute of Informatics ul. Akademicka 16, 44-100 Gliwice, Poland {Nina.Suszczanska,Przemyslaw.Szmal}@polsl.pl

Abstract. The paper describes elements of the model of Polish language semantics, which have been developed for the system Thetos; Thetos translates Polish texts into a dialect of the Polish Sign Language. Translation takes place after the stage of semantic analysis, ﬁrst of all after the predicate-argument structure of sentence is determined. Among other things, we discuss issues connected to semantic interpretation of syntactic groups (SGs), in that – semantic classiﬁcation of noun SGs and verb SGs, and problems of deducing semantic relations from syntactic ones. We also discuss in brief problems of introducing lexical and non-lexical facial expressions into sign language utterances being generated. Keywords: NLP, semantic processing, Thetos system, Polsyn analyzer, Polsem analyzer.

Introduction The research discussed below has been done in the framework of the Thetos project devoted to automatic translation of Polish texts into the Polish Sign Language [1,2]. Translation engages Thel, an intermediate language whose elements are Polish words in basic form and control markers. Each Thel utterance is transformed to an animated gesture sequence shown by an avatar (see e.g. http://thetos.aei.polsl.pl). Translation of utterances into Thel takes place after the stage of semantic analysis of the input text is ﬁnished. In our project1 , the principle of partitioning the processing into levels (see e.g. [3]) was adopted. For processing on each level a distinct analyzer is responsible, which processes the result provided by the preceding stage. The work of the Polsem semantic analyzer follows the stages of morphologic processing performed by the Polmorf analyzer [4] and the syntactic processing done by Polsyn [5]. Such approach is advantageous not only from the point of view of linguistics but also of software engineering, because it allows for implementation of a complex, multicomponent system with distributed responsibility (competence). The latter is concordant with the complex software system design theory, particularly when 1

This work was supported in part by the Polish Ministry of Science and Higher Education in 2008-2010 under Grant No. N N114 208334.

Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 234–244, 2011. c Springer-Verlag Berlin Heidelberg 2011

Semantic Analyzer in the Thetos-3 System

235

object-oriented approach is adopted [6]. The partition into levels and modules makes a system ﬂexible and readable and in the same time more reliable. This property is especially important in case if any changes both in transformation model and in system requirements have to be introduced. For the analyzers, we observe the principle to proceed independently, if only possible, without reference to the knowledge of domains situated at higher levels of processing stages hierarchy. However, we admit some pragmatically reasoned deviations. For example, on the syntax level, in order to reduce lexical ambiguity the lexical semantics can be used. To problems of disambiguation the work [7] was devoted. For some ”badly” analysed fragments we also admit reprocessing the stage that lays in the hierarchy a level below. In our works, Polish syntax has been modelled with use of a ﬁve-level SGS 2 . As to semantics, it’s so complex that only some of its numerous aspects can be taken into consideration. Our system isn’t an exception in this respect. Selection of aspects to be modelled depends on requirements towards application of semantic analysis results. Depending on that, which semantics features are being set oﬀ, diﬀerent formalisms are used for modelling. In this paper we only mention those of them, which either have been applied in our model or have been our inspiration. Below we discuss the model of semantics (or rather models of its particular aspects) that is at present used in the Polsem analyzer implementation. We also discuss the principles of how Polsem works. All experiments described in this paper were done using the Linguistic Analysis Server LAS [8].

1

Semantic Processing – General Premises

Semantic analysis is a natural continuation of the syntactic one. From the point of view of language engineering, syntax is a tool, a mechanism for expressing thoughts. The task of syntactic analysis is to provide data on the structure of the sentence, on the structure of syntactic units – ”bricks” the sentence is composed of, and on relations that occur between syntactic units. In our approach, these units are syntactic groups (SGs). Details related to SGs can be found e.g. in [9,10,5]. Results of the syntactic analysis are the data to be processed during the semantic analysis. While discussing the research into semantics, one typically sets: units, whose meaning is under examination, semantic relations, semantic representation formalism, and principles of deduction on the set of semantic representation units as well as constraints imposed by pragmatics. In the discussion of our research we will follow this scheme. 2

SGS – Syntactic Groups System, originated by [11], adapted for natural language processing. On its basis the Syntactical Grammar Groups for Polish (SGGP ) has been developed; in turn, SGGP has been the basis for the Polsyn parser implementation [5].

236

J. Romaniuk, N. Suszcza´ nska, and P. Szmal

Years of work on the Thetos system have shown that for translation purposes it is essential for the semantic analysis to establish how to build the SG’s meaning from the meanings of its component SGs, and how to build the meaning of the sentence from the meanings of SGs. In the ideal case, it would be useful to know what way the meanings of sentences compose the content of the text, and what role is played by the syntactic relations in transforming individual meanings into the content. Unfortunately, at current knowledge-of-language level it is impossible to implement such a complex analyzer. In this situation we assumed the semantic analyzer to be responsible for ﬁxing semantic units considered to be content-carriers, for assigning semantic tags to them, and for recognizing relations, which exist between them. So, units under examination are SGs, ﬁrst of all at the 3rd and 4th level, and syntactic relations at the 3rd one. On recall basis, the 4th syntax level is in traditional meaning made up of sentences, the 3rd one – of sentence component parts3 . We are also interested in semantic relations between those units. In our project the semantic research is multithreaded and refers to diﬀerent aspects of semantics, but above all to detecting the predicate-argument structure of sentences.

2

Predicate-Argument Structure

One of formalisms used in semantic analysis is the predicate-argument structure (PAS). PAS is the place to record the sentence structure elements, which are important for transmitting the content, and to set relations between them. Predicate-argument representation takes the form of a graph, whose root is the predicate element, nodes – the elements that serve as root’s arguments, and edges – the relations that bind the arguments with the predicate. Semantic structure of the sentence is written in the form of a multi-argument relation p(a1 , a2 , . . . , an ), where p is predicate, ai , i = 1, 2, . . . , n, is predicate’s argument, i is argument’s position, and there is some ﬁxed a priori function ri , which corresponds to each position. There are many reasons for which a bit diﬀerent form of semantic representation: a set of binary relations ri (ai , p), seems more comfortable4 . It’s easy to arrange a system from such units, it’s easy to search synonymic equivalents for them, to compare relation sets for diﬀerent sentences, notably to analyze different relations for the same predicate. Moreover, it’s easy to pick up units that need ﬁlling dropped components, or relations that require going out of sentence boundaries as for example while analyzing anaphoric connections. 3 4

The remaining levels are: 0th level (roughly speaking) – individual tokens (e.g. words), 1st and 2nd level. It is known that relation in form p(a1 , a2 , . . . , an ), where for each ai there exists a relation ri corresponding to it, can be transformed into a set of binary relations r1 (a1 , p), . . . , rn (an , p).

Semantic Analyzer in the Thetos-3 System

237

The idea of using this formalism in our works has risen from observation of similarity of syntactic and semantic representations. In our approach, the syntactic representation of the sentence is a directed acyclic graph, whose root is a third-level verb group (V G3 ), nodes are SGj s serving as VG’s syntactic arguments, j = 1, 2, . . . , m, where m is the count of third-level SGs (SG3 s), and edges - syntactic relations between V G3 s and SG3j s. In order to get the PAS for a sentence, it is suﬃcient to superimpose the semantic graph onto the syntactic one and to deﬁne new, semantic roles for SG3j s, in this count also for this V G3 , which is associated with the predicate. The Polsem analyzer has been built just using those principles. Description of preliminary works over Polsem, in that – description of ﬁrst versions of algorithms for transforming the syntactic representation of Polish sentence into the predicate-argument representation can be found in [12]. Units of this representation are tagged SGs. It is known that predicate arguments can be characterized by their type: referential, attributive or sentence-like (see e.g. [13]). In Polsyn, ﬁrst two argument types are used. Referential arguments are: NGs (noun SGs), IGs (proper noun based SGs), and partially – PGs (prepositional SGs); attribute arguments are: PSs (adverbial SGs), MODs (modiﬁer SGs), and the PGs’ remainder. Both count and types of arguments depend on predicate properties and are deﬁned by a grammar. Description of a grammar that ﬁxes predicate requirements against semantic features of its arguments is contained in the Polish verb syntacticgenerative dictionary [14]. In order to make use of the grammar, the dictionary has been implemented in an electronic form [15,16]. In algorithms for deﬁning semantic structure of sentence we additionally use a grammar [17], which is an adaptation of Fillmore’s case grammar [18]. For research purposes, this grammar has been streamlined: apart from predicate, which is an activity (in our notation – ACTION), the following possible elements are taken into account: performer of the activity – AGENT, object, on which the activity is being performed – OBJECT, receiver – EXPERIENCER, the place where the action is being performed – LOCATION, starting point for the activity – SOURCE, destination – GOAL, time – TIME, TIMESOURCE, TIMEGOAL.

3

Semantic Interpretation of SGs

We assume the PAS for the textual part of the sentence expressed in Thel to be the same as for the input sentence. The simplest way to get translation is to ﬁll the predicate valence places in the output sentence’s PAS by corresponding elements of the input sentence’s PAS, then linearize the Thel utterance and supplement it with control markers. Unfortunately, this way one cannot get translation of a good quality. The method is only useful in simplest cases. Translation of more complicated cases requires additional operations, in this count – semantic interpretation of sentence component parts. Attempts to solve some problems connected with this were described in [19]. For example, one of problems being discussed there is connected with interpretation of serial constructions composed of SGs that represent diﬀerent types, e.g. PS and PG:

238

J. Romaniuk, N. Suszcza´ nska, and P. Szmal

vV G3(trzeba bra´c) S S S S S S S v S S / ? GSP 3 wv v v (i) N G3(T abletki)

3 P S(rano)

P G3(po poludniu)

Fig. 1. Syntactic structure of the sentence

vACT ION(trzeba bra´c) C C C C CCW $ ' v / v v v & % OBJECT(T abletki)

T IM E(rano i po poludniu)

Fig. 2. Desired semantic structure

Tabletki trzeba bra´c rano i po poludniu. (Pills should be taken in the morning and in the afternoon.) Syntactic structure of this sentence is shown in Fig. 1, desired semantic structure – in Fig. 2. It can be seen that the problem consists in determining the semantic role of arguments expressed by SGs of diﬀerent types (PS and PG): arguments that play the same semantic roles, in speciﬁc circumstances can build serial constructions. In this case the conjunction group (GSP) plays the role of a grouping operator. Discovering semantic characteristics of PSs and PGs is connected with an adequate semantic classiﬁcation of their component elements. In this purpose, classiﬁcation systems for PSs and PGs have been developed. The grammar for PSs interpretation has been given through a dictionary, as it is in case of the aforementioned grammar that ﬁxes requirements for predicate’s arguments. The semantic dictionary for adverbs contains 3500 entries. For each entry there is an indication of its syntactic class as well as of semantic class and subclass. Additionally, information is provided if given adverb can serve as a semantic

Semantic Analyzer in the Thetos-3 System

239

modiﬁcation operator and if so, there is an indication of SG types, to which this modiﬁer can be applied. Interpretation of PG in the role of predicate argument looks a bit diﬀerently. A preposition and an NG are the enclosing PG’s components, but the PG’s interpretation not only depends of characteristics of PG’s semantic components, but also of features of ACTION and, possibly, of features of AGENT. That means that for PG interpretation, semantic classiﬁcation not only of prepositions and nouns, but also of verbs is required. Features of NG are determined with use of an attribute structure mechanism [20] on the basis of attributes of right-hand-side components of SGGP production. SG’s semantic features together with rules for selecting the SG’s representative (i.e. SG’s main component) allow for determining semantic features of SG in whole. For example, simplifying, we can say that semantic features of NG’s representative almost always characterise the NG in whole. All this suggests that a noun classiﬁcation system is needed as well as an appropriate semantic dictionary. By arrangement of it a modiﬁed classiﬁcation system from the work [21] has been used. For research purposes a list of semantic features of nouns (21 pieces), which were arranged in a hierarchy5, has been drawn up; a relationship of inheritance and multi-base succession was established between the features. On this hallmarking system deduction rules were introduced. For example, one of the rules has the form: If an oﬀspring feature can be attributed to the word, then attribution of the base feature is not required. Or otherwise: If a word is assigned a feature F which is an oﬀspring one then there belong to the word all the features inherited by F. Such approach permits generation of a set of features for an NG on the basis of one or two semantic features of its representative. For example, for noun with the feature plant the features inanimateness and concreteness are generated. To characterize a PG we use elements of the set of semantic tags, to which the set of semantic tags of PS enters as a subset. Similar work has been carried out for VG, which, as mentioned above, is associated with the predicate. Structure of VG is complex, and semantic features are determined on the basis of features of VGs main component speciﬁed by the SGGP. Usually such component is a verb. For the research purposes 6 attributes have been selected: movement, state, ownership, contact, emotions, and mental states. Semantic interpretation rules will be shown in simple terms by example. The preposition po (after) in conjunction with a group of noun in locative with the semantic feature time or event creates a time adverbial, which is later associated with the attribute TIMESOURCE: po latach, po 1945 roku, po ´smierci, po I wojnie ´swiatowej (after years, after 1945, after death, after the First World War). NG feature animateness may witness syntactic construction to change its role from adverbial into object and take the semantic role of SOURCE: po m¸ez˙ u, po rodzicach (after husband, after parents). The feature place (spatiality) of NG in locative characterizes place adverbial and attribute LOCATION: po Beskidach, po morzu (over the Beskidy Mountains, over the sea). The same feature of the 5

At the hierarchy top the features concreteness and abstractness are situated.

240

J. Romaniuk, N. Suszcza´ nska, and P. Szmal

group of noun in accusative determines the direction of AGENT’s motion, while NG acts as an endpoint GOAL for the ACTION: po suﬁt, po Hiszpani¸e (up to the ceiling, up to Spain). Use of NG of other classes requires additional analysis of the features of both ACTION and the AGENT: po cz¸e´sci, po artykuly (in part, for articles)6 .

4

Semantic Relations

Relations in the predicate-argument model do not reﬂect the entire sentence semantics. For example, in this formalism one cannot express the relation, which emerges between the subject of an event and the place of the event. Also interpretation of passive sentences like Cieplo promieniowalo od pieca (Heat radiated from the furnace)7 makes a great diﬃculty. Likewise, it is diﬃcult to interpret the internal relations occurring between the components of SG3 s. For these purposes, in semantic analyzers additional formalisms are used such as deduction mechanisms for sets of semantic relations arranged in some system: semantic networks, frames, ontologies, etc. The arrangement of such systems falls within the competence ﬁelds of knowledge representation and is not discussed here. Of course, integration of formalisms gives enormous power to express the semantics of sentences in a formal way, but by practical application of those mechanisms for our purposes the problem arises how to automatically ﬁnd semantic relations. As part of our project, eﬀorts have been made to ﬁnd the mapping syntactic to semantic relations, RelSyn ⇒ RelSem in short. We will show the process of interpretation on the example of the relationship dop2(A, B), where A and B are second level NGs. The relation being investigated is syntactic subordination relation and means that A is a syntactically superior group, B is a group subsidiary of A, the type of dependence is dop2. This relation can be semantically interpreted in several ways, among which we chose the following seven: A is a part of B, B is a part of A, B belongs to A, B consists of A, A is an attribute of B, A is an object of Bs activity, A is an attribute of Bs activity. Choosing the correct interpretation depends on many factors, including the properties of grammatical, semantic and pragmatic arguments of the relation. Examples of the results of the interpretation of the relation dop2 are shown in Fig. 3. More information about the interpretation of syntactic relations can be found in the work [22]. Inter-sentence relations: anaphoric relations and ellipses, provide another example of relations, where the semantic interpretation is necessary. An approach to the analysis of anaphora was published in the paper [23], an algorithm for interpretation of certain types of ellipses can be found in [24].

5

Facial Expressions and Gestures

Semantic interpretation of SG is needed not only in the generation of verbal statement in Thel. The non-verbal (control) part also requires a semantic analysis of 6 7

Source of the examples: IPI PAN Corpus (http://korpus.pl/) Source of the example: Pola´ nski’s syntactic-generative dictionary.

Semantic Analyzer in the Thetos-3 System RelSyn: dop2(A,B) A, features B, features zadaniem programu program (program) zadanie (task) (program’s task) (noun, G. sing.) (gerund, I. sing) elementy ﬁrmy ﬁrma (company) element (element) (company elements) (G. sing.) (N. pl.) interwalwysylki wysylka (shipment) interwal(interval) (interval of shipment) (gerund, G. sing.) (noun, N. sing) plik dokument´ ow dokument (document) plik (ﬁle) (ﬁle of documents) (G. pl.) (N. sing.) Example

241

Semantic interpretation A is an object of B’s activity B is a part of A A is an attribute of B’s activity B consists of A; A is a part of B

Fig. 3. The RelSyn ⇒ RelSem transformation

input text. One of the problems is the detection of signs of emotions in the text. Speech of the Deaf is full of facial expression, which is – analogously to body movements – a mechanism for supporting the transfer of information. Up to now, the authors of translation in the Thetos system have been focusing on the verbal content transfer, which does not require deep semantic analysis. Such content is transmitted by means of lexical gestures and lexical facial expressions of items8 . To non-lexical gestures very little attention has been paid because their interpretation is closely connected to the semantic interpretation of the SG. A source of mimics can be signs of facial expression of emotion. In [25,26] ﬁrst attempts to automatically identify manifestations of emotions in the text and transfer them into the language of the Deaf are discussed. Taken for research purposes, a classiﬁcation of the mechanisms of language that are used to express emotions in text and emotional states is given. The elements of emotion expressing are identiﬁed in eﬀect of interpretation of the arguments of sentence’s PAS. The basis for interpretation is information contained in semantic dictionaries. In the research discussed in the above-mentioned papers, from between many aspects of emotion expression we chose a mathematically meant sign of emotion. The sign is connected with expressing a positive, negative, or neutral emotion. Another aspect of emotion – its intensity – was considered in a trivial way with 2 values possible: emotion present or absent. Recently a multilevel emotion scale has been adopted. It is supported by the semantic dictionary for adjectives that contains 3500 entries, for verbs – more than 600, and for nouns – 500. This set is completed by the mentioned above adverb dictionary (3500 entries), which besides ﬁelds that enable PSs and PGs classiﬁcation also contains ones that characterize the sign and intensity of emotional load. All these dictionaries are used in Thetos during linearization; they, inter alia, allow to signal – by means of respective control marks put in output utterance – what is the required avatar’s facial expression. 8

A lexical facial expression of the Deaf is such a facial expression, which corresponds to a certain word having the gesture expression in the sign language. A non-lexical facial expression means such an expression, which is not associated with a certain word but with the general semantics of the sentence.

242

6

J. Romaniuk, N. Suszcza´ nska, and P. Szmal

Requests

One of the few emotional manifestations, which is properly automatically recognized and transferred to Thel, is a request. Both cases are analyzed, of requests expressed explicitly and implicitly. Request sign is treated as a non-verbal gesture, despite the fact that a request in the source text can be expressed in words. If the text contains a clearly expressed request, e.g. mam do Pana pro´sb¸e (I have a request to you), then the expression in Thel gets only one word: ja-mie´c-do-panpro´sba (I-have-to-you-request). Moreover, in Thel, depending on the context, two words may be used: ask and request. These two words mean the same emotiongesture request because both words to ask and a request have the same sign. For example: – Prosz¸e ci¸e. (ja-prosi´c-ty). Please. (I-ask-you). – Przynie´s mi wod¸e (ty-przynie´s´c-pro´sba-woda). Bring me water (you-bring-request-water). – Zostale´s poproszony o wod¸e (ty-zosta´c-poprosi´c-o-woda). You were asked to give water (you-are-request-give-water).

7

Conclusion

In language engineering rather a little attention is paid to the semantic transformations, this step is usually either ignored or combined with a pragmatic transformation step. In various studies of engineering, under the term semantics one understands modeling various aspects of the theory of semantics and even pragmatics, but the aim of the study is ”understanding” and storing the content of utterances in a form suitable for further electronic processing. The results of these studies are rather modest and the aspects of semantics they are related to are so diﬀerent that the comparison of them seems a bit too early. The only thing that can be noted is that none of the currently used formalisms can alone solve the problem of automatic understanding of text. Hence the need for additional research, which should provide more complex mechanisms for the semantic processing of utterances, and further – of text. Unfortunately, we also have not yet managed to piece together a single coherent system from the various parts of semantic modeling. The current version of the Polsem analyzer is based on a simpliﬁed version of semantic analysis – ﬁnding the sentence’s PAS and recognition and interpretation of several cases of a request. All other algorithms are in the stage of reﬁnement, implementation, or experimenting. In parallel, the necessary semantic dictionaries are being drawn up. Although it could be interesting for the reader, on this stage of Polsem development it is diﬃcult to make an exhaustive evaluation of analyzer’s capacity; it cannot go far beyond discussion given in this paper. In conclusion a few words about our current research. Generally, it is concerned with the problem of identifying the types of syntactic relations and SGs,

Semantic Analyzer in the Thetos-3 System

243

whose presence may indicate the occurrence of a description of emotional states. Those investigations involve three threads. The ﬁrst is classiﬁcation of types of emotional expressions (descriptions of states, assessments, requests), suitable for automatic recognition. The second is development of a complete list of SG types and syntactic relations, which express the emotional evaluation. The third thread is focused on elaboration of corresponding semantic dictionaries. The translation of Polish texts into the Polish Sign Language needs the transfer of the most important facts from the text. We believe that the identiﬁcation of emotions in a sentence and their translation into gestures is one of the ﬁrst steps to achieve this.

References 1. Szmal, P., Suszcza´ nska, N.: Selected Problems of Translation from the Polish Written Language to the Sign Language. Archiwum Informatyki Teoretycznej i Stosowanej 13(1), 37–51 (2001) 2. Szmal, P., Suszcza´ nska, N.: Using Thetos, Text into the Sign Language Translation System for Polish. In: DIALOR 2005: 9th Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL) (poster), Nancy, France (2005) 3. Przepi´ orkowski, A.: Superﬁcial Processing of the Polish Language. Exit, Warszawa (2008) (in Polish) 4. Lubi´ nski, M., Suszcza´ nska, N.: POLMORPH, Polish Language Morphological Analysis Tool. In: Proceedings of the 19th IASTED Int. Conf. Applied Informatics - AI 2001, Innsbruck, Austria, pp. 84–89 (2001) 5. Suszcza´ nska, N., Szmal, P., Simi´ nski, K.: The Deep Parser for Polish. In: Vetulani, Z., Uszkoreit, H. (eds.) LTC 2007. LNCS, vol. 5603, pp. 205–217. Springer, Heidelberg (2009) 6. Flasi´ nski, M.: Introduction to Analytic Methods of Information Systems Design. WNT, Warszawa (1997) (in Polish) 7. Suszcza´ nska, N., Szmal, P.: Problems of Disambiguation in the Thetos-3 System. Speech and Language Technology 11, 185–194 (2008) 8. Kulik´ ow, S.: Implementation of the Linguistic Analysis Server for the Thetos System – the Text into the Sign Language Translator. Studia Informatica 24(3(55)), 171–178 (2003) (in Polish) 9. Suszcza´ nska, N.: SG-model of Polish Language Syntax. Speech and Language Technology, Pozna´ n 7 (2003) (in Polish) 10. Suszcza´ nska, N.: SG-Grammar of Polish Syntax. In: Conf. Speech Analysis, Synthesis and Recognition in Technology, Linguistics and Medicine, Krak´ ow, pp. 113–117 (2005) (in Polish) 11. Gladkij, A.V.: Natural Language Syntactic Structures in Automated Communication Systems. Nauka, Moskwa (1985) (in Russian) 12. Suszcza´ nska, N., Szmal, P., Kulik´ ow, S.: Continuous Text Translation using Text Modeling in the Thetos System. International Journal of Computational Intelligence 1(4), 255–258 (2005) 13. Vetulani, Z.: Man-machine Communication. In: Computer Modeling of Linguistic Competence. Exit, Warszawa (2004) (in Polish) 14. Pola´ nski, K.: Syntactic-Generative Dictionary of Polish Verbs. Wyd. PAN, Warszawa-Wroclaw-Katowice-Gda´ nsk (1980) (in Polish)

244

J. Romaniuk, N. Suszcza´ nska, and P. Szmal

15. Grund, D.: Computer Implementation of the Polish Verbs Syntactic-Generative Dictionary. Studia Informatica 21(3(40)), 243–256 (2000) (in Polish) 16. Grund, D.: Access Function Library to Data in the Syntactic Dictionary of Polish Verbs. Studia Informatica 23(4(51)), 71–83 (2002) (in Polish) 17. Bach, M.: Construction Methods of Database Search Tasks in the Process of Translating Queries Formulated in the Natural Language. PhD. thesis, Gliwice (2004) (in Polish) 18. Fillmore, C.: The Case for Case. In: Bach, E., Harms, R.T. (eds.) Universals in Linguistic Theory, New York (1968) 19. Bach, M., Romaniuk, J., Suszcza´ nska, N.: Semantic Interpretation of Adverbial and Prepositional Groups in the Series Construction Analysis (in Polish). In: VI Conf. In˙z ynieria Wiedzy i Systemy Ekspertowe (Knowledge Engineering and Expert Systems), Wroclaw, pp. 257–267 (2006) 20. Suszcza´ nska, N., Szmal, P.: Categorial Grammar Elements in the Thetos System’s Parser. In: 2nd Language and Technology Conference, Pozna´ n, pp. 338–342 (2005) 21. Bach, M.: Electronic Dictionaries of Polish Nouns and Prepositions Semantic Features. Studia Informatica (4), 173–182 (2001) (in Polish) 22. Suszcza´ nska, N.: Automatic Identiﬁcation of Relations Between Key Abstractions of the Domain Problem. In: V Conf. In˙z ynieria Wiedzy i Systemy Ekspertowe (Knowledge Engineering and Expert Systems) Wroclaw, pp. 319–326 (2003) (in Polish) 23. Kulik´ ow, S., Romaniuk, J., Suszcza´ nska, N.: A Syntactical Analysis of Anaphora in the Polsyn Parser. In: Proceedings of the Intelligent Information Systems, pp. 444–448 (2004) 24. Romaniuk, J., Suszcza´ nska, N., Szmal, P.: Automatic Analysis of Elliptic Sentences in the Thetos System. In: Gardent, C., Gaiﬀe, B. (eds.) Proceedings of the DIALOR 2005: 9th Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL), Nancy, France, pp. 147–150 (2005) 25. Romaniuk, J., Suszcza´ nska, N.: Studies on Emotions Translation in the Thetos System. In: Proceedings of the GESPIN 2009 - Gesture and Speech in Interaction Conference, Pozna´ n (2009) 26. Romaniuk, J., Suszcza´ nska, N., Szmal, P.: An Attempt to Automatically Translate into the Sign Language Emotionally Colored Textual Utterances. Speech and Language Technology 12 (2010) (in print)

Unsupervised and Open Ontology-Based Semantic Analysis Amal Zouaq1, Michel Gagnon2, and Benoît Ozell2 1

Simon Fraser University, School of Interactive Arts and Technology, 13450 102 Ave. Surrey, BC V3T 5X3, Canada 2 Ecole Polytechnique de Montréal, C.P. 6079, succ. Centre-ville, Montréal (QC), H3C 3A7 [email protected], {michel.gagnon,benoit.ozell}@polymtl.ca

Abstract. This paper presents an unsupervised and domain independent semantic analysis that outputs two types of formal representations: discourse representations structures and flat scope-free logical forms. This semantic analysis is built on top of dependency relations produced by a statistical syntactic parser, and is generated by a grammar of patterns named α-grammar. The interest of this grammar lies in building a clear linguistically-grounded syntax-semantic interface using a representation (dependencies) commonly used in the natural language processing community. The paper also explains how semantic representations can be annotated using an upper-level ontology, thus enabling further inference capabilities. The evaluation of the α-Grammar on a hand-made gold standard and on texts from the STEP 2008 Shared Task competition shows the interest of the approach. Keywords: semantic analysis, α-Grammar, patterns, upper-level ontology.

1 Introduction Computational semantics aims at assigning formal meaning representations to natural language expressions (words, phrases, sentences, and texts), and uses these meaning representations to draw inferences. Given the progress made in computational syntax, with the availability of robust statistical parsers, it is now possible to envisage the use of syntactic parsers for a semantic analysis. This paper introduces a semantic analysis pipeline based on dependency grammars that generates two types of semantic representations: flat and scope-free logical forms and discourse representation structures (DRS) [9]. The pipeline itself includes a syntactic analysis, a semantic analysis and a semantic annotation which extract respectively dependency relations, meaning representations and ontology-based annotations of these meaning representations. The pipeline is modular by nature and enables an easy change and update of the components involved at each step. The semantic analysis itself is performed through a grammar of patterns called α-Grammar. One main interest of such grammar is its ability to provide a syntax-semantic interface between dependency grammars (which gain more and more importance in the current NLP research [2]) and semantic formalisms, thus enabling future reuse from a practical and theoretical point of view. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 245–256, 2011. © Springer-Verlag Berlin Heidelberg 2011

246

A. Zouaq, M. Gagnon, and B. Ozell

These formalisms are then annotated using an ontology which defines formally a reusable set of roles and promotes the interoperability of the extracted representations between semantic analyzers. In particular, we focus here on upper-level ontologies, which are independent from any domain and define concepts at a high-level. After explaining the motivation and theory behind our research (section 2), the paper presents the α-Grammar which outputs our semantic representations, details some of its patterns called α-structures and gives examples of the obtained representations (section 3). Section 4 details the annotation of the semantic representations and briefly explains the word-sense disambiguation algorithms involved at this step. Finally section 5 evaluates the logical forms and the discourse representation structures extracted from two corpora. Section 6 analyzes the obtained results, draws some conclusions and introduces further work.

2 Motivation, Theory and Practice The goal of this research is to create an open and unsupervised semantic analysis. Open analysis means that it can be applied on many types of texts and many domains. Unsupervised analysis means that we do not provide any training example to the system. Open information extraction is one recent challenge of the text mining community [15] and it is also one objective of the computational semantics community. In fact, Bos [3, 4] underlines that the availability of robust statistical syntactic analyzers makes it possible to envisage a deep and robust semantic analysis. One way to perform this analysis is to build a syntax-semantic interface that is to create semantic representations from syntactic representations, which are generated by a statistical syntactic parser. Here we focus on dependency grammars. In fact, dependencies are recognized as the optimal base for establishing relations between text and semantics as they abstract away from the surface realization of text and they can reveal nonlocal dependencies within sentences [12]. Moreover, there are many semantic theories based on dependency grammars such as DMRS [5] and the meaning-text theory [10]. Thus, developing a formal method to transform dependency formalism into a semantic representation is desirable from a practical and theoretical point of view. Now the question is what kind of semantic representation should be adopted. Here we focus on two types of representations: predicative logical forms and discourse representation structures. Depending on the depth of analysis required for a particular application, one can choose flat and scope-free logical forms or discourse representation structures, which are powerful representations that cover a wide range of linguistic phenomena in a unified framework [4]. In order to implement these ideas, we used the Stanford dependency parser [7] to obtain the syntactic representations. The Stanford dependencies have been used successfully in several areas [6, 15] and they can be distinguished as stated by [6] by their rich grammatical hierarchy (with the possibility of under-specifying the relation with the label “dep”) and the fine-grained description of NP-internal dependency relations. This last characteristic enables a better handling of the meaning of NP phrases. Next, we decided to use Prolog to implement a grammar for semantic analysis, also called the α-Grammar.

Unsupervised and Open Ontology-Based Semantic Analysis

247

3 The α-Grammar, a Pattern-Based Grammar Following the approach of [15], we propose an α-Grammar which transforms dependency representations into logical forms and discourse representation structures in a compositional way. This grammar is based on sound and clear linguistic principles and identifies patterns in dependency representations (named α-structures). An α-structure is a representation whose nodes represent variables coupled with syntactic parts-of-speech and whose edges represent syntactic dependency relations. These dependency relations and parts-of-speech constitute constraints on the patterns. Discovering a pattern in a dependency graph means instantiating an α-structure with the current sentence lexical information. Each pattern is linked to a rewriting rule that creates its semantic representation. A rewriting rule implements a series of transformations on the pattern including node fusion, node destruction and predicate creation. The application of a rule creates a semantic representation which uses a knowledge model. The knowledge model defines general and universal categories such as Entity, Named Entity, Supertype, Event, Statement, Circumstance, Time, Number, Measure and Attribute. The category is determined by the part-of-speech (POS) and the grammatical relationships detected in the syntactic structure. A set of examples representing α-structures and their transformation through rewriting rules is shown below: •

An example of an α-structure involving a predicative event: Y /v

•

nsubj

X /n

entity(id_X, X), event(id_e, Y, id_X)

An example of an α-structure involving potential anaphora resolution through the predicate resolve: Y /n

det

X /d

entity(id_Y, Y), resolve(id_Y)

An α-grammar can be divided into two modules: the first module involves the transformation of dependency relations into tree-like representations, and the second module implements a semantic compositional analysis. 3.1 A Tree-Like Representation Conversion In order to ease the subsequent process of compositional analysis, the grammar runs a tree converter to create a tree-like representation from the dependency relations and from the parts of speech. This tree converter is currently composed of 14 rules that modify the structure of dependencies. Each rule specifies a tree segment that should be modified and implements modification operations, namely removal, add, copy and aggregate, which can be performed on nodes and/or relations. For example, one α-structure recognizes clausal complements without a subject such as in the sentence

248

A. Zouaq, M. Gagnon, and B. Ozell

Paul likes to eat fish. The transformation associated with this rule consists in copying the subject in the clausal complement as shown below using the add operation on a node (Paul) and a link (eat, Paul):

Æ The interest of this transformation is to facilitate the compositional analysis, since the clausal complement augmented by its subject can then be interpreted in an independent manner. In addition to this type of transformations, the tree converter tackles the processing of various linguistic structures including compound noun aggregation, verb modifiers aggregation, conjunctions, and negation. Compound Noun aggregation: they are identified through the “nn” relationship (noun compound modifier) and are aggregated to form a single entity. For e.g. Guatemala army in the sentence “the Guatemala army announced …” is considered as an entity after the aggregation. Noun can also be modified by adverbial modifiers such as in the example “Genetically modified food”. In this case, there is also an aggregation operation which is performed between the head and its children. Verbs and Verb Modifiers: Verbs can be modified by particles “prt” such as “made up” or “climb up” and by auxiliaries. In general, they are considered as events predicates such as “banners flap” or as statement predicates such as “the gates of the city seem getting closer”. Conjunctions: should be identified to build various sub-trees from the input dependencies. These sub-trees constitute representations of the phrases linked by the conjunctions. For example, in the sentence “There are hens and pigs in the farm”, the tree converter extracts two sub-trees « there are hens in the farm » AND « there are pigs in the farm ». This distributive interpretation of the conjunction can be erroneous in some sentences depending on the intended meaning, and further versions of the tree converter will consider the various possible interpretations of a given conjunction. Negation: In order to handle negation, the tree converter places the negation node not as a parent of the verb and removes the subject from the scope of the negation. This way of handling negation is required by the embedding of structures in the resulting DRS, as shown in the following DRS, which represents the semantics of « the cat did not eat the mouse »: --------------------------------[id1] --------------------------------resolve(id1) entity(id1,cat) NOT: ------------------------------[id2,e1] ------------------------------resolve(id2) entity(id2,mouse) event(e1,eat,id1,id2) ---------------------------------------------------------------

Unsupervised and Open Ontology-Based Semantic Analysis

249

The resulting tree-like representation can then be processed by the compositional semantic analysis. An example of such representation (in Prolog) is: root/tree(token(flap,2)/v, [nsubj/tree(token(banners,1)/n,[]), prep/tree(token(in,3)/prep, [pobj/tree(token(wind,5)/n, [det/tree(token(the,4)/d,[])])]), prep/tree(token(outside,6)/prep, [pobj/tree(token(walls,8)/n, [det/tree(token(the,7)/d,[]), prep/tree(token(of,9)/prep, [pobj/tree(token(city,11)/n, [det/tree(token(the,10)/d,[])])])])])]).

Banners flaps in the wind outside the walls of the city

3.2 A Compositional Analysis An α-grammar uses compositional analysis to output semantic representations. Fig. 1 shows how a compositional analysis coupled with logical forms is performed on the sentence “Banners flap in the wind”. The grammar starts by examining the children of the head word “flap”. The pattern nsubj(Verb, Noun) is detected and involves the creation of an event predicate event(Id,Node,IdAgent) where Node is the verb, Id an identifier for the event, and IdAgent the identifier of the agent. The agent itself, the sub-tree nsubj/tree(token(banners,1)/n,[]), should then be explored to identify its label and eventually its modifiers and determiners. Here, banners is a leaf node which corresponds to a noun, thus leading to a predicate entity(id1, banners). This predicate has never been encountered before in the sentence, thus leading to a predicate new (id1). As can be noticed, the compositional nature of an α-grammar makes it possible to use the result of a sub-analysis (namely the created predicates and the variables) and to refer to these variables in higher level analysis. This is the case for prepositional relations on events such as in(e1, id2) in fig.1. in(e1, id2)

event(e1, flap, id1) Flap/v

prep

nsubj

in/prep Banners/n

pobj

Wind/n entity(id1, banners), new(id1)

det

The/d

entity(id2, wind), resolve(id2)

Fig. 1. Compositional Semantic Analysis [14]

250

A. Zouaq, M. Gagnon, and B. Ozell Table 1. Core verbal α-structures

Core α-structures Verb-iobj-dobj Verb-dobj-xcomp

Examples

Verb-ccomp

Mary gave {Bill}iobj a {raise}dobj The peasant carries {the rabbit}dobj, {holding it by its ears}xcomp John saw {Mary swim}ccomp

Verb-expletive

{There}expl is a small bush.

Verb-acomp Verb-prep-pcomp

Amal looks {tired}acomp They heard {about {Mia missing classes }pcomp} prep

Verb-dobj

{The cat}nsubj eats {a mouse}dobj Table 2. Modifiers α-structures

Modifiers α-structures Verb-prep-pobj Verb-tmod Verb-advcl

Examples Banners flap {in {the wind}pobj}prep Vincent arrived {last night}tmod The accident happened {as the night was falling}advcl.

Verb-purpcl

Benoît talked to Michel {in order to secure the account}purpcl. The following points are {to establish}infmod. The grass bends {abruptly} advmod.

Verb-infmod Verb-advmod

There are two kinds of α-structures in an α-grammar: core α-structures (table 1) and modifiers α-structures (table 2). Core α-structures are primary linguistic constructions that are organized into a hierarchy of rules where more specific rules are fired first. For instance, the pattern “nsubj-dobj-iobj” is higher in the hierarchy than “nsubj-dobj”. This avoids the possibility of misinterpreting a particular syntactic construction by neglecting one essential grammatical relationship. Interpreting “Mary gave Bill a book” has indeed not the same logical interpretation as “Mary gave Bill”, which means nothing. Modifiers α-structures (table 2) are auxiliary patterns that complement the meaning of core α-structures such as temporal modifiers or adverbial clause modifiers. For the moment, the grammar is composed of 36 core α-structures and 17 modifiers α-structures for a total of 53 α-structures. There are also some specific grammatical relations that we would like to further explain: Determiners: they help to identify the state of the referred object in the discourse. Some determiners, such as “the” implies that the object has already been encountered in the discourse, hence leading to a predicate “resolve (id)”. Other determiners such as “a” describes a new entity and are referred by a predicate “new (id)”. These two predicates help resolve anaphora and enable to consider the sentence elements at a discourse level rather than at a sentence level.

Unsupervised and Open Ontology-Based Semantic Analysis

251

Proper Nouns: Proper nouns, which are identified by the nnp relationship, are transformed into named entities. Prepositions: Prepositions are generally considered as modifiers patterns when they modify a verb such as in the sentence “Banners flap in the wind” or a noun such as “the gates of the city”. In both cases, a predicate representing the preposition is created such as “of (id1, id2)” where id1 is the identifier of city and id2 refer to gates. There could also be a predicate between an event id, and an object id (e.g. … flap in …). Some particular patterns such as the preposition “of” as in the example above lead to the creation of an attribute relationship. E.g. attribute (city, gate). Possessive Pronouns: Possessive pronouns enable the creation of implicit possessive relationships. For example “Benoit washes his car” implies that “Benoit has a car”, which is represented in the logical form. The same kind of deduction is used for constructions such as “The peasant’s eyes…” This enables building world knowledge. 3.3 Examples The α-grammar outputs either resolved or underspecified representations in the form of flat scope-free logical expressions and in the form of discourse representation structures. Underspecified representations means that certain ambiguities are left unresolved in the semantic output such as for example the predicate “resolve (id)” for anaphora resolution. An independent component can then be used to deal with these ambiguities. The following table illustrates the DRS-output of our α-grammar on two sentences from the competition STEP 2008 Shared Task. This competition is meant to compare the results of semantic analyzes on a shared corpus of small texts. A) An object is thrown with a horizontal speed of 20 meters per second from a cliff that is 125 m high.

B) The object falls for the height of the cliff.

--------------------------------[id2,e1,id1,id3,id4,id5,id6] --------------------------------entity(id2,object) entity(id1,undefined) event(e1,thrown,id1,id2) entity(id3,speed) entity(id4,meters) entity(id5,second) per(id4,id5) num(id4,20) of(id3,id4) attribute(id3,horizontal) with(e1,id3) entity(id6,cliff) from(e1,id6) ---------------------------------

-----------------------------[id1,e1,id2,id3] -------------------------------resolve(id1) entity(id1,object) event(e1,falls,id1) resolve(id2) entity(id2,height) resolve(id3) entity(id3,cliff) of(id2,id3) for(e1,id2) ------------------------------

As can be seen, the α-grammar correctly identifies the first part of the sentence in A “An object is thrown with a horizontal speed of 20 meters per second from a cliff” and discovers correctly the entities, events (with a correct handling of the passive voice), prepositional relations, attributes and numerical relations. However, the fragment “that is 125 m high” is ignored by the grammar. We also would like to

252

A. Zouaq, M. Gagnon, and B. Ozell

emphasize that the relation from(e1,id6) correctly applies the preposition “from” on the event “thrown”, despite the long distance dependencies. In B, the entire sentence is correctly analyzed.

4 Ontology-Based Semantic Analysis Once the semantic representations are obtained either through logical forms or through discourse representation structures, there is a need to annotate these representations using a formal and interoperable structure. In fact, one of the drawbacks of current semantic analysis is the multiplicity of the adopted formal representations, which hinder their comprehension, exchange and evaluation. Standardizing these representations through ontological indexing may help these issues. Moreover, one of the goals of computational semantics is the ability to perform inferences on the obtained representations. Using ontologies to describe the various predicates, discourse referents and conditions enables further reasoning, and builds a bridge with the semantic web community and with other semantic analysis initiatives such as semantic role labeling and textual entailment. Upper-level ontologies can provide this formal definition of a set of roles and enable the indexing of semantic representations in an interoperable way. One of these upper-level ontologies is the Suggested Upper Merged Ontology (SUMO) [11], which is widely used in the NLP community and which has gone through various development stages and experimentations, making it stable and mature enough to be taken as a “standard” ontology. Moreover, SUMO has been extended with a Mid-Level Ontology (MILO), and a number of domain ontologies, which allow coverage of various application domains while preserving the link to more abstract elements in the upper level. One interesting feature of SUMO is that its various sub-ontologies are independent and can be used alone or in combination. In our current semantic analysis, we only exploit the upper level, meaning that we take into account only the SUMO ontology itself. Another interesting point of SUMO is its mapping of concepts and relations to the WordNet lexicon [8], a standard resource in the NLP community. The SUMO-WordNet mapping associates each synset in WordNet to its SUMO sense through three types of relationships: equivalent links, instance links and subsumption links. One drawback of these mappings is that they are not always consistent: sometimes verbs are mapped to SUMO relationships and sometimes to concepts. Although these mappings cannot be considered perfect in their original form, they constitute an excellent demonstration of how a lexicon can be related to an ontology and exploited in a semantic analysis pipeline. To annotate the semantic representations and obtain SUMO-based DRS and/or SUMO-based logical forms, we tested multiple word sense disambiguation algorithms mainly inspired from Lesk algorithm and its derivatives such as [1]. We also used the most frequent sense baseline as this is commonly done in WSD competitions. These algorithms had to be applied based on a given context. In this respect, we tested various contexts such as word windows, sentence windows, and graph-based contexts extracted from the semantic logical representations [13] obtained in the semantic analysis. An example of SUMO-based logical form annotation is: outside(e1, id3), of(id3, id4), entity(id4, SUMO:City), resolve_e(id4), entity(id3, SUMO: StationaryArtifact),

Unsupervised and Open Ontology-Based Semantic Analysis

253

resolve_e(id3), in(e1, id2), entity(id2, SUMO: Wind), resolve_e(id2), event(e1, SUMO: Motion, id1), entity(id1, SUMO: Fabric), new_e(id1).

5 Evaluation The evaluation of this research is not a simple task as it involves various modules and outputs and requires a multi-dimensional evaluation: logical forms evaluation, DRS evaluation and Word sense disambiguation evaluation. One other issue that faces the evaluation of semantic analysis is the lack of gold standard on which to compare our representations. In order to tackle this issue, this evaluation relies on two corpora: 1) a first corpus of 185 sentences that we have manually analyzed and annotated to build a complete gold standard. This corpus is extracted from children stories such as Alice in Wonderland; 2) a corpus of seven texts that have been used in the STEP 2008 shared task [3] and whose objective is the evaluation of a semantic analysis. However, there is no defined gold standard on this corpus, and the task consists mainly in defining criteria to judge the effectiveness of the extracted representations based on the advice of a human expert. We are aware of the limited size of this evaluation but as can be noticed in the STEP 2008 shared task, this is a limit common to all semantic analysis systems. 5.1 Logical Form Evaluation Our logical form evaluation was carried on the first corpus. This corpus helped us in performing the logical form evaluation as well as the semantic annotation evaluation. Two metrics from information retrieval were used: precision and recall. Precision = items the system got correct / total number of items the system generated Recall = items the system got correct / total number of relevant items (which the system should have produced) Here the items designate entities and events (Table 3). Table 3. The logical form analysis results in terms of entities and events

Entities Events

Precision % 94.98 94.87

Recall % 80.45 85.5

From these experiments, it is clear that our semantic analysis is promising. Most of the time, the incorrect entities and events are due to a wrong syntactic parsing from the Stanford Parser. There are also some patterns that are not yet identified which make the recall lower. These results should be later completed with an evaluation of the whole logical representation and not be limited to entities and events. 5.2 DRS Evaluation The DRS evaluation was carried on the STEP 2008 shared task corpus. This corpus was enriched by two texts taken randomly from Simplepedia leading to a total of 58 sentences. Overall, 51 sentences were semantically analyzed by our grammar and 7

254

A. Zouaq, M. Gagnon, and B. Ozell

were ignored due to an erroneous syntactic analysis and to the lack of the appropriate patterns in the grammar. These 51 sentences were analyzed using 38 of our 53 αstructures, resulting in a rate of 72% for α-structures effective usage. In order to evaluate the obtained DRS, we first calculated a metric of precision for the conditions of each DRS in the following way: Precision = number of correct conditions / overall number of generated conditions in a DRS Second, the expert assessed the recall of each DRS by extracting the conditions of a DRS that should have been generated but that were missing due to a wrong analysis. Recall = number of correct conditions / overall number of conditions in a DRS Table 4 summarizes the average score per sentence of overall conditions, correct conditions and missing conditions and presents the obtained precision and recall values. Table 4. Mean values per sentence

#conditions 6,7

# correct conditions 5,5

# missing conditions 3,6

Precision (%) 81

Recall (%) 67

We can notice a very reasonable value of precision on these real-world examples considering that our grammar is in its first development phase. Our results also show that half of the sentences obtain more than 90% precision. However, the recall is still to be improved. Table 5 shows a more fine-grained analysis of the obtained DRS by calculating the precision obtained on the various condition categories especially entities, events and attributes. All the other categories are classified under the label “other”. These results are those obtained after the analysis of the 51 sentences. Table 5. Mean values by DRS condition categories

# conditions Entities Events Attributes Others Total

Precision %

152

# correct conditions 139

56 44 81 333

43 37 54 273

77 84 64 82

91

We can notice that entities are generally well-identified, followed by attributes and then by events. The errors made by our grammar in event recognition are mostly due to missing α-structures (7 cases over 13), to errors in the syntactic analysis (3 cases) or to an erroneous conversion into a tree structure (2 cases). Regarding attributes, all the errors are made on the same text and are related to particular modifiers wrongly

Unsupervised and Open Ontology-Based Semantic Analysis

255

interpreted (e.g. “The other gas giants are Saturn and Uranus”, “only…”, etc.). Finally the results of the “other” label indicate that further development should be made to enhance our grammar.

6 Conclusion and Further Work This paper presented the α-Grammar, a semantic analysis pattern-based grammar that produces discourse representation structures and logical forms from free texts. With the increase in the use of dependency grammars as a syntactic formalism, building a conversion process from dependency relations to semantic representations is justified from a practical point of view. Moreover, our approach proposes a semantic analysis pipeline where the various modules (the syntactic analysis, the semantic analysis and the ontology-based annotation) are independent, meaning that they can be easily evolved or replaced from a software engineering perspective. The other interest of our work is the ability to standardize the generated semantic representations through the use of an upper-level ontology. This also enhances the inference capabilities over the extracted representations. Finally, our computational semantics approach is domainindependent and unsupervised, which enables better reuse in multiple domains and applications. In future work, we plan to enhance the grammar by discovering new patterns using manual analysis of texts but also automatic pattern learning approaches. This will help us improve the precision and recall of the semantic analysis. We also plan to handle more complex discourse structures and anaphora resolution. Finally, we would like to extend the scale of the corpora used for the evaluation and to compare our DRS with the DRS extracted by Boxer [4] on the same corpora. Acknowledgements. The authors would like to thank Prompt Quebec, UnimaSoft Inc. and the FQRNT for their financial support. Amal Zouaq is funded by a postdoctoral fellowship from the FQRNT.

References 1. Banerjee, S., Pedersen, T.: Extended gloss overlaps as a measure of semantic relatedness. In: Proc. of the 18th Int. Joint Conf. on AI, Mexico, pp. 805–810 (2003) 2. Bonfante, G., Guillaume, B., Morey, M., Perrier, G.: Réécriture de graphes de dépendances pour l’interface syntaxe-sémantique. In: Proc. of TALN 2010, Montreal (2010) 3. Bos, J.: Introduction to the Shared Task on Comparing Semantic Representations. In: Proceedings of STEP 2008 Conference, pp. 257–261. College Publications (2008a) 4. Bos, J.: Wide-Coverage Semantic Analysis with Boxer. In: Proceedings of STEP 2008 Conference, Research in Computational Semantics, pp 277–286. College Publications (2008b) 5. Copestake, A.: Slacker semantics: Why superficiality, dependency and avoidance of commitment can be the right way to go. In: Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pp. 1–9. Association for Computational Linguistics, Athens (2009)

256

A. Zouaq, M. Gagnon, and B. Ozell

6. De Marneffe, M.-C., Manning, C.D.: The Stanford typed dependencies representation. In: COLING Workshop on Cross-framework and Cross-domain Parser Evaluation (2008) 7. De Marneffe, M.-C., MacCartney, B., Manning, C.D.: Generating Typed Dependency Parses from Phrase Structure Parses. In: Proc. of LREC, pp. 449–454 (2006) 8. Fellbaum, C.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998) 9. Kamp, H., Reyle, U.: From Discourse to Logic. Introduction to Model-theoretic Semantics of Natural Language. Formal Logic and Discourse Representation Theory, Studies in Linguistics and Philosophy (1993) 10. Melcuk, I.: Dependency Syntax: Theory and Practice. State Univ. of New York Press, Albany (1988) 11. Pease, A., Niles, I., Li, J.: The Suggested Upper Merged Ontology: A Large Ontology for the Semantic Web and its Applications. In: Proc. of the AAAI Workshop on Ontologies and the SW, Canada (2002) 12. Stevenson, M., Greenwood, M.A.: Dependency Pattern Models for Information Extraction, Research on Language & Computation, pp. 13–39. Springer, Heidelberg (2009) 13. Zouaq, A., Gagnon, M., Ozell, B.: Can Syntactic and Logical Graphs help Word Sense Disambiguation? In: Proc. of LREC 2010 (2010) 14. Zouaq, A., Gagnon, M., Ozell, B.: Semantic Analysis using Dependency-based Grammars and Upper-Level Ontologies. International Journal of Computational Linguistics and Applications 1(1-2), 85–101 (2010) 15. Zouaq, A.: An Ontological Engineering Approach for the Acquisition and Exploitation of Knowledge in Texts, PhD Thesis, University of Montreal (2008) (in French)

Non Compositional Semantics Using Rewriting Paul Bédaride1 and Claire Gardent2 1 INRIA/LORIA 615 rue du jardin botanique, 54600 Villers lès Nancy, France [email protected] 2 CNRS/LORIA 615 rue du jardin botanique, 54600 Villers lès Nancy, France [email protected]

Abstract. We present a system for recognising sentential entailment which combines logical inference with a semantic calculus producing “normalised semantic representations” using a cascade of rewriting systems. We start by presenting the core rewrite rules underlying our semantic calculus. We then focus on the detection of entailment relations between sentence pairs involving noun/verb alternations and we show that the system correctly predicts a range of interactions between basic noun/verb predications and semantic phenomena such as quantiﬁcation, negation and non factive contexts.

1

Introduction

As has been repeatedly argued, detecting whether a given sentence S1 implies some other sentence S2 is a basic semantic task that natural language understanding systems must be able to perform. Consequently, entailment detection has been the focus of intense research in particular since the inception, in 2005, of the RTE (Recognising textual entailment) challenge [6]. In this paper, we focus on detecting entailments between sentences involving a nominal/verbal alternation such as for instance, the following sentence pairs: (1) a. “Assuming no dramatic fluctuation in interest rates , the association expects to achieve near record earnings in 1990.” → “If interest rates do not fluctuate dramatically , near record earnings are expected in 1990 .” → “Unless interest rates fluctuate dramatically, near record earnings are expected .” The approach we propose takes a middle path between the logical approach adopted by semanticists and the similarity based reasoning resorted to by many RTE systems [6]. As [10] has argued, while the ﬁrst approach produces semantic representations that are too brittle to handle real text (for example, [4]’s system was only able to ﬁnd a proof for less than 4% of the problems in the RTE1 test set), the second fails to adequately handle commonplace semantic phenomena such as e.g., negation, quantiﬁcation or non-factive contexts. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 257–267, 2011. c Springer-Verlag Berlin Heidelberg 2011

258

P. Bédaride and C. Gardent

To overcome these shortcomings, we combine a logic based approach with a robust calculus of semantic representations in which joint syntactic/semantic structures produced by a semantic role labeler (SRL) are rewritten into ﬁrst order logic (FOL) formulae. Because our semantic calculus is based on joint SRL – rather than simply phrase – or dependency-structures, it provides an appropriate level of abstraction from syntactic diﬀerences that are irrelevant to entailment detection. For instance, it ensures that the semantic representations for “Rome was destroyed by the Barbarians”, “The Barbarians have destroyed Rome”, “Rome’s destruction by the Barbarians” and “Barbarians destruction of Rome” are all identical1 . Additionally, the use we make of a general rewriting system allows for abstractions and generalisations that are diﬃcult to capture in a strictly compositional system à la Montague. For instance, it permits capturing both the restriction and the scope of a quantiﬁer with a single rule applying to a non local fragment of the dependency tree namely, the fragment containing the determiner, its associated nominal and the verb phrase over which the quantiﬁer scopes. More generally, the use of a general rewriting system on joint syntactic/semantic structures facilitates semantic construction in two ways. First, all structures built before semantic construction can be referred to by the rewrite rules i.e., the phrase structure tree, the dependency graphs and the SRL predicate/argument labelling. Second, decisions can be made based on global (i.e., the whole NP substree) rather than local (i.e., a word and its immediate argument as done in most lexical frameworks) information. In practice, this means that, instead of having to specify a lambda term for each possible syntactic environment of a given word type, we can deﬁne very general rules for syntactically similar constituents such as NPs. An additionnal interesting property of the rewriting approach to semantic construction is that it makes it particularly simple to choose how much to rewrite i.e., how ﬁne grained semantic construction should be. For instance, we could choose to only rewrite arguments and ignore modiﬁers. The paper is structured as follows. We ﬁrst sketch the semantic role labeler we developed for verbs and nouns (Section 2). We then show ﬁrst, how to derive logic based semantic representations from the joint syntactic/semantic structures output by this labeler and second, how to used the resulting logical formulae to recognize (non) sentential entailments (Section 3). Finally in section 4, we report on a ﬁrst evaluation showing that the resulting system permits recognising some of the expected inferences. 1

Note that although grammar based systems are in principle able to capture most of these equivalences, in practice they often fail to because the representations they produced closely reﬂects the input string and in particular, contains most of its function words. For instance, the representations associated by Boxer [5] with the four “Rome phrases” above are all diﬀerent: the passive diﬀers from the active in that it contains a by predication reﬂecting the use of the agent phrase; the nominal versions diﬀer from the verbal one in that it identiﬁes a topic whilst the verbal one does not; and the two nominal versions diﬀer in that they each contain either a by or an of predication depending on the function words they contain.

Non Compositional Semantics Using Rewriting

2

259

Semantic Role Labelling

In order to recognise that “David Hiddleston was killed” can be inferred from “The avalanche killed David Hiddleston on the spot”, it must ﬁrst be recognised that X killed Y entails Y was killed. Consequently, many of the recent entries in the annual Recognizing Textual Entailment (RTE) competition have used rewriting in a variety of ways, though often without distinguishing it as a separate subproblem. Based on this observation, we developed an entailment detection system in which rewriting system is modelled using a standard rewriting tool called GrGen [7] which is at once eﬃcient, notationally expressive and used in multiple domains (e.g., formal calculus, combinatoric algebra, operational semantics). The ﬁrst component of this entailment detection system is a hybrid semantic role labeler which combines the Stanford statistical parser with a set of rewrite rules converting the Standford dependency structures (SD) produced by that parser into joint syntactic/semantic structures where verbs and nouns predications are assigned identical thematic representations. In particular, the thematic structure of each of the four versions of “Rome was destroyed by the Barbarians” given above will be identical. For further details about this SRL procedure, we refer the reader to [1, 2] where we explain how we derive rewrite rules which permit capturing the predicate/argument structure of verbal and nominal propositions. To evaluate our semantic role labeller, we computed recall and precision against the ConLL 2009 data, a corpus of written English annotated with joint syntactic/semantic dependency structures. This gives a mesure of how good the system is at normalising verbal and nominal dependency structures. The results are given in the following table: precision recall f-score Unlabeled 83.56% 65.36% 73.35 Labeled 72.66% 56.83% 63.78 The unlabeled score gives the proportion of correct predicate/argument dependency found (for verbs and nouns) while the labeled score additionally checks the speciﬁc relation holding between predicate and argument. The overall score situate our labeller in the middle range of ConLL 2009 joint labellers (F1 ranging from 36.05 to 85.44) with a reasonably good precision but a low recall due partly to the fact that the Stanford parser often fails to return the correct analysis2.

3

Building and Using Semantic Representations

The second component of our RTE system is a semantic module which includes two steps. First, the joint SRL structures output by the semantic role labeller described in the previous section are rewritten into ﬁrst-order logic formulae. Second, entailment is checked by applying automated reasoners to these formulae. We now describe each of these two steps. 2

[9] report a label F-mesure of 86.3% on section 23 of the Penn Treebank.

260

3.1

P. Bédaride and C. Gardent

From Labelled Dependency Structures to FOL Formulae

The translation of a joint syntactic/semantic structure into a FOL formula is again performed using rewriting. In what follows, we illustrate the semantic representations built by our system and the rewrite rules which we use to construct these. Semantic Representations. Our semantic calculus outputs neo-davidsonian semantic representations [8] where verbs introduce an existentially quantiﬁed event variable and verb arguments denote entities related to this event by a thematic relation. As is well known, this allows for a simple treatment of modiﬁcation. It also simpliﬁes the treatment of entailments between sentences containing the same verb used with distinct syntactic arity (e.g., 2). (2) T: “John sends a book to Mary” sem: john(j) ∧ ∃B.(book(B) ∧ mary(m) ∧ ∃S.(send(S) ∧ arg0(S, j)∧ arg1(S, B) ∧ arg2(S, m))) H: “a book is sent” sem: ∃B.(book(B) ∧ ∃S.(send(S) ∧ arg1(S, B)) Entailment: Vrai The Rewriting System. Our semantic calculus consists of six rewrite rules which apply in a ﬁxed order, each rule taking as input the output of the previous step. We ﬁrst, sketch each of these rules. We then illustrate their working (section 3.2) by means of an example. The intuition behind our semantic calculus is as follows. First we create as many nodes as there are nodes in the predicate/argument structures built by the previous rewriting step i.e., the semantic role labelling step. That is, we create a node for each verb, each noun and each argument labelled as such by the semantic role labelling procedure. Second, we expand each of these nodes to a subtree representing the meaning of the corresponding constituent (i.e., the meaning of the corresponding NP, V and/or PP). Third, we link these local trees together. Fourth, we label each nodes with the appropriate predicate, connector or variable. Fifth we conjoin the resulting trees together. More speciﬁcally, the main rewrite rules are the following. create_root : creates the root node of the formulae representing the meaning of the sentence. create_nodes : creates as many nodes as there are predicates and arguments identiﬁed by the semantic role labelling procedure. expand_nodes : expands each created node with a sub-tree representing the meaning of the corresponding constituent. connect : link sub-trees together using the appropriate connectives and scope domain. label_nodes : label nodes with the appropriate literals. link_to_root : link the trees representing the meaning of each clause in the sentence to the root node.

Non Compositional Semantics Using Rewriting

261

Depending on the speciﬁc syntactic construction considered (relative clause, deverbal nominal or quantiﬁed NP), the general connect rule will be instantiated in three slightly diﬀerent ways. The system always terminates because nothing is ever removed from the structures (rewriting is cumulative and progressively enriches each structure built by the preceding step) and each rule contains a negative ﬁltering pattern which ensures that the same rule can only apply once to the same structure. Negation and Sentence Connectives. Space is lacking to explain in detail the treatment of other semantic phenomena such as e.g., negation or sentence connectives. Brieﬂy, we currently deal with these by introducing another rule which searches the syntactic structure for connectors and negation words and rewrites them by combining the semantics of these words with that of their arguments. If necessary, the semantics of the arguments can be modiﬁed to account for the interaction with the context. For instance, an existential will be rewritten to a universal when it occurs in a universally quantifying context (e.g., “If a man owns a donkey, he feeds it”). 3.2

Illustrating Example

The examples in (3) show the type of formulae our calculus produces. Note in particular, that in our approach, scope reﬂects linear order. That is, we do not account for scope ambiguity and take as default interpretation the interpretation given by the linear order of the words in the sentence. (3) a. “John dances with every woman” john(j) ∧ ∀w.(woman(w) ⇒ (∃d.dance(d) ∧ arg0(d, j) ∧ arg1(d, w))) b. “Every man who dances with every woman, lives in Paris” ∀m.((man(m)∧∀w.(woman(w) ⇒ ∃d.(dance(d)∧arg0(d, m)∧arg1(d, w)))) ⇒ (∃l.live(l) ∧ arg0(l, m) ∧ arg1(l, p)))

c. “Every man dances with every woman who lives in Paris” ∀m.(man(m) ⇒ (∀w.(woman(w)∧(∃l.live(l)∧arg0(l, w)∧arg1(l, p))) ⇒ ∃d.(dance(d) ∧ arg0(d, m) ∧ arg1(d, w)))) d. “Every man loves to dance with every woman” ∀m.(man(m) ⇒ (∃d.(dance(d) ∧ arg1(f, d) ∧ arg0(d, m) ∧ ∀w.(woman(w) ⇒ arg1(d, w)))))

To illustrate the working of our calculus, we now run through an example and show how each of the six rules of our calculus successively applies to build the semantic formula representing the meaning of “Every man loves a woman”. The ﬁrst step applies the create_root and create_nodes rule to create a root and a set of “seed” nodes each of which is licensed by and related to, a node created by the semantic role labelling procedure. Figure 2 shows the structures built at this stage. Four nodes are added to the dependency and the SRL graphs built by the previous steps : the node named r which will be the root of the semantic formula being built and is linked to the S-node, the node named hp

262

P. Bédaride and C. Gardent

Constituents

Dependencies

SRL

Semantic

Node

cstnode

depnode

srlnode

semnode

Edge

cstedge

depedge

srledge

semedge semcedge

Fig. 1. Legend for the representation of graphs

licensed by the predicate node introduced by the verb “loves” and the two nodes ha0, ha1 licensed by the two verbal arguments “man” and “woman”. The second step uses the expand_nodes rule to build the substructures that can be associated with the root and the seed nodes given the available dependency and SRL structures. If the seed is linked to a dependency node of nominal category (i.e., if it is licensed by an NP), the subtree built will be that of a generalised quantiﬁer i.e., a tripartite structure consisting of a quantiﬁer, a restriction and a scope where the quantiﬁer will be determined by the speciﬁc determiner dominated by the noun (e.g., a universal for “every” or “all” and an existential for “a”). Figure 3 shows the structure built by the expand_nodes rule for ha0 (“every man”) and for ha1 (“a woman”)3 . Seed nodes licenced by a predicate (i.e., a verb or a deverbal nomial) trigger the construction of a structure representing an existentially bound event variable. Figure 3 (structure in the middle) shows the structure added in our example, when processing the predicative p node licensed by “loves”. The next step (connect rule) connects the substructures built so far to each other. In essence, this step determines the scope of each scope bearing operator. As mentioned above, scope is determined by the linear order of words in the sentence and in general, scope will be speciﬁed by adding an edge between (the scoping node of) the scope bearing operator and the head of the semantic substructure licensed by the next syntactic argument in the sentence (e.g., by adding a link from the scoping node sa0 of “every man” to the root node ha1 of the head of the subformula licenced by “a woman”). Other cases include relative clauses and sentential arguments. For relative clauses, the restriction node of the relativised argument is linked to the head of the following argument in the relative clause or to its verb if the relative clause contains no argument. Further, sentential arguments are linked to the verb they complement. Here we only show the more general case of how verb and NP semantics connect together. Figure 4 shows the linking edges added by the connect rule given the example we consider. The scoping node sa0 of “every man” is linked to the root node ha1 of the head of the subformula licenced by “a woman”). Further, the restriction+scope node rsa1 of the existential “a 3

Since conjunction is commutative, we do not preserve for existential quantiﬁers, the distinction between scope and restriction.

Non Compositional Semantics Using Rewriting

:esemhead

r:semroot

263

:ncst cst : ROOT

:ecst :ncst cst : S

:ecst

:ecst :ncst

:ncst

cst : N P

cst : V P

:ecst

:ecst

:ecst

:ecst

:nword

a0:nword

p:nword

wid : 1 word : “every” lemma : “every” used : F alse cst : DT

wid : 2 word : “man” lemma : “man” used : F alse cst : N N

wid : 3 word : “loves” lemma : “love” used : T rue cst : V BZ

:det ha0:semconj

:nsubj :esemhead

ad he

m se :e

hp:semconj :arg0

:epred :nprop

wid : 3 word : “love” lemma : “loves” used : T rue rewrittenby : nx0V nx1

:ncst pos : N P

:ecst

:ecst

:nword

a1:nword

wid : 4 word : “a” lemma : “a” used : F alse cst : DT

wid : 5 word : “woman” lemma : “woman” used : F alse cst : N N

:det

:dobj

:esemhead ha1:semconj

:arg1

Fig. 2. Phrase structure, dependency graph, semantic role labelling and seed and root nodes for the sentence “Every man loves a woman”

p:nword

a0:nword

:con0 ra0:semconj

varid : 2

:semhead ha1:semconj :con0 :semexists

:semscope

:semimpl

:semexists

:semquant

:con0

:con0

:semrest

varid : 0

:semhead hp:semconj :semscope

:semforall

:semquant :semrest

:con0

:semscope

:semrest

:semquant

:semhead ha0:semconj

a1:nword

varid : 1

:con0

:con0

rsp:semconj

rsa1:semconj

:con1 sa0:semconj

Fig. 3. Substructures built by the expand_nodes rule for “every man”, “loves” and “a woman” respectively

264

P. Bédaride and C. Gardent sa0:semconj

rsp:semconj

rsa1:semconj

:con0

:con0

ha1:semconj

hp:semconj

Fig. 4. Connecting substructures using the connect rule

woman” is linked to the root node hp of the predicate “loves” of the head of the subformula licenced by “a woman”). Next, the label_nodes rule expand existing semantic nodes with substructures representing the literals that should be associated with these nodes. For arguments, the restriction node is expanded with a substructure bearing as predicate, the lemma of the nominal licensing the argument and as variable, the variable bound by the quantiﬁer licensed by the argument determiner. Similarly, a substructure is added for verb and noun predicates which contains as predicate, the lemma of the licensing verb or deverbal noun and as variable, the variable bound by the existential quantiﬁer licensed by this verb/deverbal. Further, substructures are added for each of the verb arguments where each such substructure relate the verb event variable to the argument variable via the thematic role relation given by the SRL procedure. Figure 5 show the substructures added by the label rules in our example. ra0:semconj

rsp:semconj

rsa1:semconj

:con0

:con0

:con0

:sempred

:sempred

:con0

pred : “man”

:sempred

:con0

pred : “love”

pred : “woman”

:var1

:var0

:var0 :semvar

:sempred

:semvar

:sempred

:semvar

varid : 0

pred : “arg0”

varid : 2

pred : “arg1”

varid : 1

:var0

:var1

:var0

:var1

:semvar

:semvar

:semvar

:semvar

varid : 2

varid : 0

varid : 2

varid : 1

Fig. 5. Substructures added by the label_nodes rule

The last step consists in linking the root of the semantic formula being built to dangling substructures (link_root rule). In our example, this results in yielding as ﬁnal output structure the structure shown in Figure 6. From this representation, we can then deterministically derive the following logical formula for the sentence “Every man loves a woman”: ∀V 0.(man(V 0) ∧ ∃V 1.(womam(V 1) ∧ ∃V 2.(love(V 2) ∧ arg0(V 2, V 0) ∧ arg1(V 2, V 1))))

Non Compositional Semantics Using Rewriting

:semhead

:ncst

:esemhead

r:semroot

265

cst : ROOT :ecst :ncst

cst : S :ecst

:ecst

:ncst

:ncst

cst : N P

cst : V P :ecst

:ecst

:ecst

:ecst p:nword

a0:nword

wid : 1 word : “every” lemma : “every” used : F alse cst : DT

wid : 2 word : “man” lemma : “man” used : F alse cst : N N

wid : 3 word : “loves” lemma : “love” used : T rue cst : V BZ

:nsubj

:ncst

pos : N P :ecst

:ecst

0 rg

:det

:e p

:a

re

d

:nword

:semhead

:nprop

:semquant

ha0:semconj

:semforall

varid : 0

:semrest

:semscope

:con0

:semscope :det :dobj

:arg1

:con0

:semexists

varid : 1

:con0

:con0

rsp:semconj

rsa1:semconj

:con0

:sempred

:con0

:sempred

:con0

name : “love” :con0

:semquant

:con0 :con0

:con0

:con1 sa0:semconj

:semrest

ha1:semconj

hp:semconj

varid : 2

:semimpl

:con0

wid : 5 word : “woman” lemma : “woman” used : F alse cst : N N

:semexists

:con0

ra0:semconj

a1:nword

wid : 4 word : “a” lemma : “a” used : F alse cst : DT

:semhead

:semscope

:semhead

:semrest

:semquant

wid : 3 word : “love” lemma : “loves” used : T rue rewrittenby : nx0V nx1

:nword

name : “woman” :var1

:var0

:sempred

:sempred

:semvar

:sempred

:semvar

name : “man”

name : arg0

varid : 2

name : arg1

varid : 1

:var0

:var0

:var1

:var0

:var1

:semvar

:semvar

:semvar

:semvar

:semvar

varid : 0

varid : 2

varid : 0

varid : 2

varid : 1

:con0

Fig. 6. Full ﬁnal representation of the sentence “Every man loves a woman”

266

3.3

P. Bédaride and C. Gardent

Checking Entailment

Given the above translation into FOL, textual entailment between sentences can be tested by checking for logical entailment between the associated FOL formulae [3]. In practice, we get formulas for the 5 ﬁrst syntactic analyses and select the analysis with the highest semantic score where the scoring system favors longer predications (i.e., predications with a higher number of dependents) over shorter ones. We then check logical entailment between the two representations associated with the most highly scored analysis of each of the two sentences to be compared.

4

Evaluation

To evaluate our approach on entailment detection, we manually built a benchmark of 20 sentence pairs involving a N/V variation. The benchmark encompasses 4 main types of entailment patterns dubbed respectively, simple, light-verb, sem and adj/adv. A simple pattern is one such as 4 where the entailment depends only on a nominalisation. (4) “Legislation to lift the debt ceiling is ensnarled in the ﬁght over cutting capital-gains taxes.” → “Capital-gains are taxed.” A light-verb pattern involves a light verb construction such as for example: (5) a. “An acceleration of investments gives Japanese companies control of large, highly visible U.S. corporations, such as Columbia Pictures Entertainment Inc.” → “Japanese companies control U.S. corporations.” → “An acceleration of investments controls U.S. corporations.” b. “It is operating under Chapter 11 of the federal Bankruptcy Code, giving the company court protection from creditors’lawsuits.” → “Chapter 11 of the federal Bankruptcy Code protects the company.” → “The company court protects from creditors’ lawsuits” The adj/adv type illustrates the interaction between predication and modiﬁers: (6) “Countries with inadequate protections for intellectual-property rights could be hurting themselves.” → “Countries which inadequately protect intellectual-property rights could be hurting themselves.” Finally, a sem pattern illustrates the interaction between basic predications and semantic phenomena such as quantiﬁcation, negation and non factive contexts. For instance, (1) illustrates the interaction of if (verbalised by assuming, if or unless) negation (verbalised by unless, no) and N/V relations (fluctuation/fluctuate). For each of the sentence pairs contained in the benchmark, the system correctly predicts the (non) entailment relation.

Non Compositional Semantics Using Rewriting

5

267

Conclusion

Although it remains limited in scope, the system presented here lays the basis for an approach to entailment detection that combines a robust semantic calculus with logical based reasoning. It thereby departs from [4] in that the semantic representations are less brittle and from [10] in that it integrates both the role labelling abstraction of SRLs and logical rather than natural logic reasoning. We have illustrated the potential of the approach by showing how it could handle a limited range of interaction between nominal predication, verbal predication and logical connnectives. Current work concentrates on extending the system coverage and onevaluating it on a full size benchmark designed to illustrate a wider range of interaction between basic predications and the various semantic phenomena potentially present in their sentential context.

References [1] Bedaride, P., Gardent, C.: Normalising semantics: a framework and an experiment. In: IWCS 2009, International Conference on Computational Semantics, Tilburg, The Netherlands (2009) [2] Bedaride, P., Gardent, C.: Noun/verb inference. In: 4th Language and Technology Conference, Poznan, Poland (2009) [3] Blackburn, P., Bos, J., Kohlhase, M., de Nivelle, H.: Inference and computational semantics. In: Bunt, H., Thijsse, E. (eds.) IWCS-3, pp. 5–19 (1999) [4] Bos, J., Markert, K.: When logical inference helps determining textual entailment (and when it doesn’t). In: Pascal, Proceedings of the Second Challenge Workshop, Recognizing Textual Entailment (2006) [5] Curran, J.R., Clark, S., Bos, J.: Linguistically motivated large-scale nlp with c&c and boxer. In: ACL 2007 demo, pp. 29–32 (2007) [6] Dagan, I., Glickman, O., Magnini, B.: The PASCAL recognising textual entailment challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 177–190. Springer, Heidelberg (2006) [7] Geiß, R., Batz, G.V., Grund, D., Hack, S., Szalkowski, A.: GrGen: A fast SPObased graph rewriting tool. In: Corradini, A., Ehrig, H., Montanari, U., Ribeiro, L., Rozenberg, G. (eds.) ICGT 2006. LNCS, vol. 4178, pp. 383–397. Springer, Heidelberg (2006) [8] Higginbotham, J.: On events in linguistic semantics. In: Higginbotham, J., Pianesi, F., Varzi, A. (eds.) Speaking of Events. Oxford University Press, Oxford (2000) [9] Klein, D., Manning, C.D.: Accurate unlexicalized parsing. In: ACL, Sapporo, Japan, pp. 423–430 (July 2003), http://www.aclweb.org/anthology/P03-1054 [10] MacCartney, B.: Natural language inference. Ph.D. thesis, Stanford University (2009)

Defining Specialized Entailment Engines Using Natural Logic Relations Elena Cabrio1,2 and Bernardo Magnini1 1

2

FBK-Irst, Via Sommarive 18, 38123 Povo, Trento - Italy University of Trento, Via Sommarive 14, 38123 Povo, Trento - Italy {cabrio,magnini}@fbk.eu

Abstract. In this paper we propose a framework for the deﬁnition and combination of specialized entailment engines, each of which able to deal with a certain aspect of language variability. Such engines are based on transformations, and we deﬁne them taking advantage of the conceptual and formal tools available from an extended model of Natural Logic (NL). Given a T,H pair, each engine performs atomic edits to solve the speciﬁc linguistic phenomenon it is built to deal with, and assigns an entailment relation as the output of this operation. NL mechanisms of semantic relations composition are then applied to join the output of each single engine, in order to obtain a global entailment judgment for a pair. Keywords: Textual Entailment, Natural Logic.

1

Introduction

The Recognizing Textual Entailment (RTE) task [6] [7] aims at capturing a broad range of inferences that are relevant for multiple Natural Language Processing applications, and consists of deciding, given two text fragments, whether the meaning of one text (the text T ) is entailed, i.e. can be inferred, from another text (the hypothesis H ), as in pair 2051 : T: A team of European and American astronomers say that a recently discovered extrasolar planet, located not far from Earth, contains oceans and rivers of hot solid water. The team discovered the planet, Gliese 436 b, [...]. H: Gliese 436 b was found by scientists from America and Europe.

Given a T,H pair, three diﬀerent judgments are allowed: yes, if there is a degree of entailment between T and H (T H), contradiction, when some degree of contradiction is found (T ¬H), and unknown, in case neither entailment nor contradiction is detected between T and H (T H ∧ T ¬H) [10]. As the above example shows, various levels of complexity can be highlighted in TE, involving almost all linguistic phenomena of natural languages, including lexical, syntactic and semantic variations. While most of the research so far in 1

All the examples reported in this paper are extracted from RTE-5 data set [1].

Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 268–279, 2011. c Springer-Verlag Berlin Heidelberg 2011

Deﬁning Specialized Entailment Engines Using Natural Logic Relations

269

TE has proposed omnicomprehensive approaches, we think that crucial progress may derive from a focus on single linguistic phenomena involved in TE and on their combination. Some recently published works share our same considerations, and attempts to better analyze the linguistic phenomena that are relevant for TE and their distribution in the data have been carried out, highlighting a new interest in the TE community towards this direction. In (Sammons et al. 2010) [18], limitations in the existing RTE evaluation scheme are highlighted, and a linguisticallymotivated analysis of entailment data based on a step-wise procedure to resolve entailment decision is presented. Bentivogli et al. (2010) [2] propose a methodology for the creation of specialized data sets for TE, made of monothematic T-H pairs, i.e. pairs in which only one linguistic phenomenon relevant to the entailment relation is highlighted and isolated. A pilot resource is created annotating a sample of RTE data with the linguistic phenomena underlying the entailment relation, and extracting monothematic pairs from the original ones. Taking advantage of this work, in [3] we present a methodology for the qualitative evaluation of TE systems based on the analysis of the system behavior on monothematic pairs with respect to the behavior on corresponding original pairs. As introduced before, our intuition is that the more a system is able to correctly solve the linguistic phenomena underlying the entailment relation separately, the more the system should be able to correctly judge more complex pairs, in which diﬀerent phenomena are present and interact in a complex way. Such intuition is motivated by the notion of meaning compositionality, according to which the meaning of a complex expression is determined by its structure and by the meaning of its constituents [8]. In a parallel way, we assume that it is possible to recognize the entailment relation of a T,H pair (i.e. to correctly judge the entailment/contradiction relation) only if all the phenomena contributing to such a relation are resolved. Basing on such considerations, we propose a framework for the deﬁnition and combination of specialized entailment engines, each of which able to deal with a certain aspect of language variability. In [13] we provided a preliminary deﬁnition of this method assuming a distance-based approach; here we oﬀer a more generic framework based on transformation, and we deﬁne it taking advantage of the conceptual and formal tools available from an extended model of Natural Logic (NL) [16]. Given a T,H pair, each engine performs atomic edits to solve the speciﬁc linguistic phenomenon it is built to deal with, and assigns an entailment relations as the output of this operation. We provide an operational deﬁnition of atomic edits allowed for a speciﬁc phenomenon in terms of application of entailment rules. Once the specialized engines have assigned an entailment relation to each phenomena relevant to inference in a speciﬁc pair, NL mechanisms of semantic relations composition are applied to join the output of each single engine, in order to obtain a global entailment judgment for a pair. The paper is structured as follows. Section 2 introduces the extended model of Natural Logic, mainly describing the aspects that we borrow from this method

270

E. Cabrio and B. Magnini

to enrich our framework, such as the basic entailment relations and the strategy for their composition. In Section 3, the transformation-based framework for the specialized entailment engines is deﬁned. Section 4 describes the role of entailment rules and provides more details on our deﬁnition of atomic edits. Section 5 shows how relation combination mechanisms are adopted in our framework, and an example of application on RTE data is provided. Section 6 concludes the paper drawing ﬁnal remarks and proposing some future work.

2

Extended Model of Natural Logic

Natural Logic (NL) provides a conceptual and formal framework for analyzing natural inferential systems in human reasoning, without full semantic interpretation. Originating in Aristotle’s syllogysms, it has been revived in the ’80s in works of van Benthem [21], S´ anchez Valencia [19], and Nairn et al. [17]. In this Section we would like to introduce the concepts of the NL framework that we used to give shape to our model of specialized entailment engines, to account for natural language inference problems. In particular, in [16] Mac Cartney and Manning propose a natural language inference model based on natural logic, which extends the monotonicity calculus to incorporate semantic exclusion, and partly uniﬁes it with Nairn et al.’s account of implicatives. First, the authors deﬁne an inventory of basic semantic relations (set B) including representations of both containment and exclusion, by analogy with set relations2 (shown in Table 1). Such relations are deﬁned for expressions of every semantic type: sentences, common and proper nouns, transitive and intransitive verbs, adjectives, and so on. This aspect is relevant for our goals, since we would like to handle variability in natural language inference at diﬀerent linguistic levels. Table 1. Set B of basic semantic relations, as deﬁned in [16] symbol x≡y xy xy xˆy x|y xy x#y

name equivalence forward entailment reverse entailment negation alternation cover independence

example couch ≡ sofa crow bird European French humanˆnonhuman cat | dog animal ≡ nonhuman hungry # hyppo

set theoretic deﬁnition x=y x⊂y x⊃y x∩y = 0∧x∪y = U x ∩ y = 0 ∧ x ∪ y = U x ∩ y = 0 ∧ x ∪ y = U (all other cases)

In B, the semantic containment relations ( and ) of the monotonicity calculus are preserved, but are decomposed into three mutually exclusive relations: equivalence (≡), (strict) forward entailment (), and (strict) reverse entailment (). Two relations express semantic exclusion: negation(ˆ), or exhaustive exclusion (analogous to set complement), and alternation (|) or non-exhaustive exclusion. Another 2

In a practical model of informal natural language inference, they assume the nonvacuity of the expressions.

Deﬁning Specialized Entailment Engines Using Natural Logic Relations

271

relation is cover (), or non-exclusive exhaustion; ﬁnally the independence relation (#) covers all other cases (non-equivalence, non-containment, non-exclusion, and non-exhaustion). The relations in B are mutually exclusive, and it is possible to deﬁne a function β(x, y) that maps every ordered pairs of non vacuous expressions to the unique relation in B to which it belongs. Furthermore, a model to join () semantic relations is provided, as shown in Table 2. It could happen that the result of joining two relations is not a relation in B, but the union of such relations (speciﬁcally {≡, , , |, #}), meaning that the relation is not determined (refer to [16] for further details, and for explanations on the theoretical foundation of the model). The total relation, notated as •, is the relation that contains all pairs of (non-vacuous) expressions and conveys zero information about them. Table 2. Join table for relations in B, as deﬁned in [16] ≡ ˆ | #

≡ ≡ ˆ | #

≡| # ≡| # | ˆ| # | ˆ| # # | #

ˆ ˆ | ≡ #

| | | ˆ| # ˆ| # | ≡| # ≡| # | # #

# # | # # # | # # •

After providing the basic deﬁnitions of the building blocks of their model of natural language inference, the authors of [16] describe a general method for establishing the semantic relations between a premise p and an hypothesis h. The steps are as follows: 1. Find a sequence of atomic edits (i.e. deletion, insertion, or substitution of a subexpression) < e1 , ..., en > which transforms p into h 2. For each atomic edit ei : (a) Determine the lexical semantic relation β(ei ); (b) Since β(ei ) depends on properties of the context of the expression in which e is applied, compute the projection of β(ei ) upward through the semantic composition tree of the expression, while respecting the monotonicity properties of each node along the path (more details on how it is performed are described in [16]); 3. Join atomic semantic relations across the sequence of edits. This model has been implemented in software as the NatLog system, and has been evaluated on both i) on the FraCaS test suite [5], which contains 346 NLI problems, divided into nine sections, each focused on a speciﬁc category of semantic phenomena, and ii) on the RTE-3 test suite [9]. Better results ([14] [15]) are obtained on the ﬁrst test suite with respect to RTE data, in which a diversity of types of inference (e.g. paraphrase, temporal reasoning, relation extraction)

272

E. Cabrio and B. Magnini

that NatLog is not designed to address is present. In [4] strategies of hybridizing the model with broad-coverage RTE systems have been experimented. In our framework we take advantage of this model, adopting both the set of semantic relations and the mechanisms for their combination. A step further, we provide an operational deﬁnition of atomic edits, in terms of application of entailment rules expressing the knowledge of a certain linguistic phenomenon (Section 4).

3

Transformation-Based TE and Specialized Entailment Engines

We assume a transformation-based framework for Textual Entailment, meaning that in order to assign the correct entailment relation to a given pair, the text T has to be transformed into H by means of a set of edit operations (e.g. insertion, deletion, substitutions). Assuming Fregean meaning compositionality, the hypothesis of this work is that the correct entailment relation (ER) can be assigned to a T, H pair combining the entailment relations (equivalent to the semantic relations described in Section 2) separately assigned to the diﬀerent linguistic phenomena involved in the entailment relation between T and H. In other words, given ERi (T, H), the relation assigned to T and H for a certain linguistic aspect i, we assume that: n [ERi (T, H)] ER(T, H) = COM Bi=1

(1)

where i potentially ranges over all the linguistic phenomena involved in textual entailment, and COM B is the composition function. According to our initial assumptions, and in line with the NL approach deﬁned in Section 2, we expect the possibility to assign to each linguistic phenomenon relevant to inference in a pair one of these relations. The correct combination of all the relations would result in the correct assignment of the entailment relation to the pair. In our transformation-based framework, the assignment of such relations to the linguistic phenomena relevant to inference in a given pair is the result of the application of edit operations to the portions of T and H expressing that phenomenon. Each single transformation (i.e. atomic edit ) can have a diﬀerent granularity, according to the linguistic category of the phenomenon that is considered. For instance, transformations relative to lexical phenomena would probably involve single words, while syntactic transformations would most likely involve syntactic structures. Basing on the assumptions introduced before, a specialized entailment engine for a certain linguistic phenomenon i outputs a judgment (i.e. entailment relation ERi , deﬁned as in the NL model) between T and H, such that: ERi (T, H) = #

if i does not aﬀect T and H (either i is not present in the pair or it is not relevant to inference)

Deﬁning Specialized Entailment Engines Using Natural Logic Relations

273

⎧ ≡, if i contributes to the entailment in T and H ⎪ ⎪ ⎨ ˆ, | if i contributes to the contradiction in T and H ERi (T, H) = , if the truth of H wrt T remains unknown on the ⎪ ⎪ ⎩ basis of i As an example, let’s suppose a specialized entailment engine which only detects entailment due to the active-passive alternation between T and H, and suppose the following T-H pairs: T1 H1 H2 H3 H4

John paints the wall. The wall is white. The wall is painted by John. The wall is painted by Bob. The wall is coloured by John.

When the engine EEa−p is applied to the examples, according to our deﬁnition we will obtain the following results (ERa−p is the relation assigned with respect to the phenomenon of active-passive alternation): ERa−p (T 1, H1) = (T 1#H1) because there is no active-passive alternation in the pair; ERa−p (T 1, H2) = (T 1 ≡ H2) because the application of an active-passive rule allows to preserve the entailment between T1 and H2; ERa−p (T 1, H3) = (T 1 | H3) because, although an active-passive alternation is present in the pair, the corresponding rule can not be applied, this way contributing to the contradiction in the pair. More generally, we distinguish four cases in the behavior of a specialized entailment engine EEi . The neutral case, when the linguistic phenomenon i does not occur in a certain pair. We say that the TE engine EEi is “neutral” with respect to i, when it can not produce any evidence either for the entailment or the contradiction between T and H. With respect to phenomenon i, a relation of independence exists between T and H (T #H). The positive case, when the phenomenon i occurs and contributes to establish an entailment relation between T and H. Both the relations of equivalence (T ≡ H) and forward entailment (T H) fall within this case. We consider equality, i.e. when T and H are made of the same sequence of tokens, as a special case of the positive situation. The negative case, when the phenomenon i occurs and contributes to establish a contradiction relation between T and H. Negative cases may correspond to two

274

E. Cabrio and B. Magnini

situations: (i) explicit knowledge about contradiction (e.g. antonyms, negation) or (ii) a mismatch situation, where it is not possible to apply an entailment rule, and as a consequence, a certain degree of contradiction emerges from the T-H pair (see the T1-H3 pair on active-passive alternation). Both the relations of negation (TˆH) and alternation (T | H) fall within this case. The unknown case, when the phenomenon i occurs but is it not possible to prove the truth of H wrt T on the basis of i, as for hyponymy/hyperonymy (e.g. T: John is a football player ; H2: John is a goalkeeper ). Both the relations of cover (T H) and reverse entailment (T H) fall within this case. In our model, the last three cases are deﬁned in the same way as the judgments allowed in the TE task, while the neutral case is a speciﬁc possible behavior of the specialized entailment engines framework. In this framework, a specialized engine should ﬁrst recognize the phenomenon i it is built to cope with, and only if i is detected in the pair the engine will output one of the three possible judgments. It must be anticipated here that the specialized engines’ absence of judgment (i.e. neutral case for all the engines of a set) has to be interpreted as the absence of common phenomena between T and H, resulting in the assignment of the unknown judgment for that pair. Even if the neutral and the unknown case could result in the assignment of the same entailment relation, from our viewpoint the engines’ behavior is qualitatively diﬀerent. Specialized engines must be disjoint one from the other, meaning that they should not cover the same linguistic phenomenon. In addition to the relation between T and H according to the phenomenon i, a specialized engine returns the set of atomic edit operations between T and H allowed by the application of speciﬁc knowledge of phenomenon i. In the next Section we provide an operational deﬁnition of atomic edits, basing on the notion of entailment rules.

4

Entailment Rules and Atomic Edits

In our transformation-based framework, atomic edits are applied to subportions of T and H expressing a certain linguistic phenomenon, and their granularity is deﬁned by the linguistic phenomenon they describe. More speciﬁcally, we deﬁne the allowed transformations (i.e. atomic edits) for a certain linguistic phenomenon through a set of entailment rules for that speciﬁc phenomenon. An entailment rule is deﬁned as either a directional or bidirectional relation between two sides of a pattern, corresponding to text fragments with variables (typically phrases or parse sub-trees, according to the granularity of the phenomenon they formalize) [20]. The left-hand side of the pattern (LHS) entails the rights-hand side (RHS) of the same pattern under the same variable instantiation. In addition, a rule may be deﬁned by a set of constraints, representing variable typing (e.g. PoS, Named Entity type) and relations between variables, which have to be satisﬁed for the rule to be correctly applied [2]. Supposing to have a repository of all the entailment rules expressing the knowledge about the linguistic phenomena relevant to inference, we could

Deﬁning Specialized Entailment Engines Using Natural Logic Relations

275

associate an entailment relation both to the correct and to the incorrect application of the rule. For instance, the correct instantiation of the entailment rule for active/passive alternation expressed as: Pattern:

X V1 Y ⇔ Y V2 by X

Constraint:

SAME LEMMA(V1,V2) TYPE(V1)= ACTIVE FORM ; TYPE(V2)=PASSIVE FORM

and instantiated as e.g. T: John paints the wall ⇔ H1: The wall is painted by John maintains the equivalence relation between T and H (T ≡ H), and the pair (T,H1) should be marked as entailment. The wrong instantiation of the same rule as in H2: The wall is painted by Bob produces an alternation relation (T | H), and the contradiction judgment should be assigned to the pair (T, H2). Following the same criteria, for hyponymy/hyperonymy the entailment rule is expressed as: Pattern: X ⇒ Y Constraint: HYPONYMY(X,Y) and instantiated as e.g. T: John is a football player ⇒ H1: John is an athlete. According to this phenomenon, a forward entailment relation exists between T and H (T H) and the pair (T,H1) should be marked as entailment. Instead, the invertion of the directional entailment rule as in H2: John is a goalkeeper produces a reverse entailment relation between T and H (T H), and the pair (T,H2) should be marked as unknown.

5

Combination Based on Natural Logics

In Sections 3 and 4 we have provided the basic deﬁnitions of the elements of our framework, introducing the specialized entailment engines and describing their possible behaviours in terms of entailment relations to be assigned to the linguistic phenomena relevant to inference in a given pair. The inference problem is therefore decomposed into a sequence of atomic inference problems, separately solved by a set of disjoint precision-oriented engines, each of which outputs i) the entailment relation corresponding to the processed linguistic phenomenon, and ii) the set of transformations between T and H allowed by the application of entailment rules for that speciﬁc phenomenon. In this Section we want to go a step further, taking advantage of the mechanisms of relation composition provided by the extended model of Natural Logic to combine the outputs of the entailment engines, in order to obtain a global judgment for a pair. Table 2 in Section 2 shows the relations resulting from joining the atomic semantic relations across the sequences of edits. Exploting the same strategy, we compose step by step the entailment relations generated by each engine to determine the global entailment relation for a pair.

276

E. Cabrio and B. Magnini

Relation composition is deterministic, and in general it follows intuitive rules (e.g. ≡ composed with ≡ yields ≡, composed with yields ). At each step, the result may be either a basic entailment relation, or the union of such relations, with larger unions conveying less information about entailment (i.e. every union relation which result from joining relations in B contains #, and thus can be approximated by #). As a drawback, it must be noticed that composition tends to degenerate towards # both because composing # with any relation yields that relation, and because composing a chain of randomly-selected relations tends toward # as the chain grows longer. In our framework, such relation is assigned if the engine is neutral with respect to a certain pair, meaning that the phenomenon is it built to deal with is not present. In this case, such relation is not counted in the composition phase3 . 5.1

Order of Composition

The fact that the engines are disjoint does not guarantee that they are independent, which means that the order of their application does aﬀect the ﬁnal result. For instance, considering the pair T: John paints the wall - H: The wall is coloured by John, it seems diﬃcult to apply the active-passive transformation before the lexical transformation between “paint” and “colour” has been applied. We therefore assume a cascade of disjoint entailment engines, where each engine takes as input the output of the previous engine, deﬁned as the set of edit transformations from T to H related to phenomenon i. The order in which the engines are run does not correspond to sentence order, but is deﬁned through linguistically-motivated heuristics; this ordering deﬁnes a path from T to H through intermediate forms. As a ﬁrst approximation, we ﬁrst run the engines whose transformations apply to single tokens, like lexical phenomena (e.g. synonymy, hypernymy), then the engines involving structures, like syntactic phenomena (e.g. active/passive alternation, argument realization) and discourse phenomena (e.g. coreference), and ﬁnally reasoning (e.g. spatial, temporal reasoning, common background knowledge). With respect to ﬁnal entailment judgment, if the combination of the relations separately assigned to the diﬀerent linguistic phenomena present in T and H is either ≡ or , the entailment judgment is assigned to the T-H pair; if it is eitherˆor | the contradiction judgment is assigned; if it is either , , or # the unknown judgment is assigned. 5.2

Example of Application of the Proposed Framework to RTE Pairs

In the framework we presented, we suppose to have a set of specialized engines covering the most frequent phenomena relevant to inference, and behaving as discussed in Section 3. As an example, we run them in the order hypothesized 3

If all the engines output #, it means that no phenomena are in common between T and H, i.e. the relation is unknown.

Deﬁning Specialized Entailment Engines Using Natural Logic Relations

277

before on a contradiction pair extracted from RTE-5 test set (pair 125) [1]. Table 3 shows the application of the whole method on this pair. Three engines are activated on this pair, namely i) EElex:semantic opposition , ii) EEsynt:argument realization , and iii) EEsynt:apposition . After performing the atomic edits on the phenomenon detected (applying the corresponding entailment rule), each engine outputs both the entailment relation assigned to that operation (ERi ), and an intermediate form of H expressing the instantiation of the rule in that speciﬁc pair (Hi ). The entailment relation produced by each engine is then combined with the one assigned by the previous engine (ERCOMB ), following the combination scheme described in Table 2. The ﬁnal entailment relation is |, and the contradiction judgment is assigned to the pair. The engines of the set that do not detect the phenomenon they are built to deal with assign the relation #, meaning that they are neutral with respect to this pair, and their judgment is not taken into account in the combination phase. Table 3. Application of the decomposition methodology to an original RTE pair Text snippet (pair 125) T

Mexico’s new president, Felipe Calderon seems to be doing all the right things in cracking down on Mexico’s drug traﬃckers. [...]

EE

H1 Mexico’s outgoing president, Felipe xy Calderon, [...] antonym(x, y) H2 The outgoing president of Mexico, Felipe x’s y ⇔ y of x Calderon, [...] H3 Felipe Calderon is the outgoing president x, y ⇔ y is x of Mexico. apposition(y, x)

lex:sem opp

EE

synt:argr ealiz

EE

synt:apposition

H

6

entailm. rule

Felipe Calderon is the outgoing President of Mexico.

ERi

ER COM B

|

|

≡

|

≡

| |

Conclusions

In this paper we have presented a framework for the deﬁnition of specialized entailment engines, each of which able to deal with a certain aspect of language variability. We took advantage of the conceptual and formal tools available from an extended model of Natural Logic (NL) [16], to deﬁne clear strategies for their combination. In a transformation-based framework, each engine performs atomic edits to process a certain linguistic phenomenon, and assigns an entailment relations as the output of this operation. NL mechanisms of semantic relations composition are then applied to join the output of each single engine, in order to obtain a global entailment judgment for a pair. The framework described aims at providing a model to decompose the complexity of the Textual Entailment problem, assuming Fregean meaning compositionality principle. With respect to the model described in [16] in which a lot of eﬀort is made to establish the proper projectivity signatures for a broad range of quantiﬁers, implicative and factives, and other semantic relation, our work is less ﬁne-grained, since it relies on the expressivity of the entailment rules to

278

E. Cabrio and B. Magnini

model a certain linguistic phenomenon. On the other hand, as far as a linguistic phenomenon can be expressed through entailment rules it can be modeled in our framework, guaranteeing a broader coverage on RTE problems. Further work has to be done in the implementation of such model, adopting a modular architecture that accounts for the properties of the engines described before. Each engine must be carefully shaped to reward its precision, since the errors made in the initial steps will propagate, via entailment composition, to the system’s ﬁnal output. Pilot experiments have been carried out, demonstrating the value of the model on real RTE data.

References 1. Bentivogli, L., Magnini, B., Dagan, I., Dang, H.T., Giampiccolo, D.: The Fifth PASCAL Recognizing Textual Entailment Challenge. In: Proceedings of the TAC 2009 Workshop on Textual Entailment, Gaithersburg, Maryland, November 17 (2009) 2. Bentivogli, L., Cabrio, E., Dagan, I., Giampiccolo, D., Lo Leggio, M., Magnini, B.: Building Textual Entailment Specialized Data Sets: a Methodology for Isolating Linguistic Phenomena Relevant to Inference. In: Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC), Valletta, Malta, May 19-21 (2010) 3. Cabrio, E., Magnini, B.: Toward Qualitative Evaluation of Textual Entailment Systems. In: Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010: Posters), Beijing, China, August 23-27 (2010) 4. Chambers, N., Cer, D., Grenager, T., Hall, D., Kiddon, C., MacCartney, B., Ramage, D., Yeh, E., Manning, C.D.: Learning Alignments and Leveraging Natural Logic. In: Proceedings of the ACL 2007 Workshop on Textual Entailment and Paraphrasing, Prague, Czech Republic, June 28-29 (2007) 5. Cooper, R., et al.: Using the framework. Technical Report LRE 62-051 D-16, The FraCaS Consortium (1996), http://www.cogsci.ed.ac.uk/~fracas/ 6. Dagan, I., Glickman, O.: Probabilistic Textual Entailment: Generic Applied Modeling of Language Variability. In: Proceedings of the PASCAL Workshop on Learning Methods for Text Understanding and Mining, Grenoble, France, January 26-29 (2004) 7. Dagan, I., Dolan, B., Magnini, B., Roth, D.: Recognizing textual entailment: Rational, evaluation and approaches. In: Natural Language Engineering (JNLE), October 2009, vol. 15, Special Issue 04, pp. i-xvii. Cambridge University Press, Cambridge (2009) ¨ 8. Frege, G.: Uber Sinn und Bedeutung. Zeitschrift f¨ ur Philosophie und philosophische Kritik 100, 25–50 (1892) 9. Giampiccolo, D., Magnini, B., Dagan, I., Dolan, B.: The Third PASCAL Recognising Textual Entailment Challenge. In: Proceedings of the ACL 2007 Workshop on Textual Entailment and Paraphrasing. Prague, Czech Republic, June 28-29 (2007) 10. Giampiccolo, D., Trang Dang, H., Magnini, B., Dagan, I., Cabrio, E.: The Fourth PASCAL Recognising Textual Entailment Challenge. In: Proceedings of the TAC 2008 Workshop on Textual Entailment. Gaithersburg, Maryland, USA, November 17 (2008) 11. Lakoﬀ, G.: Linguistics and natural logic. Synthese 22, 151–271 (1970)

Deﬁning Specialized Entailment Engines Using Natural Logic Relations

279

12. Kouylekov, M., Magnini, B.: Tree Edit Distance for Textual Entailment. In: Proceedings of the Recent Advances in Natural Language Processing Conference (RALNP 2005), Borovets, Bulgaria, September 21-23 (2005) 13. Magnini, B., Cabrio, E.: Combining Specialized Entailment Engines. In: Proceedings of the 4th Language & Technology Conference (LTC 2009), Poznan, Poland, November 6-8 (2009) 14. MacCartney, B., Manning, C.D.: Natural Logic for Textual Inference. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007), Prague, Czech Republic, June 23-30 (2007) 15. MacCartney, B., Manning, C.D.: Modeling semantic containment and exclusion in natural language inference. In: Proceedings of the 22nd International Conference on Computational Linguistics (COLING 2008), Manchester, UK, August 18-22 (2008) 16. MacCartney, B., Manning, C.D.: An extended model of natural logic. In: Proceedings of the Eighth International Conference on Computational Semantics (IWCS-8), Tilburg, The Netherlands, January 7-9 (2009) 17. Nairn, R., Condoravdi, C., Karttunen, L.: Computing relative polarity for textual inference. In: Inference in Computational Semantics (ICoS-5), Buxton, UK, April 20-21 (2006) 18. Sammons, M., Vydiswaran, V.G.V., Roth, D.: Ask Not What Textual Entailment Can Do for You. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010), Uppsala, Sweden, July 11-16 (2010) 19. S´ anchez Valencia, V.: Studies on Natural Logic and Categorial Grammar. PhD Thesis, University of Amsterdam (1991) 20. Szpektor, I., Shnarch, E., Dagan, I.: Instance-based evaluation of entailment rule acquisition. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007), Prague, Czech Republic, June 23-30 (2007) 21. Van Benthem, J.: The semantics of variety in categorial grammar. In: Buszkowski, W., Marciszewski, W., Van Benthem, J. (eds.) Categorial Grammar, pp. 33–35. John Benjamins, Amsterdam (1988) 22. Van Benthem, J.: A brief history of natural logic. Technical Report PP-2008-05, Institute for Logic, Language & Computation (2008), http://www.illc.uva.nl/Publications/ResearchReports/PP-2008-05.text. pdf

Czech Senior COMPANION: Wizard of Oz Data Collection and Expressive Speech Corpus Recording and Annotation Martin Gr˚ uber, Milan Leg´at, Pavel Ircing, Jan Romportl, and Josef Psutka Department of Cybernetics, Faculty of Applied Sciences, University of West Bohemia, Czech Republic {gruber,legat,ircing,rompi,psutka}@kky.zcu.cz http://kky.zcu.cz

Abstract. This paper presents part of the data collection eﬀorts undergone within the project COMPANIONS whose aim is to develop a set of dialogue systems that will be able to act as an artiﬁcial “companions” for human users. One of these systems, being developed in Czech language, is designed to be a partner of elderly people which will be able to talk with them about the photographs that capture mostly their family memories. The paper describes in detail the collection of natural dialogues using the Wizard of Oz scenario and also the re-use of the collected data for the creation of the expressive speech corpus that is planned for the development of the limited-domain Czech expressive TTS system. Keywords: data collection, corpus recording, expressive speech synthesis, dialogue system.

1

Introduction

The research area of the automatic dialogue systems is recently receiving a considerable surge of attention from the scientiﬁc teams dealing with speech technologies and natural language processing. It is largely due to the fact that automatic speech recognition (ASR) and speech synthesis (TTS) systems have made considerable progress in recent years which allowed their utilization in various areas. However, one should bear in mind that those two components still constitute only a “front-end” and “back-end” of a system that would be able to engage in a natural dialogue with a human user. What still needs a lot of research eﬀort are the “central” modules dealing with natural language understanding (NLU), information extraction (IE) and dialogue management (DM). Since human dialogues are very complex and require both speciﬁc and background knowledge and reasoning capabilities of all participants, the development of a general-purpose, unrestricted computer dialogue system is currently unfeasible. Thus, when designing a dialogue system, we ﬁrst need to restrict its domain to make the problem solvable. Ideally, the computer should be able to act in the same way human would at least in a given domain. For example, rather simple Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 280–290, 2011. c Springer-Verlag Berlin Heidelberg 2011

Data Collection and Expressive Corpus Recording and Annotation

281

dialogue systems are nowadays often encountered when calling to a centre providing information about train schedules or services oﬀered by a telecommunication company, etc. More advanced dialogue systems operating in the restaurant domain were presented in [1], [2]. In the research being done within the COMPANIONS project [3] (www. companions-project.org), it was decided to develop a computer system that would be able to conduct a natural dialogue with elderly users, mostly to keep the company and letting them to stay mentally active. As this restriction is still not suﬃcient enough, it was decided to narrow the task further to the reminiscing about family photographs. The system was named “Senior Companion” and was originally planned to be developed in two languages - Czech and English. No dialogue system can be designed without prior knowledge about the speciﬁcs of the conversations that such system is going to deal with. Therefore at least a small sample of representative dialogues needs to be gathered, even when the developers plan to use rule-based techniques in the NLU, IE and DM modules. When (as is the case of the COMPANIONS project) there is an intention to employ machine-learning algorithms in all those modules, the amount of representative data that are necessary to gather is even more crucial. Therefore this paper deals mostly with the data gathering eﬀorts undergone in the preparation of the development of the Czech Senior Companion. The paper is organized as follows - Chapter 2 describes the basic premises of the data collection method and technical measures taken to ensure representative and high-quality corpus. Chapter 3 contains a brief description of the gathered corpus of natural dialogues, both quantitative and qualitative. Chapter 4 explains how the data from the dialogue corpus were re-used for the recording of the expressive speech corpus that will be used for the development of the new limited-domain TTS system and Chapter 5 presents the process of the TTS corpus annotation with communicative functions.

2

Data Collection Process

We have decided to employ the Wizard of Oz (WoZ) approach [1] in order to gather a corpus of human-computer dialogues. It means that human subjects were placed in front of the computer screen and were told that the program they are interacting with is fully autonomous, i.e. using “artiﬁcial intelligence” techniques to conduct a natural dialogue. In reality, the automatic speech recognition, understanding and response generation was simulated by a human operator (the “wizard”). Only the speech produced by a computer was genuinely generated using a TTS system coupled with 3D avatar (“talking head”) [4], which further reinforced the subjects’ belief that they are truly interacting with a computer only. The wizard acted as a dialogue partner the role of which was to stimulate the conversation and to give the user the feeling of being listened to by someone. This task was managed by using the set of typical questions, backchannel

282

M. Gr˚ uber et al.

utterances and also pre-recorded non-speech dialogue acts expressing comprehension, amusement, hesitation, etc. To keep the dialogue smooth and natural, the crucial thing was to have pre-prepared sentences and questions that will be used. These sentences were saved in a so-called scenario. However, sometimes the dialogue does not follow the prepared scenario exactly, so the task of the wizards was to type the appropriate sentences on-line. This could have caused unnatural pauses in some cases but in general this problem was not so serious. The recording of natural dialogues consists of separate sessions. In each session, one elder person (subject) was left alone in a recording room where necessary recording equipment was placed in. The setup of the recording room is depicted in Figure 1.

Fig. 1. Recording room setup

In the recording room, the subject faces an LCD screen and two speakers, the speech is recorded by two wireless microphones, and the video is captured by three miniDV cameras. There is also one surveillance web-camera, just to monitor the situation in the recording room. The only contact between a user and the computer was through speech, there was no keyboard nor mouse on the table. A snapshot of the screen presented to human subjects is shown in Figure 2. On the left upper part of the LCD screen, there is visualized 3D model of a talking head. This model is used as the avatar, the impersonate companion that should play a role of the partner in the dialogue. Additionally, on the right upper part, there is shown a photograph which is currently being discussed. On the lower half of the screen, there is a place used for displaying subtitles (just in case the synthesized speech is not intelligible suﬃciently). The subtitles were used only during the ﬁrst few dialogues. Later, the subtitles were not displayed because the generated speech was understandable enough and subjects did not have any problems to comprehend it.

Data Collection and Expressive Corpus Recording and Annotation

283

Fig. 2. Snapshot of the WoZ system interface - user side

The speech was captured by two wireless microphones. One microphone was used for the speech of the subject, the second one recorded the speech of the avatar. For high quality recording, an external preampliﬁer and an external Creative Sound Blaster Extigy sound card were used. Almost all audio recordings are stored using 22kHz sample rate and 16-bit resolution. The ﬁrst six dialogues were recorded using 48kHz sample rate, later it was reduced to the current level according to requirements of the ASR team. The video of the session was also recorded, using three miniDV cameras. The subjects were recorded from the front, side and back view to provide data usable in various ways. The front view could be later used for the audio-visual speech recognition (where such viewing angle will be beneﬁcial for lip-tracking) and also for the emotion detection algorithms. Along with the side view, it can be also used for 3D head modelling. Since in the side view there was captured not only face but also the whole upper part of a body, it can be used for hands gesture and body movement tracking. The back view shows what was displayed on the LCD screen and in some cases what the speaker point at on the photograph. This information can be useful for example for tagging people on the picture when they are pointed at by the user while talking about them. This could be helpful for computer vision while seeking for the subjects pictured on the photo.

3

Dialogue Corpus Characteristics

First some statistics - the current data set contains 65 dialogues. Based on gender, the set of speakers can be divided into 37 females and 28 males. Mean age of the speakers is 69.3 years; this number is almost the same for both male and female speakers. The oldest person was a female, 86 years old. The youngest one was also a female, 54 years old. All the recorded subjects were native Czech speakers; two of them spoke a regional Moravian dialect. This dialect diﬀers

284

M. Gr˚ uber et al.

from regular Czech language in pronunciation and, also, a little in vocabulary. Duration of each dialogue was limited to 1 hour, as this was the capacity of tapes used in miniDV cameras, resulting in average duration 56 minutes per dialogue. During the conversation, 8 photographs were discussed in average (maximum was 12, minimum 3). Each participant of the experiment was asked to ﬁll out a short questionnaire right after the recording session, asking question both about their computer literacy and the rating of the just ﬁnished dialogue with the computer. Approximately one half of the subjects stated in this form that they have a computer at home. Nevertheless, most of them do not use it very often. Almost all the dialogues were rated as friendly and smooth. And even more, the users were really enjoying reminiscing about their photos, no matter that the partner in the dialog was an avatar. This was conﬁrmed not only by the questionnaires by also by wizards’ observations during the sessions. A rather representative excerpt from one of the dialogues is presented in Table 1. Table 1. Excerpt from a WoZ dialogue AVATAR: What about this photo? USER: Well, this is my son with his dog, his name is Cindy. AVATAR: What is your son’s name? USER: And the son’s name is Roman. AVATAR: How old is your son? USER: He is 28 years old. AVATAR: Do you have other children? USER: Yes, I have one more son. AVATAR: What is your son doing? USER: This son, Roman, has his own company and the other is named Jiˇr´ı and he works in a warehouse in Plzeˇ n AVATAR: Tell me something about them. USER: Well, both of them are rather good sons, one of them is single, the other is divorced because just his wife with that Lucinka left for Norway and he stayed alone whereas the other son Roman has a girlfriend that he is only probably going to marry.

To summarize, we have gathered more than 60 hours of speech data and, most importantly, we feel that we have a rather good knowledge about the way in which the conversation about the photographs usually develops and what kinds of “system” responses were the most appropriate for keeping the conversation rolling. Last, but not the least importantly, we have found out that the avatar operated by wizards, although equipped with a neutral voice only and a very limited set of facial expressions, is able to elicit quite a strong emotional

Data Collection and Expressive Corpus Recording and Annotation

285

response from the users. This is an important ﬁnding since the idea of an artiﬁcial companion being able to both detect and generate aﬀective response is one of the hallmarks of the COMPANIONS project. The resulting dialogue corpus can be of course also readily used for various machine-learning procedures, designed mainly to tailor the ASR system to the speciﬁc domain, such as re-training of the language models. Since we have quite a large amount of speech data for each individual user, we can also extensively test new speaker adaptation methods [5]. Moreover, we have devised a way how the recorded data can be used to design and record a speech corpus for limited-domain expressive speech synthesis. The principle of this method is described in the following two chapters.

4

Design and Recording of the Expressive Speech Corpus

Development of the aﬀective TTS system is a challenge that has still not been satisfactorily resolved. The main problem is that even just the classiﬁcation of the aﬀective (non-neutral) speech utterances is diﬃcult. Many methods of emotional (aﬀective) state classiﬁcation have been proposed. Very brieﬂy and in simplicity - the basic distinction is whether a particular classiﬁcation system is categorical, or dimensional. Among many we can name a categorical classiﬁcation system [6] which distinguishes emotional states such as anger, excitement, disgust, fear, relief, sadness, satisfaction, etc. In a dimensional model, emotions are deﬁned as positions (or coordinates) in a multidimensional space where each dimension stands for one property of an emotional state. Various dimensions have been proposed out of which a widely accepted set is the one presented in [7] with two axes: valence (positive vs. negative) and arousal (high vs. low activation). Other models also consider a third dimension that is power or dominance and some even a fourth dimension: unpredictability. However, as was mentioned above, it is quite diﬃcult to classify human speech according to either one of these models with the perspective of ﬁnding out acoustic correlates useful for generation purposes. Therefore instead of labeling the emotions in the utterances (aﬀective states) explicitly, we have settled for the assumption that a relevant aﬀective state (of the conversational agent) goes implicitly together with a communicative function (CF) of a speech act (or utterance) which is more controllable than the aﬀective state itself. It means that we do not need to think of modelling an emotion such as ”guilt” per se - we expect it to be implicitly in an utterance like ”I am so sorry about that” with a communicative function ”apology”. Thus we have decided to proceed with the aﬀective TTS corpus creation as follows. First, we hired a professional female speaker (stage-player) and instructed here not to express a speciﬁc emotions but just to put herself in the place of a Senior Companion. In order to facilitate such an empathy, a special software application was developed - it played back the parts of the WoZ dialogues where the subject was speaking (to provide the speaker with the relevant context) and at the time where the avatar have originally spoken, the dialogue was paused

286

M. Gr˚ uber et al.

and the speaker was prompted to record the avatar’s sentence herself. The text of the actual sentence was displayed on the screen even when the real (context) dialogue was being played so that the speaker had enough time to get acquainted with it before the recording. The recording equipment was again carefully selected and set-up in order to ensure the highest possible technical quality of the corpus - the speaker was placed in the anechoic room and the recording was done using a professional mixing desk. The glottal signal was captured along with the speech. That way we have recorded approximately 7,000 of (mostly short) sentences. Those were carefully transcribed and annotated by communicative functions (CF) described bellow.

5

Annotation Using Communicative Functions

The set of CFs was partial inspired by [8] and is listed in Table 2. The expressive speech corpus was annotated using communicative functions by means of a listening test. An additional label for “other” communicative function was introduced for the test purposes only — this label is not listed in the table. The test was aimed to determine objective annotation on the basis of several subjective annotations as the perception of expressivity is always subjective and may vary depending on particular listener. A special web application working on the client-server basis was developed for the listening test purposes. This way listeners were able to work on the test Table 2. Set of communicative functions comunicative function example directive Tell me that. Talk. request Let’s get back to that later. wait Wait a minute. Just a moment. apology I’m sorry. Excuse me. greeting Hello. Good morning. goodbye Goodbye. See you later. thanks Thank you. Thanks. surprise Do you really have 10 siblings? sad empathy I’m sorry to hear that. It’s really terrible. happy empathy It’s nice. Great. It had to be wonderful. showing interest Can you tell me more about it? conﬁrmation Yes. Yeah. I see. Well. Hmm. disconﬁrmation No. I don’t understand. encouragement Well. For example? And what about you? not speciﬁed Do you hear me well? My name is Paul.

Data Collection and Expressive Corpus Recording and Annotation

287

from their homes without any contact with the test organizers. Various measures were undertaken to detect possible cheating, carelessness or misunderstandings. The test participants have been instructed to listen to the recordings very carefully and subsequently mark communicative function(s) that are expressed within the given sentence. The number of CFs that should be assigned to a single sentence was not prescribed, this decision was left to listeners’ discretion. In order to facilitate their job, listeners also had a few sample sentences labelled with communicative functions available during the whole course of the test. That way we obtained subjective annotations that of course somehow vary across the listeners. A proper combination of those subjective annotations was needed in order to objectively annotate the expressive recordings. Therefore an evaluation of the listening test was made. We utilized two approaches to the inference of the objective annotation: – The ﬁrst way is a simple majority method. Using this easy and intuitive approach, each sentence is assigned a communicative function that was selected by the majority of the listeners. If this majority accounts for less then 50% of all listeners, the classiﬁcation of the sentence is considered to be unreliable. – The second approach is based on maximum likelihood method. Maximum likelihood estimation is a statistical method used for ﬁtting a statistical model to data and providing estimates for the model’s parameters. The maximum likelihood estimator is consistent. It means that having a suﬃciently large number of observations (annotations in our case), it is possible to ﬁnd the value of statistical model parameters with arbitrary precision. The parameter calculation is implemented using the EM algorithm [9]. Knowing the model parameters we are able to infer the objective annotation. Precision of the estimate is one of the outputs of this model. A sentence labelled with a communicative function with low precision can be eliminated from the expressive corpus. Comparing these two approaches, 35 out of 7287 classiﬁcations were marked as untrustworthy using maximum likelihood method and 571 using simple majority method. The average ratio of listeners who marked the same communicative function for particular sentence using simple majority approach was 81%, when untrustworthy classiﬁcations were excluded. Similar measure for maximum likelihood approach cannot be easily computed as the model parameters and the estimate precision depend on number of iteration in the EM algorithm. Finally, we decided to use the objective annotation obtained by maximum likelihood method. We have also successfully used this approach in recent works regarding speech synthesis research, see [10]. Further, we need to conﬁrm that the listeners marked the sentences with communicative functions consistently and achieved some measure of agreement. Otherwise the subjective annotations could be considered as accidental or the communicative functions inappropriately deﬁned and thus the acquired objective annotation would be false. For this purpose, we make use of two statistical measures for assessing the reliability of agreement among listeners.

288

M. Gr˚ uber et al.

One of the measures used for such evaluation is Fleiss’ kappa. It is a statistical measure for assessing the reliability of agreement between a ﬁxed number of raters when assigning categorical ratings to a number of items or classifying items. We calculated this measure among all listeners separately for each communicative function. Computation of overall Fleiss’ kappa is impossible because the listeners were allowed to mark more than one communicative function for each sentence. However, the overall value can be evaluated as the mean of Fleiss’ kappas of all communicative functions. Another measure used here is Cohen’s kappa. It is a statistical measure of inter-rater agreement for categorical items and takes into account the agreement occurring by chance as well as Fleiss’ kappa. However, Cohen’s kappa measures the agreement only between two listeners. We decided to measure the agreement between each listener and the objective annotation obtained by maximum likelihood method. Again, calculation of Cohen’s kappa was made for each communicative function separately. Thus we can ﬁnd out whether particular listener was in agreement with the objective annotation for certain communicative function. Finally, the mean of Cohen’s kappas of all communicative functions was calculated. Results of agreement measures are presented in Table 3. Value of Fleiss’ and Cohen’s kappa vary between 0 and 1, the higher value the better agreement. More detailed interpretation of measure of agreement is in [11]. Table 3. Fleiss’ and Cohen’s kappa and occurrence probability for various communicative functions and for the “consecutive CFs” label. For Cohen’s kappa, mean value and standard deviation is presented, since Cohen kappa is measured between annotation of each listener and the reference annotation. communication function DIRECTIVE REQUEST WAIT APOLOGY GREETING GOODBYE THANKS SURPRISE SAD-EMPATHY HAPPY-EMPATHY SHOW-INTEREST CONFIRM DISCONFIRM ENCOURAGE NOT-SPECIFIED OTHER mean

Fleiss’s Measure kappa of agreement 0.7282 Substantial 0.5719 Moderate 0.5304 Moderate 0.6047 Substantial 0.7835 Substantial 0.7408 Substantial 0.8285 Almost perfect 0.2477 Fair 0.6746 Substantial 0.6525 Substantial 0.4485 Moderate 0.8444 Almost perfect 0.4928 Moderate 0.3739 Fair 0.1495 Slight 0.0220 Slight 0.5434 Moderate

Cohen’s Cohen’s Measure Occurr. kappa kappa SD of agreement probab. 0.8457 0.1308 Almost perfect 0.0236 0.7280 0.1638 Substantial 0.0436 0.7015 0.4190 Substantial 0.0073 0.7128 0.2321 Substantial 0.0059 0.8675 0.1287 Almost perfect 0.0137 0.7254 0.1365 Substantial 0.0164 0.8941 0.1352 Almost perfect 0.0073 0.4064 0.1518 Moderate 0.0419 0.7663 0.0590 Substantial 0.0344 0.7416 0.1637 Substantial 0.0862 0.6315 0.3656 Substantial 0.3488 0.9148 0.0969 Almost perfect 0.1319 0.7153 0.1660 Substantial 0.0023 0.5914 0.3670 Moderate 0.2936 0.3295 0.2292 Fair 0.0736 0.0391 0.0595 Slight 0.0001 0.6632 Substantial

Data Collection and Expressive Corpus Recording and Annotation

289

The Fleiss’ kappa mean value of 0.5434 means that the measure of interlisteners agreement is moderate. The Cohen’s kappa mean value of 0.6632 means that the measure of agreement between listeners and objective annotation is substantial. As it is obvious from Table 3, communicative functions OTHER and NOT-SPECIFIED should be considered as poorly recognizable. It is understandable when taking into consideration their deﬁnitions. Additionally, in Table 3, there are also shown probabilities of the particular communicative functions occurrence when maximum likelihood method was used for the objective annotation obtaining. It is obvious that communicative functions SHOW-INTEREST and ENCOURAGE are the most frequent.

6

Conclusions and Future Work

This paper described data collection and annotation eﬀorts needed for preparation of the corpora that were and/or are going to be used for the development of the Czech Senior Companion dialogue system. Since the TTS corpora annotation is ﬁnished, the unit-selection algorithm in the Czech TTS system will be modiﬁed by changing the target-cost function so that the function will include a new feature for communicative function representation. The modiﬁed unit-selection algorithm will be hopefully able to generate speech expressing various communicative functions with implicit acoustic emotional cues.

Acknowledgements This work was funded by the Ministry of Education of the Czech Republic, project No. 1M0567, and in part by the Companions project (www.companionsproject.org) sponsored by the European Commission as part of the Information Society Technologies (IST) programme under EC grant number IST-FP6-034434.

References 1. Whittaker, S., Walker, M., Moore, J.: Fish or Fowl: A Wizard of Oz Evaluation of Dialogue Strategies in the Restaurant Domain. In: LREC 2002, Gran Canaria, Spain (2002) 2. Strauss, P.-M., Hoﬀmann, H., Scherer, S.: Evaluation and User Acceptance of a Dialogue System Using Wizard-of-Oz Recordings. In: IE 2007, Ulm, Germany, pp. 521–524 (2007) 3. Wilks, Y.: Artiﬁcial companions. Interdisciplinary Science Reviews 30, 145–152 (2005) ˇ 4. Zelezn´ y, M., Krˇ noul, Z., C´ısaˇr, P., Matouˇsek, J.: Design, implementation and evaluation of the czech realistic audio-visual speech synthesis. Signal Processing 12, 3657–3673 (2006) 5. Zaj´ıc, Z., Machlica, L., M¨ uller, L.: Reﬁnement approach for adaptation based on combination of MAP and fMLLR. In: Matouˇsek, V., Mautner, P. (eds.) TSD 2009. LNCS, vol. 5729, pp. 274–281. Springer, Heidelberg (2009)

290

M. Gr˚ uber et al.

6. Ekman, P.: Basic emotions. In: Dalgleish, T., Power, M.J. (eds.) The Handbook of Cognition and Emotion, pp. 45–60. John Wiley & Sons Ltd, New York (1999) 7. Russell, J.A.: A circumplex model of aﬀect. Journal of Personality and Social Psychology 39(6), 1161–1178 (1980) 8. Syrdal, A., Kim, Y.-J.: Dialog speech acts and prosody: Considerations for TTS. In: Speech Prosody 2008, Campinas, Brazil (2008) 9. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Statist. Soc. Ser. B 39(1), 1–38 (1977) (with discussion) 10. Romportl, J.: Prosodic phrases and semantic accents in speech corpus for czech TTS synthesis. In: Sojka, P., Hor´ ak, A., Kopeˇcek, I., Pala, K. (eds.) TSD 2008. LNCS (LNAI), vol. 5246, pp. 493–500. Springer, Heidelberg (2008) 11. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)

Abstractive Summarization of Voice Communications Vincenzo Pallotta1, Rodolfo Delmonte2, and Antonella Bristot2 1

Department of Computer Science Webster University, Geneva Route de Collex 15 CH-1293 Bellevue, Switzerland [email protected] 2 Department of Language Science Università “Ca Foscari” 30123 – Venezia, Italy [email protected]

Abstract. Abstract summarization of conversations is a very challenging task that requires full understanding of the dialog turns, their roles and relationships in the conversations. We present an efficient system, derived from a fullyfledged text analysis system that performs the necessary linguistic analysis of turns in conversations and provides useful argumentative labels to build synthetic abstractive summaries of conversations. Keywords: summarization, dialogue, argumentation.

1 Introduction We consider an open problem in Natural Language Processing (or Computational Linguistics) that is the generation of text summaries from voice communication that is spoken dialogs occurring between two or more people. Voice communications can happen either during face-to-face encounters or when parties are remotely connected and using communication services such as VoIP, teleconferencing or just telephones. In this work, we start from the assumption that voice communication can perfectly transcribed and that transcription is segmented into turns assigned to each speaking participants. While this assumption might seem restrictive, most of corpora of voice communications are of this kind. Moreover, the cost of transcribing speech is far lower than the cost of summarization. The problem of automatic transcription of voice communication is beyond the scope of this work. However, the algorithm we present has characteristics that make it robust with respect to speech disfluency, either naturally occurring or generated from recognition errors of automatic transcription systems. Automatic summarization of voice communications would be a very useful tool that allows enterprises in extracting knowledge from meeting records and integrating this knowledge into corporate knowledge bases for future access by questionanswering. For instance, one could ask the system questions like “why this decision was made” or “who rejected the proposal made by XX?” or “How the decision of Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 291–302, 2011. © Springer-Verlag Berlin Heidelberg 2011

292

V. Pallotta, R. Delmonte, and A. Bristot

doing X impacted the progress of the project Y?” While very likely to be asked by user of meeting recording systems [1], these kinds of question are very hard to answer because one has to consider several meetings and know about the context of these meetings (e.g. the projects, the people, the roles, the duties, the agenda, the corporate policies and strategies). Moreover, it requires a deep understanding of the meeting situations such as its dynamics, the rules of order adopted, the specific language used, and the culture-specific rituals. Producing automatic summaries of meeting would benefit the enterprise because it would turn implicit and tacit information into explicit, actionable knowledge. It would save time because people who did not attend the meeting could have a quick insightful glimpse of what happened during the meeting without having to replay the whole recording with a meeting browser [2]. In more general terms, this approach to summarization of spoken data would be also beneficial to cope with information overload arising from many audio-visual data on TV channels and on the Internet such as broadcasted news, talk shows, podcasts, webinars, etc. 1.1 Related Work The problem of automatic speech summarization was initially investigated in the 80’ in the context of several DARPA projects [3]. Automatic summarization of meetings has been typically approached in a so-called “extractive” fashion that is by extracting excerpts of the dialogs and by assembling them into a hopefully coherent text [4, 5]. In fact, this method has severe limitations due to the intrinsic characteristics of the source data: conversations are not as coherent as ordinary narrative text (such as news or scientific articles) and obtaining a coherent text from conversations is practically impossible using the extractive approach. Any system that performs extractive summarization must be evaluated against human-annotated test datasets. As pointed out in [6], inter-annotator agreement is very low for this type of task, which makes test data nearly useless for evaluation. Intuitively, selecting salient content from conversations is a really difficult task and subjective selection of excerpts leads to fairly different results. Moreover, the proposed solutions for extractive summarization of meetings have already reached their qualitative upper bounds [7]. Abstractive summarization of dialogs has only been recently considered as an important challenge in several research projects among which the IM21 project, ICSI Meeting Recorder2 project, the European FP6 AMI3 project, and the DARPA’s CALO4 project. At the beginning, these projects mainly focused on the first (and simplest) part of the problem that is capturing and storing the multimodal events occurring during a voice communications and provide a multimedia record of the events. This record includes aligned media such as audio, video, speech transcription, pointers to document used, gestures, gaze and additional information such as the emotional status of the speakers (involved, bored, etc.). These types of information are indeed very useful for the subsequent step of analysis whose goal is to provide an 1

http://www.im2.ch/ http://www.icsi.berkeley.edu/Speech/mr/ 3 http://www.amiproject.org/ 4 http://caloproject.sri.com/ 2

Abstractive Summarization of Voice Communications

293

automatic summary of the meeting. On this side, some advances have been achieved such as extracting (very basic) “dialog acts” [8], detecting “agreement and disagreement” [9, 10], segmenting the meeting into “thematic zones” [11], and, only recently, detecting “decision points” [12, 13], “action items” [14, 15], and “subjective statements” [16]. Also in the framework of the AMI project, a group was investigating on the automatic argumentative structuring of multi-party dialogs [17]. Unfortunately, although very relevant, none of these works have directly addressed the problem of abstractive summarization of voice communication and most of the advances on the detection of rich content information from meetings were mainly exploited in improving extractive summarization, which, as said before, can be only improved up to a given (unsatisfactory) upper bound. The problem of abstractive summarization of voice communications is apparently much harder to solve than its extractive version. It requires almost the full understanding of the source data [18]. Moreover, as also recognized by [19], in order to perform the generation of an abstractive summary it is necessary to classify participants’ contributions according to their informativeness and function in the conversation so that an appropriate ontology of the meeting could be adequately populated. In [20] the problem of abstractive summarization of conversations is framed as a sentence compression problem. This work is an important step in filling the gap between extractive and abstractive summarization. However, they recognize the need of a sophisticated extraction and generation algorithm to achieve full abstractive summarization. In fact, their approach is not appropriate for generating abstracts of conversations where turns are already highly compressed. We conclude our review of related work by looking at [21] where a language generation algorithm produces indicative abstracts of meeting transcriptions. The existing annotations of the AMI corpus data are used to create a content representation that is subsequently fed to a sentence planner. The semantic representation produced is then rendered into a surface form with surface generator. This work is in spirit very similar to ours with the notable exception that the semantic representation is not as rich as ours. Basically, only topics, very general meeting stages (opening, debating, closing), and temporal sequencing of topics discussed are used to generate the abstract. As we will show in the next sections, using similar (or even simpler) generation techniques and with the information we are able to extract, we can provide much richer indicative abstracts. In order to extract the information that allows us to provide better abstracts we need a very powerful syntactic analyzer (i.e. a parser) and a mapping of the analysis onto semantic and pragmatic representations. This step is obtained through a discourse analyzer and an argumentative analysis algorithm. 1.2 Paper Outline The paper is organized with a presentation of the system in section 2 which we evaluated on a manually annotated corpus of meeting transcriptions. In section 3 we present a preliminary generation system that provides summaries in form of memos. A final section is devoted to the conclusions and future work.

294

V. Pallotta, R. Delmonte, and A. Bristot

2 Automatic Argumentative Analysis We start from transcribed turns in multi-party dialogs such as those available from corpuses built within the projects we described earlier. Each turn is assigned to a speaker and the transcription is assumed to be 100% accurate5. Our solution to the problem of abstractive summarization of voice communications would be learning a general semantic/pragmatic structure for meeting dialogs data that could be specialized for different voice communication types such as brainstorming, design, formal meetings with rules of order, board meetings, and interviews. We were looking at pervasive pragmatic phenomena in dialogs and we realized that most of encountered dialogs include argumentative processes. Basically, exchange of opinions, questions-answers, negotiations and collaborative decision-making are at the hearth of voice communications. Unfortunately, argumentative processes are not sufficiently lexically marked in dialogs to justify extractive summarization. For instance, extracting a turn containing the single word “no” cannot summarize a disagreement to a proposal about an issue. This action is the result of a complex, sometimes long exchange between two or more participants. By simply extracting some turns from the dialog some essential parts of the context are lost. In contrast, it would be much better to provide a description where an argumentative-like terminology is used such as, for instance, “X disagreed to the proposal made by Y on the issue raised by Z”. This paraphrase is made of words that do not occur in the dialog. Our system is capable to recognize a small but relevant set of argumentative action that can be used to produce such description and thus creating a meaningful abstractive summary of the analyzed dialog. While the generation of the summaries might be quite standard given the extracted information, we will mainly focus in this paper on the recognition of argumentative events as the core component required for implementing an abstractive summarizer of voice communications. Computing semantic representations for argumentative annotation of conversations is a particularly hard task which – when compared to written text processing – requires the following additional information to be made available: • • • •

adequate treatment of fragments; adequate treatment of short turns, in particular one/two-words turns; adequate treatment of first person singular and plural pronominal expressions; adequate treatment of disfluency, thus including cases of turns made up of just such expressions, or cases when they are found inside the utterance; • adequate treatment of overlaps; • adequate treatment of speaker identity for pronominal co-reference; In addition, every dialog turn must receive one polarity label, indicating negativity or positivity, and this is computed by looking into a dictionary of polarity items. This can subsequently used to make decisions in argumentative automatic classification algorithm.

5

We plan to relax the latter constraints in future implementations of our system and deal with output from automatic speech recognition systems.

Abstractive Summarization of Voice Communications

295

2.1 Argumentative Structure – Issues and Theories We consider the adoption of a structured representation of conversational content based on argumentation theory. The argumentative structure defines the different patterns of argumentation used by participants, as well as their organization and synchronization in the discussion. A dialog is decomposed into several stages such as issues, proposals, and positions, each stage being possibly related to specific aggregations of elementary dialog acts [22]. Moreover, argumentative interactions may be viewed as specific parts of the discussion where several dialog acts are combined to build such an interaction; as for instance, a disagreement could be seen as an aggregation of several acts of reject and accept of the same proposal. From this perspective, we adopted an argumentative coding scheme (or ontology), the Meeting Description Schema [23]. In MDS, the argumentative structure of a meeting is composed of a set of topic discussion episodes (a discussion about a specific topic). In each discussing topic, there exists a set of issue discussion episodes. An issue is generally a local problem in a larger topic to be discussed and solved. Participants propose alternatives, solutions, opinions, ideas, etc. in order to achieve a satisfactory decision. Meanwhile, participants either express their positions and standpoints through acts of accepting or rejecting proposals, or by asking questions related to the current proposals. Hence, for each issue, there is a corresponding set of proposals episodes (i.e., solutions, alternatives, ideas, etc.) that are linked to a certain number of related positions episodes, for example, a rejection to a proposed alternative in a discussing issue, or questions and answers. 2.2 Computing Argumentative Annotations The core of our solution is based on adapting and extending GETARUNS [24, 25], a system for text understanding developed at the University of Venice. GETARUNS is organized as a pipeline that includes two versions of the system: the Partial and the Deep GETARUNS. The Deep version, used in this work, is equipped with three main modules: 1. The lower module for parsing, where sentence strategies are implemented; 2. The middle module for semantic interpretation and discourse model construction which is cast into Situation Semantics; 3. The higher module where reasoning and generation takes place. Automatic Argumentative Annotation (A3) is carried out by a special module of the GETARUNS system activated at the very end of the computation of the each dialog. This module takes as input the complete semantic representation produced by the system. The elements of semantic representation we use are the following ones: • the Discourse Model, which include individuals, sets, classes, cardinality, properties related to entities by means of their semantic indices; • information about spatio-temporal locations of events with logical operators and semantic indices; • vectors of informational structure containing semantic information at propositional level, computed for each clause;

296

V. Pallotta, R. Delmonte, and A. Bristot

• vectors of discourse structure with discourse relations computed for each clause from informational structure and previous discourse state; • dialog acts labels associated to each utterance or turn; • overlaps information computed at utterance level; • topic labels associated to semantic indices of each entity marked as topic of discourse; • all utterances with their indices as they have been automatically split by the system. 2.3 The A3 Algorithm The A3 algorithm, described in details in [26], uses the following 21 discourse relations: statement, narration, adverse, result, cause, motivation, explanation, question, hypothesis, elaboration, permission, inception, circumstance, obligation, evaluation, agreement, contrast, evidence, hypoth, setting, prohibition. These relations are then mapped onto five general argumentative labels: ACCEPT, REJECT/DISAGREE, PROPOSE/SUGGEST, EXPLAIN/JUSTIFY, REQUEST. In addition we use the label DISFLUENCY for all those turns that contain fragments that are non-sentences and are not semantically interpretable. The algorithm is outlined as follows: 1. 2. 3. 4.

5.

6

It recovers Dialog Acts for each dialog turn as they have been assigned by the system6; It recovers Overlaps as they have been marked during the analysis; It produces a Polarity attribute for the Opinion label, which can take one of two values: Positive or Negative; It produces a list of Hot Spots (i.e. a set of turns in sequence where the interlocutors overlap each other frequently) and builds up Episodes (i.e. a set of turns in which a single speaker “arguments” his/her topics without interruption). Episodes may occasionally be interrupted by overlaps or by short continuers, backchannel or other similar phenomena by other speakers without however grabbing the floor; It assigns a set of argumentative labels, one for each clause. The system then chooses the label to associate to the turn utterance from multiple candidates by looking at a hierarchy of argumentative labels graded for Pragmatic Relevance which establishes that, for instance, Question is more relevant than Negation, which is in turn more relevant than Raising an Issue, etc.

The dialog act set used by GETARUNS differs from that used in the ICSI corpus and described in [22]. Details are available in [26].

Abstractive Summarization of Voice Communications

297

2.4 Experimental Results When we started using the system to parse ICSI corpus of meetings [27], we realized that the generated semantic representation and the output of the syntactic parser were both inadequate. So we worked at the deficiencies we identified in an empirical manner. This approach made us aware of the peculiarities of spoken dialog texts such as the ones made available in ICSI corpus, and to the way to implement solutions in such a complex system. ICSI dialogs are characterized by the need to argument in an exhaustive manner the topics to be debated which are the themes of each multiparty dialog. The mean length of utterances/turns in each dialog we parsed was rather long. This makes ICSI dialogs hard to compute. From a count of number of words per turn, we came up with the following average figures: • • • •

percent of turns made of one single word: 30% percent of turns made of up to three words: 40% average number of words x turn overall: 7 average number of words x turn after subtracting short utterances: 11

These values correspond to those found for PennTreebank corpus where we can count up to 94K sentences for 1M words – again 11 words per sentence. In analyzing ICSI, we found turns with as much as 54 words depending on the topic under discussion and on the number of people on the floor. Our system has been used to parse the first 10 dialogs of the ICSI corpus for a total number of 98523 words and 13803 turns. This has been done to “train” the system: what happened was that, for the first 5 dialogs, we had to take care of failures. We also had to tune all the modules and procedures carefully. In particular, the module for argumentative automatic classification was incrementally improved in order to cover all conventional ways to express agreement. For this reason, we then chose two random additional dialogs to test this second task. We had one skilled linguist to provide a turn level annotation for argumentative labels: we don’t have any inter-annotator agreement measure in this case, even though we expect the annotation to be in line with experiments on the same subject presented in [28], namely a Kappa of 0.42 and a raw agreement of 62%. On a total of 2304 turns, 2251 have received an argumentative automatic classification, with a Recall of 97.53%. We computed Precision as the ratio between Correct Argumentative Labels/Found Argumentative Labels, which corresponds to 81.26%. The F-score is 88.65%.

3 Abstract Summarization of Conversations A complementary way to provide users with simplified access to conversational content is by means of summaries. Analyzing and summarizing conversations (or dialogues) is very challenging [29]. Many existing summarization techniques are tailored for the narrative genre and can hardly be adapted to the dialogue genre. We advocate for abstractive summarization of conversational content. Abstractive summarization of narrative texts is typically based on sentence compression and/or

298

V. Pallotta, R. Delmonte, and A. Bristot

paraphrase [20]. Our approach differs from that of [21] because our indicative abstracts are not limited to exclusively thematic content. Instead, our abstractive summarization system generates descriptions of the conversation dynamics based on both thematic content and argumentative structure we are able to automatically extract as described in section 2. The output of our system consists instead of a memo made of several sections, namely describing the conversational settings, participants’ number and names, statistics about the interactivity of participants (e.g. the degree of competitiveness), the topics discussed, and the arguments. Arguments are grouped into episodes bound together by thematic cohesion. It’s important to notice the use of pronouns in the paraphrase and the insertion of discourse markers to indicate the continuation by the same speaker. Formulating adequate paraphrases may sound awkward sometimes and this is due to the inherent difficulty of the task. The following is a (partial) example of a memo which is generated by our system from the analysis of an ICSI meeting (only turns and speaker with no additional annotations): MEETING MEMO GENERAL INFORMATION ON PARTICIPANTS The participants to the meeting are 7. Participants less actively involved are Ami and Don who only intervened respectively for 38 and 68 turns. LEVEL OF INTERACTIVITY IN THE DISCUSSION The speaker that has held the majority of turns is Adam with a total of 722 turns, followed by Fey with a total of 561. The speaker that has undergone the majority of overlaps is Adam followed by Jane. The speaker that has done the majority of overlaps is Jane followed by Fey. Jane is the participant that has been most competitive. DISCUSSION TOPICS The discussion was centred on the following topics: schemas, action, things and domain. The main topics have been introduced by the most important speaker of the meeting. The participant who introduced the main topics in the meeting is: Adam. The most frequent entities in the whole dialogue partly coincide with the best topics, and are the following: action, schema, things, 'source-path-goal', person, spg, roles, bakery, intention, specific, case, categories, information, idea. ARGUMENTATIVE CONTENT The following participants: Andreas, Dave, Don, Jane, Morgan expressed their dissent 52 times. However Dave, Andreas and Morgan expressed dissent in a consistently smaller percentage. The following participants: Adam, Andreas, Dave, Don, Jane, Morgan asked questions 55 times. The remaining 1210 turns expressed positive content by proposing, explaining or raising issues. However Adam, Dave and Andreas suggested and raised new issues in a consistently smaller percentage. The following participants: Adam, Andreas, Dave, Don, Jane, Morgan expressed acceptance 213 times.

Abstractive Summarization of Voice Communications

299

EPISODE ISSUE No. 1 In this episode we have the following argumentative exchanges between Don, Jane and Morgan: Don raises the following issue [um so - so it definitely had that as a concept .] then he, overlapped by Morgan, continues [so th ... it has a single timeline .] Morgan accepts the previous explanation [uhhuh, .] Then Don elaborates the following explanation [and then you can have lots of different sections.] then he continues [each of which have i.d.’s attached to it .] [and then you can refer from other sections to those i.d.’s .] [if you want to .] [so that you start with - with a timeline tag .] [timeline .] [and_then, you, have, a, bunch, of, times, .] [I don’t ... I don’t remember exactly what my notation was .] Jane, overlapped by Don, raises the following issue [oh; i, remember seeing an example of this .] Jane, overlapped by Dave, suggests the following explanation [actually we - we use a generalization of the - the sphere format .] [yeah so there is something like that .] [but it’s um probably not as sophist... .] Don asks the following question [well what does H.T.K. do for features ?] then he, overlapped by Jane, continues [or does it even have a concept of features ?] Jane provides the following answer [they h... it has its own ... .] then she , overlapped by Morgan, continues [I mean Entropic has their own feature format that’s called like s... s.d. or some s... s.f. or something like that.] Morgan accepts the previous explanation [yeah.] Don, overlapped by Morgan, provides the following explanation [you know we’ll do something where we - some kind of data reduction where the prosodic features are sort ... uh either at the word level or at the segment level .] Morgan agrees [right, .] then he continues [or - or something like that .] [they’re not going to be at the phone level and they’re n- ... not going to be at the frame level when we get done with sort of giving them simpler shapes and things .] [and so; the main thing is just being able ... .] [well I guess the two goals .] Don disagrees with the previous explanation [um one that Chuck mentioned is starting out with something that we don’t have to start over that we don’t have to throw away if other people want to extend it for other kinds of questions .] Morgan agreees [right, .] ...

The generated memo outlines the extracted argumentative structure and can be used to generate a more succinct abstract where many details of the conversation are

300

V. Pallotta, R. Delmonte, and A. Bristot

hidden. For instance, explanations and other types of elaborations around proposals or issues can be omitted. Moreover, the details of agreements and disagreements (e.g. the reasons) can be omitted. Finally, overlaps (recognized as backchannels) can also be removed. For the previous example, the episode 1 can be summarized in the following manner: Don, Jane and Morgan discussed about “single timeline”. Jane raised an issue: “I remember seeing an example of this”. Don asked Jane to clarify and Morgan accepted Jane’s explanation. Morgan agreed and provided an explanation. Don disagreed..

From the above abstract it is possible to understand what type of interaction was ongoing between the three participants without having to look at the details. From this abstract one can already figure out that the topic “single timeline” was particularly hot, that no agreement was reached at that point, and that Jane’s opinion was shared by Morgan, but not by Don.

4 Conclusions We have presented work carried out to extend and adapt a natural language understanding system designed for text analysis in order to make it fit for dialogue understanding as the core component of abstractive summarization of voice communications technology. We implemented a set of enhancements to cope with typical dialogue related problems, such as presence of non-sentential fragments, elliptical fragments interpretable as speech acts, massive presence of generic nonreferential pronominal expressions, etc. We implemented a number of additional components: an algorithm that takes care of overlaps and uses that information to split current utterances and temporally realign the conversational flow. We outlined the A3 algorithm for the automatic argumentative labelling and linking of conversation turns, which relies on discourse relations and other semantic markers determined by the semantic component of the system. The system has been evaluated for the argumentative classification module. Results are very encouraging. However, we note that in that task, labels which may cause great uncertainty and are highly ambiguous (e.g. explain and justify), have been lumped together to facilitate the classification task. 4.1 Future Work Of course we intend to complete the analysis of all dialogues contained in the ICSI corpus and refine our algorithms. In particular we still need to work at the level of DECISION labelling, and to improve the discrimination of really argumentative from pragmatically irrelevant utterance, a choice that in some cases is hard to make on an automatic basis. We would like to formally assess the robustness of the system by applying the algorithm to the output of automatic transcription systems and evaluate its degree of degradation. We also need to design an evaluation framework for abstractive summaries of conversations because the current metrics used for evaluating extractive summaries are not well adapted (e.g. ROUGE). We agree with [30] that the evaluation of

Abstractive Summarization of Voice Communications

301

abstractive summaries must be extrinsic, i.e. based on human judgement or indirectly observed as its helpfulness in performing a given task.

References [1] Pallotta, V., Seretan, V., Ailomaa, M.: User requirements analysis for Meeting Information Retrieval based on query elicitation. In: Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL 2007), Prague, pp. 1008–1015 (2007) [2] Cremers, A.H.M., Hilhorst, B., Vermeeren, A.P.O.S.: What was discussed by whom, how, when and where? personalized browsing of annotated multimedia meeting recordings. In: Proceedings of HCI 2005, Edinburgh, UK, pp. 1–10 (2005) [3] Zechner, K.: Automatic Summarization of Open-Domain Multiparty Dialogues in Diverse Genres. Computational Linguistics 28(4) (2002) [4] Murray, G., Renals, S., Carletta, J.: Extractive summarization of meeting recordings. In: Proceedings of the 9th European Conference on Speech Communication and Technology, pp. 593–596 (2005) [5] Garg, N., Favre, B., Reidhammer, K., Hakkani-Tür, D.: ClusterRank: A Graph Based Method for Meeting Summarization. In: Proceedings of Interspeech 2009, Brighton, UK (2009) [6] Buist, A.H., Kraaij, W., Raaijmakers, S.: Automatic Summarization of Meeting Data: A feasibility Study. In: Proceedings of the 15th CLIN Conference (2005) [7] Riedhammer, K., Gillick, D., Favre, B., Hakkani-Tür, D.: Packing the Meeting Summarization Knapsack. In: Proceedings of the 9th International Conference of the ISCA (Interspeech 2008), Brisbane, Australia, pp. 2434–2437 (2008) [8] Ang, J., Liu, Y., Shriberg, E.: Automatic dialog act segmentation and classification in multiparty meetings. In: Proceedings of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Philadelphia (March 2005) [9] Galley, M., McKeown, K., Hirschberg, J., Shriberg, E.: Identifying Agreement and Disagreement in Conversational Speech: Use of Bayesian Networks to Model Pragmatic Dependencies. In: Proceedings of 42nd Meeting of the ACL, Barcelona, July 21-26 (2004) [10] Hillard, D., Ostendorf, M., Shriberg, E.: Detection of agreement vs. disagreement in meetings: Training with unlabeled data. In: Proceedings of HLT-NAACL 2003 (2003) [11] Hsueh, P., Moore, J.D.: Automatic topic segmentation and labeling in multiparty dialogs. In: Proceedings of Spoken Language Technology Workshop, pp. 98–101. IEEE, Los Alamitos (2006) [12] Hsueh, P., Moore, J.D.: Automatic Decision Detection in Meeting Speech. In: PopescuBelis, A., Renals, S., Bourlard, H. (eds.) MLMI 2007. LNCS, vol. 4892, pp. 168–179. Springer, Heidelberg (2008) [13] Hsueh, P., Moore, J.D.: What Decisions Have You Made: Automatic Decision Detection in Conversational Speech. In: Proceedings of HLT-NAACL (2009) [14] Niekrasz, J., Purver, M., Dowding, J., Peters, S.: Ontology-Based Discourse Understanding for a Persistent Meeting Assistant. In: Proceedings of the AAAI Spring Symposium Persistent Assistants: Living and Working with AI, Stanford (2005) [15] Murray, G., Renals, S.: Detecting action items in meetings. In: Popescu-Belis, A., Stiefelhagen, R. (eds.) MLMI 2008. LNCS, vol. 5237. Springer, Heidelberg (2008)

302

V. Pallotta, R. Delmonte, and A. Bristot

[16] Raaijmakers, S., Truong, K., Wilson, T.: Multimodal subjectivity analysis of multiparty conversation. In: Proc. of EMNLP 2008, Honolulu, HI, USA (2008) [17] Rienks, R.J., Heylen, D., van der Weijden, E.: Argument diagramming of meeting conversations. In: Proceedings of the Workshop on Multimodal Multiparty Meeting Processing at the 7th International Conference on Multimodal Interfaces (ICMI), Trento, Italy, pp. 85–92 (2005) [18] Hahn, U., Mani, I.: The Challenges of Automatic Summarization. IEEE Computer 33(11), 29–36 (2000) [19] Murray, G., Carenini, C., Ng, R.: Interpretation and Transformation for Abstracting Conversations. In: Proceedings of Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL (HLT-NACL), Los Angeles, California, pp. 894–902 (June 2010) [20] Liu, F., And Liu, Y.: From Extractive to Abstractive Meeting Summaries: Can It Be Done by Sentence Compression? In: Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, Suntec, Singapore, August 4, pp. 261–264 (2009) [21] Kleinbauer, T., Becker, S., Becker, T.: Indicative Abstractive Summaries of Meetings. In: Proceedings of 4th Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI 2007), Nagoya, Japan (2007) [22] Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., Van Ess-Dykema, C., Meteeret, M.: Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational Linguistics 26(3), 339 (2000) [23] Pallotta, V.: Framing Arguments. In: Proceedings of the International Conference on Argumentation, Amsterdam, Netherlands (June 2006) [24] Delmonte, R.: Computational Linguistic Text Processing – Logical Form, Semantic Interpretation, Discourse Relations and Question Answering. Nova Science Publishers, New York (2007) [25] Delmonte, R.: Computational Linguistic Text Processing – Lexicon, Grammar, Parsing and Anaphora Resolution. Nova Science Publishers, New York (2009) [26] Delmonte, R., Bristot, A., Pallotta, V.: Towards Automatic Dialogue Understanding. In: Proceedings of the International Conference on Spoken Communication (GSCP 2009), Naples, Italy, February 23-25 (2009) [27] Janin, A., Baron, D., Edwards, J., Ellis, D., Gelbart, D., Morgan, N., Peskin, B., Pfau, T., Shriberg, E., Stolcke, A., Wooters, C.: The ICSI Meeting Corpus. In: Proceedings of IEEE/ICASSP 2003, Hong Kong, April 6-10, vol. 1, pp. 364–367 (2003) [28] Pallotta, V., Seretan, V., Ailomaa, M., Ghorbel, H., Rajman, M.: Towards an argumentative coding scheme for annotating meeting dialogue data. In: Proceedings of the 10th International Pragmatics Conference, IPrA 2007, Göthemburg, July 8-13 (2007) [29] Maybury, M.: Keynote on Searching Conversational Speech. In: Proceedings Workshop on Searching Spontaneous Conversational Speech Part of the ACM SIGIR 2007 Conference, Amsterdam, July 27 (2007) [30] Murray, G., Kleinbauer, T., Poller, P., Renals, S., Becker, T., Kilgour, J.: Extrinsic Summarization Evaluation: A Decision Audit Task. In: Popescu-Belis, A., Stiefelhagen, R. (eds.) MLMI 2008. LNCS, vol. 5237, pp. 349–361. Springer, Heidelberg (2008)

Natural Language Based Communication between Human Users and the Emergency Center: POLINT-112-SMS Zygmunt Vetulani and Jacek Marciniak Adam Mickiewicz University Department of Computer Linguistics and Artificial Intelligence ul. Umultowska 87, 61-614 Poznań, Poland {vetulani,jacekmar}@amu.edu.pl

Abstract. We present a prototype of the POLINT-112-SMS system intended to support information management and decision making in emergency situations. The system is able to interpret short message service (SMS) texts, and understands and processes information provided by the human user. We present the project's specific development methodology and the main modules of the system: SMS gate, NLP Module (processing Polish), Situation Analysis Module (SAM), Dialogue Maintenance Module (DMM), Visualizing Module (VIS) as well as some resources developed within the project (PolNet). Keywords: artificial intelligence, computer understanding systems, SMS, human-computer interaction, crisis management tools, public security.

1 Credits This work has been partly supported by the Polish Ministry of Science and Higher Education, grant R00 028 02 (the “Polish Text Processing Technologies for the Public Security Oriented Applications” project within the Polish Platform for Homeland Security /PPBW/).

2 Introduction Natural language is the most common and most sophisticated communication system used by humans. It was decisive for the evolutionary success of humans and human dominance over the rest of the animate world. Natural language has, however, several serious drawbacks1. Noisy environments, different IQ/cultural backgrounds of speakers, emotional attitudes, stress and insufficient understanding of the situation in which the speech act is performed may have negative impact on the quality of communication acts. This is particularly true when natural language is used in emergency situations. Human language processing technologies may help to solve such problems. 1

Which confirms the view that evolution has not ended yet!

Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 303–314, 2011. © Springer-Verlag Berlin Heidelberg 2011

304

Z. Vetulani and J. Marciniak

The main goal of the project is to design and implement an advanced high-level system to support natural language communication between an emergency management system (e.g. the 112 telephone service) and services or individuals needing to exchange information. It aims at facilitating information exchange in difficult conditions (time pressure, fragmentary data, uncertainty of information, deliberate disinformation). In the project we are focusing on text input (text messages) for reasons provided below. The prototype of the system presented here (POLINT-112-SMS) was tested at a soccer match played in front of a large crowd (over 20,000 spectators). Such a large crowd is a well-identified potential source of dangerous situations which can easily get out of control. The role of security staff is to detect risks and to report them to the Decision Making Body. The system is intended to support the staff in its role of information provider and to improve decision premises. The following assumptions are made: -

-

information may be sent by people acting independently of each other at the stage of information acquisition, the information is sent to the system and (possibly) also to human operators, the information is send in the form of SMS texts written in natural language (Polish) or in a controlled subset (sublanguage) of natural language (a subset of Polish), the information concerns a specific situation/event, the information entered into the system may be untrustworthy, incomplete or imprecise; also, contradictory statements may occur, the system has passive language competence, i.e. it understands language and processes information, in particular, the system visualizes the reported situations and answers user questions online.

The following Fig. 1 illustrates the main functionalities of POLINT-112-SMS with special emphasis on information flow in the decision process supported by POLINT112-SMS.

Fig. 1. Application context of the POLINT-112-SMS system

Natural Language Based Communication

305

It is important to say that the decision of choosing the text mode of communication and in particular the popular SMS technology was a deliberate and well-motivated choice. Voice-based communication has important drawbacks in an emergency context: it is hardly practicable in a very noisy environment and the content is easily accessible to unauthorized people (which is undesirable). The vision of the target system assumes that the dedicated end users are: the police, emergency services, the army. The system will support interaction between Emergency Center and information supplier. The main functionalities are: a) information acquisition from the user, b) information processing (understanding) in order to built a coherent model of the situation (on the base of information from various sources and in real time), c) coherence checking and maintaining, credibility evaluation d) possibility of replacing/assisting the human in some cases (formulating supplementary questions and/or using dialogue control) in case of information bottleneck (unavailable operators, call overload).

3 Challenging Aspects of the Project 3.1 User Modeling The interactive nature of the system means that its architecture must support dialogue management. In order to make the dialogue efficient and ergonomic it is necessary to apply an appropriate user model. Unfortunately, there is no consensus on theoretical models of the communicational behavior of the potential user of the system – security agents in emergency situation. Emergency situations of the kind we are interested in are (fortunately) rare and (unfortunately) unpredictable, so it is hard to simply base the system design on real-life situations. We therefore decided to proceed to simulation-based experiments in order to obtain the initial user requirements set (cf. also Section 4). 3.2 Linguistic Challenge of SMS Processing SMS processing is a major linguistic challenge for the project. It is well known that the natural language of SMS users exhibits important variations with respect to standard language. There is relatively little research done in this area (but cf. Fairon in [4]). There was no SMS language model for Polish and, in particular, prior to the project there was no scientifically grounded knowledge about user aptitude to use the standard natural language when communicating using SMS text messages. The intuitive feeling was that the use of the SMS-based communication environment will require a sublanguage, relatively close to the natural language, based on some (easy to learn) simplifying conventions. Considering the lack of theoretical foundations we decided to apply the incremental software development methodology as described in Section 5.

306

Z. Vetulani and J. Marciniak

3.3 Knowledge Representation and Reasoning In order to build a system able to assist a human the way the other humans do, it is essential to supply the system with a significant amount of knowledge. The knowledge is necessary for reasoning about events, individuals, time and space. The challenging task is to elaborate an ontological system to support human-like reasoning. A natural solution is to use a lexical database (wordnet). Such a tool for Polish did not exist at the time of the project kick-off so we decided to build PolNet, a WordNet for Polish ([13]).

4 System Development Methodology Building any software system requires the application of an adequate development methodology which takes into consideration the characteristics of the system and needs of the end user. Building systems with a natural language interface presupposes the application of a methodology that: -

allows the system to be provided with linguistic competence permitting unconstrained communication between human and the system, allows the implementation of knowledge acquisition and reasoning mechanisms permitting reasonable system reactions that match human expectations while interacting in natural language.

In order to reduce arbitrariness at the system design phase we have applied the model of system development based on the earlier, simpler model described in [10] (cf. Fig. 2).

Fig. 2. Initial system development model

Box 1 represents the basic elements of our approach consisting in gathering initial empirical knowledge about the way humans use language in communication acts (prior to the design and implementation phase2 of Box 2, first iteration). The model takes into account both common sense knowledge and simulation experiments involving domain experts (here, police officers). Experiments at this stage (Box 1) consisted in setting up scenarios and observing human linguistic behavior. These experiments 2

In the Figures 2 and 3 the term "Implementation" stands for "Design/Redesign and Implementation".

Natural Language Based Communication

307

permitted us to build the initial language model on the basis of recorded speech acts (human-human) observed during experimental sessions and a corpus of SMS dialogues collected in "Wizard of Oz" experiments. Experiments in Box 3 (at the iteration "i") produced small corpora to serve the "Design-and-Implementation" step of Box 2 (at the iteration "i+1"). In fact, we applied a model even more sophisticated that the one in Fig. 2, in which the iterative loop of Boxes 2 and 3 were enriched in the way presented in Fig 3. Here, by “Wizard-Assisted Experiment” we mean an experimental Human-System session in which a human actor ("Wizard")3 assists the System by taking control of the session when some of system modules (or the whole system) fails. This rescue measure is necessary in order to avoid a failure of the experiment session as a consequence of a dysfunction of some element or submodules which are marginal with respect to what is actually being tested. (E.g. for testing the syntactic coverage, failure at the semantic or pragmatic level can be avoided by the hidden intervention of the Wizard). Such a solution permits observations of the system functioning at a relatively very early stage of its development, i.e. before the system would reach the maturity level typical of the classical beta-testing phase. This solution permits a considerable saving in terms of the resources of expensive experts whose role is to interact with the system and to speed up system development (cf. [6], a project technical report).

Fig. 3. Final development model

5 Elements of the Logical/Physical Model. System Architecture The system is composed of several modules. The most important are the following: the SMS Gate, the NLP Module, the Dialogue Maintenance Module (DMM), the Situation Analysis Module (SAM), the World Knowledge Module /including Polish WordNet (PolNet)/, the Visualization Module ([12]). 1. The SMS Gate is a module to enable communication with the user by means of SMS texts. It is composed of two submodules, of which one is responsible for sending 3

The role of Wizard is like in the Wizard-of-Oz methodology.

308

Z. Vetulani and J. Marciniak

Fig. 4. Polint-112-SMS Modules

messages, while the other for receiving them. The SMS Gate communicates directly with the NLP Module. 2. The NLP Module is the main module responsible for surface understanding on the grounds of basic text processing (morphological analysis, parsing). Surface understanding procedures return partially completed frames leaving unresolved coreferences and some ambiguity issues which require pragmatic-level considerations. As interaction with the user may be helpful, these problems are processed by the Situation Analysis Module (SAM) and the Dialog Maintenance Module (DMM). The NLP Module communicates directly with the SMS Gate and with DMM. 3. The Dialogue Maintenance Module (DMM) is responsible for full understanding (“deepened” with respect to the shallow surface understanding of the NLP module) and dialogue with the user. It takes into account the data controlled by the Situation Analysis Module. Thanks to the DMM, the NLP Module focuses on transforming single sentences into data structures without storing and processing these structures. The DMM communicates directly with the NLP and SAM modules. 4. The Situation Analysis Module (SAM) is responsible for reasoning. It acts as the “brain” of the system. It controls a number of subordinate modules (responsible for supplying the ontology (PolNet), storing general knowledge, reasoning about space and time, event recognition, identification of objects, etc.). SAM reasons about the structures without communicating directly with the user. It communicates directly with the DMM (responsible for communication with the user). 5. The main role of the World Knowledge Module is to support the NLP and reasoning (SAM) modules with general information about the world concepts which is contained in a language-inspired ontology based on PolNet (see Section 7.1). PolNet is externally (but also internally, within the POLINT-112-SMS system) interfaced by the WQuery4 tool ([1], [2]). 4

WQuery is a standalone interface whose main functionalities have been integrated with the POLINT-112-SMS system.

Natural Language Based Communication

309

6. The Visualization Module (VIS) is responsible for the online on-screen display of the visual representation of current information. The module permits the user to make and store snapshots for further processing. The Visualization module communicates directly with the SAM.

Fig. 5. Visualization screen with icons representing monitored individuals, groups and events

6 Language Coverage Related Issues There is no NL understanding system as linguistically competent as humans (usually) are, but there are various domain/application-oriented systems sufficiently powerful to be useful. As general solutions are beyond our technological capacities, the engineering practice consists in limiting oneself to a precisely delimited sublanguage. Still, definition and description of the appropriate sublanguage is almost always difficult. In the case presented here, we took into account: 1. 2. 3.

the experience of the user representatives (police officers with operational expertise), simulations not involving technical environment, simulations involving a computer environment (a partially developed system) in the form of Wizard-Assisted experiments.

Measures 1 and 2 were applied at the initial phase (Box 1 of Fig. 3). Empirical observation combined with an investigation of a limited-size corpus of texts about critical situations and related themes (gathered within the project) permitted us, after having extended the system lexicon, to establish the first version of the system as an adaptation of the generic POLINT system. This version was very inadequate and we therefore decided to proceed with the “wizard-assisted” experiments (“option 3”). The experiments were carried out at a real stadium (without players or public). Scenarios were played out by actors. Additional contextual information necessary to interpret the scenarios properly was provided by means of agreed signs. E.g. waving a red flag just prior to a scene indicated “aggressive attitude”. Informers sitting in the stadium sent SMS texts to the system supervised by a human “wizard”. The role of the wizard was to intercept the dialogue control in case of system failure due to a gap in its linguistic competence (unknown word, error made by the user, too complex or too

310

Z. Vetulani and J. Marciniak

irregular syntactic structure). All wizard interventions were recorded and analyzed in order to properly fix the gaps in the parsing system (cf. NLP module). The most important findings concern: a) lexical coverage and structure, b) inventory of the most frequent syntactic structures, c) typology and structure of the most frequent errors. (Cf. for example [3] and [16] for the experimental data) Ad a. In addition to common words (mainly nouns) of general use we observed words specific to our application domain but ranked low in the general lexicon. In the stadium simulation we did not observe many abbreviations typical of everyday humanto-human SMS communication. This observation, still to be confirmed at the stage of beta testing, allows us to conclude, that the potentially hard problem of an irregular adhoc vocabulary will not obstruct system design. For isolated cases, the most practical solution is to: -

-

produce a list of the most frequently abbreviated words and propose an abbreviating convention (i.e. a common abbreviation imposed on all users); this measure must then be followed by appropriate enlargement of the system dictionary together with enlargement of the POLINT system pre-analysis procedure by abbreviation detection (using the same mechanism as for collocation detection), strongly recommend (or impose) avoiding abbreviations other than those agreed.

Ad b. Syntactic phenomena have much more impact on the quality of the system. The inspection of the experimental data permits a number of observations: -

sentences tend to be relatively short, rarely longer than 10 words, the use of relative clauses is very rare (more compact participal constructions being used instead), frequent enumerative phrases, frequent (but easy to recover) ellipsis irregular and defective use of punctuation.

The above phenomena make the observed syntax close to “telegraphic style” but much more spontaneous. In such cases parsing tools based on normative grammars will be inadequate. The solution we have proposed and (partially) implemented consists in applying pre-processing techniques in order to make the input “parsable”. Ad c. Besides the phenomena that are irregular with respect to the standard language we have noticed a huge number of errors and mistakes. Some of them are typical of quick keyboard typing and typical of fast speech. Others are due to a poor knowledge of Polish (especially concerning orthography). This was the reason for integrating the system with a spellcheck module.

7 Project Resources Each form of text processing, in particular text understanding, depends on extensive knowledge about language structures and objects. Such knowledge is typically stored in machine dictionaries, grammars, tagged and indexed corpora, etc.

Natural Language Based Communication

311

The main language resource types are: -

machine dictionaries, text corpora, digital grammars, ontologies, NLP algorithms.

The availability of necessary resources is known as an essential problem limiting technological progress. Initially, we had at our disposal (or had access to) some of them (cf. [9] for some available dictionary resources). In particular we were able to reuse the grammar and the parsing engine of earlier prototypes of the POLINT system for a subset of standard Polish (Vetulani in [8]). In the project we focused our efforts on: -

the development of PolNet – a wordnet for Polish – and of the PolNet-based ontology as a knowledge representation support, the development of a lexicon grammar for Polish, the extension of the grammatical coverage of the POLINT system (to be described in a separate publication).

7.1 PolNet In order to implement any real-scale application, real-scale resources are necessary. As sound knowledge representation tools are essential for language understanding, we considered the development of an wordnet based ontology (PolNet) (cf. [13], [14]) to be a priority. After evaluating the two main models of WordNet creation, we decided to apply the “merge model” consisting in autonomous development of the WordNet ontology (as opposed to the “expand model” consisting principally in translating an already existing one (typically Princeton WordNet)). We decided to use the EuroWordNet experience as the basic methodological reference to develop PolNet (cf. [15]). We divided our work into the following tasks: -

writing procedures (algorithm) for making synsets, creating the core of PolNet by selecting the most important words, determining their meanings and grouping them into synsets, introducing hypernymy/hyponymy relations, possibly adding missing synsets, connecting the synsets to PrincetonWordNet (considered a standard and providing links between languages) (current task). adding other internal relations (future task).

The development of PolNet proceeded efficiently due to the tools provided by the Czech partners of the PolNet project: the VisDic tool at the start-up of the project, subsequently replaced by newly developed DEBVisDic tool perfectly suited to collaborative work ([5]). This work started at the beginning of 2007 and quickly entered its mature phase resulting in over 11,600 noun-based synsets. We have now proceeded to quality checking of this basic resource and have started working on the verb-based part of PolNet in connection to the development of the lexicon grammar of Polish (within the CITTA project which is an extension of POLINT-112-SMS project for the tourism domain, over 1800 verb synsets have been defined). The resource obtained so far will be opened up for free access by the research community.

312

Z. Vetulani and J. Marciniak

7.2 Verbo-Nominal Collocations Dictionary An important effort has been made in the development of a dictionary of verbonominal collocations, i.e. compounds formed by a support verb together with a predicative noun. This research consists in retrieving from the corpus and providing syntactic-semantic descriptions of the most common collocations. This research contributes to the language resource development, and consequently to the Polish language description, as well as to the development of the POLINT-based application. (Cf. papers [7] and [11]). The essential part of the target dictionary is now being integrated with PolNet.

8 Concluding Remarks The Polint-112-SMS prototype has been implemented and has passed initial beta tests in a controlled stadium-like environment. After necessary modification of the parser and dictionaries, this prototype enables real-time information exchange between the system and several human users (in parallel) via SMS messages. The practical utility of the functionalities offered by the system is now the subject of current and planned work. The two possible roles of this computer understanding system are being investigated: 1) to enable a direct human-system interaction, 2) to assist human-to-human information exchange (text messages are captured by the system and processed automatically with feedback to the human participants). Other application domains (e.g. tourism) are being investigated (e.g. within the project CITTA for the City of Poznań, which started in 2009).

Acknowledgements Several people have contributed to the project. We address our acknowledgements to members of the regular academic staff (A. Dąbrowski, M. Nkollo, T. Obrębski, G. Vetulani), PhD students (A. Kaliska, P. Konieczka, M. Kubis, J. Osiński, P. Rzepecki, G. Taberski, A. Vetulani, J. Walkowska), students (M. Ćwiąkała, P. Kubacki, A. Kuczma, K. Silski, M. Tański, K. Witalewski, C. Wieloch W. Wojciechowska), external experts (PPBW5) (K. Andrzejczak, R. Błaszyk, J. Brzęcki, W. Ciepiela, W. Filipkowski, J. Jankowiak, D. Kucal, J. Gorczyński, R. Pawłowski, J. Szymaniak, R. Talukder, J. Urbaniak) and other colleagues involved in administrative and technical tasks.

References 1. Kubis, M.: An access layer to PolNet in POLINT-112-SMS. In: Vetulani, Z. (ed.) Proceedings of the 4th Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, Poznań, Poland, November 6-8. Wyd. Poznańskie, Poznań (2009) 2. Kubis, M.: WQuery User Guide (2009), http://www.wquery.org/user-guide.pdf 5

Polish Platform for Homeland Security.

Natural Language Based Communication

313

3. Kubis, M.: Podsumowanie eksperymentu. Wiadomości SMS pisane przez policjantów podczas symulacji przeprowadzonej na stadionie Olimpii (in Polish). (Experiment summary. SMS messages obtained in a simulation experiment at Olimpia Stadium). Project Technical Report (23.06.2009) (2009) 4. Fairon, C., Paumier, S.: A translated corpus of 30,000 French SMS. In: Proceedings of LREC 2006, Genoa, Italy (2006) 5. Pala, K., Horák, A., Rambousek, A., Vetulani, Z., Konieczka, P., Marciniak, J., Obrębski, T., Rzepecki, P., Walkowska, J.: DEB Platform tools for effective development of WordNets in application to PolNet. In: Vetulani, Z. (ed.) Proceedings of the 3rd Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, Poznań, Poland, October 5-7, pp. 514–518. Wyd. Poznańskie, Poznań (2007) 6. Marciniak, J., Vetulani, Z.: Opracowanie metodologii i zaprojektowanie instrukcji dla zespołów testujących prototyp “beta” systemu POLINT-112-SMS wraz z opracowaniem wyników testowania wewnętrznego (in Polish). (Evaluation methodology and testing handout for the POLINT-112-SMS system. Initial testing results). Project Technical Report (23.12.2008) (2008) 7. Vetulani, G.: Rzeczowniki predykatywne języka polskiego. W kierunku syntaktycznego słownika rzeczowników predykatywnych na tle porównawczym (in Polish). (Predicate nouns of Polish. Towards a syntactic dictionary of predicative nouns. Comparative approach). Adam Mickiewicz University Press. Poznań (2000) 8. Vetulani, Z.: A system for Computer Understanding of Texts. In: Murawski, R., Pogonowski, J. (eds.) Euphony and Logos, Rodopi, Amsterdam-Atlanta. Poznań Studies in the Philosophy of the Sciences and the Humanities, vol. 57, pp. 387–416 (1997) 9. Vetulani, Z.: Electronic Language Resources for POLISH: POLEX, CEGLEX and GRAMLEX. In: Gavrilidou, M., et al. (eds.) Proceedings of Second International Conference on Language Resources and Evaluation, Athens, Greece, May 30-June 2, pp. 367– 374. ELRA (2000) 10. Vetulani, Z., Marciniak, J.: Corpus Based Methodology in the Study and Design of Systems with Emulated Linguistic Competence. In: Christodoulakis, D.N. (ed.) NLP 2000. LNCS (LNAI), vol. 1835, pp. 346–357. Springer, Heidelberg (2000) 11. Vetulani, Z., Obrębski, T., Vetulani, G.: Towards a Lexicon-Grammar of Polish: Extraction of Verbo-Nominal Collocations from Corpora. In: Proceedings of the Twentieth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2007), pp. 267–268. AAAI Press, Menlo Park (2007) 12. Vetulani, Z., Marciniak, J., Konieczka, P., Walkowska, J.: An SMS-based System Architecture (Logical Model) to Support Management of Information Exchange in Emergency Stuations. POLINT-112-SMS. In: Shi, Z., Mecier-Laurent, E., Lake, D. (eds.) Intelligent Information Processing IV (Book Series: IFIP International Federation for Information Processing, Subject collection: Computer Science), vol. 288, pp. 240–253. Springer, Boston (2008) ISSN 1571-5736, 1861-2288 (Online) 13. Vetulani, Z., Walkowska, J., Obrębski, T., Marciniak, J., Konieczka, P., Rzepecki, P.: An Algorithm for Building Lexical Semantic Network and Its Application to PolNet - Polish WordNet Project. In: Vetulani, Z., Uszkoreit, H. (eds.) LTC 2007. LNCS (LNAI), vol. 5603, pp. 369–381. Springer, Heidelberg (2009)

314

Z. Vetulani and J. Marciniak

14. Vetulani, Z., Obrębski, T.: Resources for Extending the PolNet-Polish WordNet with a Verbal Component. In: Bhattacharyya, P., Fellbaum, C., Vossen, P. (eds.) Proceedings of the 5th Global Wordnet Conference on Principles, Construction and Application of Multilingual Wordnets, pp. 325–330. Narosa Publishing House, New Delhi (2010) 15. Vossen, P.: Euro WordNet. General Document, Version 3. University of Amsterdam (2003) 16. Walkowska, J.: Dialogue Organization in Polint-112-SMS. In: Vetulani, Z. (ed.) Proceedings of the 4th Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, Poznań, Poland, November 6-8. Wyd. Poznańskie, Poznań (2009)

Dialogue Organization in Polint-112-SMS Justyna Walkowska Department of Computer Linguistics and Artiﬁcial Intelligence Faculty of Mathematics and Computer Science Adam Mickiewicz University ul. Umultowska 87, 61-614 Poznań, Poland [email protected] http://www.amu.edu.pl/~zlisi

Abstract. This paper describes the organization of dialogue in the Polint112-SMS system, a multi-user knowledge-management system with natural language interface. It presents the main goals of the system’s dialoguing strategy and the most important dialogue-oriented algorithms of diﬀerent modules of the system. The algorithms allow for eﬃcient handling of: questions asked by the system to obtain missing information, the need to pair diﬀerent types of answers and questions in the asynchronous mode of communication, diﬀerent semantic interpretations proposed by the parser, the occurrence of anaphora, and a few more. Keywords: human-machine dialogue, dialogue model, SMS corpus, knowledge management system.

1

Introduction

Polint-112-SMS is a multi-agent expert system with natural language (Polish) interface designed to be used by security oﬃcers overseeing a big public event, e.g. a football game in a stadium. The information is introduced and retrieved by means of SMS messages. This environment poses a number of problems interesting from the point of view of language and knowledge processing, such as: obtaining and interpreting a dedicated corpus [12], parsing and understanding the SMS sublanguage, creating an ontology suﬃcient to describe the situation in which the system operates [11], uniﬁcation of data introduced by diﬀerent users (informing agents), detecting contradictory information, question asking and answering suited to the limited yet challenging channel of SMS communication. This paper focuses on the dialogue organization and strategy in the system. It presents in detail the goals the system needs to obtain through dialogue, and also the problems and solutions found during the design and implementation process. Some dialogue-based tasks of the system are domain-speciﬁc (like ”zero” messages described in 3.2), others are more universal and may be applied in other dialoguing knowledge-management systems. The aim of this paper is to present solutions and algorithms developed during the creation of Polint-112-SMS. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 315–326, 2011. c Springer-Verlag Berlin Heidelberg 2011

316

2

J. Walkowska

System Architecture

The architecture of Polint-112-SMS has been fully described in [10]. Since then, the model has been extended with a new component - the visualization module. The ﬁnal system architecture consists of the following modules: 1. SMS-Gate: a Java module responsible for sending and receiving SMS messages and passing them to the input of the Text Analysis Module. 2. Text Analysis Module (TAM): an SWI-Prolog/Logtalk HTTP server that parses and marks up the text found in the messages and passes the obtained semantic objects (Logtalk is an object-oriented extension to Prolog) to the Dialogue Processing Module. The module is multi-threaded in order to make sure that a particularly complex sentence cannot block the system. The parser makes extensive use of the code of the original Arity/Prolog Polint system [7]. 3. Dialogue Maintenance Module (DMM): this module will be described in detail in the remaining part of the paper. It is implemented in SWIProlog/Logtalk. The goal of the module is to manage the dialoguing process and to mediate between two other modules: the TAM and the SAM (below). 4. Situation Analysis Module (SAM): this SWI-Prolog/Logtalk module manages the knowledge of the system and is responsible for the reasoning process (data uniﬁcation, detecting global situations, changing object states when new information arrives, retrieving and conﬁrming facts, detecting contradictions in information sent by diﬀerent users). Its inﬂuence on dialogue is explained further in this paper. 5. Time-Space Module (TSM): provides time and space reasoning for the SAM based on an XML map loaded at the start of the system. It is also implemented in SWI-Prolog. 6. Visualization Module (Vis): written in Adobe Flex and Java, the Vis communicates with the SAM and the TSM to obtain and display the current state of knowledge about the world kept in the system. A number of users (supervisors and operators) can connect to the Vis by means of a a web browser.

3 3.1

Dialogue Organization The ”Philosophy” of Dialogue in the System

Polint-112-SMS is a knowledge-management system with natural language interface dedicated to a speciﬁc domain of operation (public security during a mass event). The dialoguing tasks set ahead of the system diﬀer signiﬁcantly from those of a chatterbot. The system is not supposed to be able to answer to or understand all possible messages sent to its input. Instead, it has to be able to process all information from its domain. The dialoguing goals of the system can be deﬁned as follows:

Dialogue Organization in Polint-112-SMS

317

– to obtain information concerning the state of the supervised event, – to ask for important parts of information missing from the data provided by the users, – to connect answers to questions in the asynchronous mode of message exchanging, – to detect contradictions in the provided data and ask the responsible agents for clariﬁcation, – to complete the data with previously obtained pieces of information when the informing agent uses anaphora to make the text more compact, – to provide answers to users’ questions, – to react in situations when the user stops sending information, – to log the users in to the system, – to adapt to an user’s style of dialoguing (agents’ proﬁles), – to choose the best text interpretation when the TAM oﬀers more than one possibility. The solutions and algorithms used by the system to perform these tasks are described further in this section. 3.2

Responsibilities of the Dialogue Maintenance Module

Most of the dialogue-oriented tasks are performed by the DMM, the designated dialoguing module. However, sometimes the dialoguing tasks overlap with the neighbouring modules’ functionalities, so other modules are also mentioned in this section. Logging in to the System. Depending on the system settings, logging in may be required to send information to the system and/or to obtain information from the system. The DMM is responsible for logging the users in and out, and also for checking whether a given user (introducing or requesting information) is logged in if the settings require it. If a user is not logged in and the action they attempted is forbidden by the current settings, the DMM sends this user a message informing them of the logging requirement. The users log in with a preset password, their login always being the phone number. The DMM simply checks if the number and password match and informs the user (sending an SMS message) about the result of the logging attempt. ”Zero” Messages. ”Zero” messages handling is a domain-speciﬁc system functional requirement. ”Zero” messages can be turned oﬀ in the system’s implementation. It is assumed that a logged in informing agent should contact the system every set amount of time, even if they do not need to send or obtain information, as the lack of messages from an agent may be interpreted as a sign of problems (the agent lost the phone or has been engaged in a dangerous event). For this reason the users are supposed to sent ”zero” messages when the situation is under control. If they fail to do so, the DMM’s checking thread sends them reminders and, possibly, informs the chosen ﬁeld leader.

318

J. Walkowska

Asking for Missing Information. Asking for missing information is one of the most important tasks of the DMM. The incoming messages are parsed by the TAM, which has access to PolNet (a Polish WordNet ontology ([3], [11]), into the form of objects resembling extended semantic frames [4]. There are some types of messages that are passed to the DMM in a diﬀerent way (answers such as ”yes”, ”no” or ”I don’t know”, short answers that might be proper nouns) those types will be described later. The top-hierarchy frame classes processed by the system are event, person, group, article and place. The classes have attributes that can contain atomic values (strings, numbers) or objects of other classes. Every attribute of every class has a set priority (priorities can be reset online, during the operation of the system). When an object appears at the DMM input, the DMM tries to merge it with existing objects and ﬁll it with the information that can be deduced from the current knowledge state (both methods are described below), and then checks it for empty attribute slots. The DMM excludes the slots for which the given user (sender of the data being processed) has already been asked but has not answered or admitted to lack of knowledge on the matter. If there are no such slots, the object is simply passed on to the SAM. Otherwise the highest-priority empty slot is determined. The priority of nested objects’ slots is determined by multiplication. As a quick example, let us assume that an object event has an attribute performer, with the priority value of 0.8, and in this attribute another object is nested. The nested object has an attribute name, also with the priority of 0.8. If the attribute performer is ﬁlled, but the attribute name is not, then the resulting priority of the missing information is 0.8 * 0.8 = 0.64. After the highest-priority missing attribute is found, the DMM checks if the resulting priority is higher than a threshold value determined in system settings. If not, then the information is passed to the SAM as if nothing was missing. If so - a question text is generated and sent to the agent as an SMS message. The algorithm is written below in a more structured way. Algorithm 1. Decision about Asking for More Data 1. Try to complete the incoming object with information already known to the system. 2. Check if the object contains empty slots about which the user has not yet been asked. If no, go to 6. 3. Find the highest-priority empty attribute slot (still excluding the attributes about which the user has been asked). 4. Check whether the priority of the slot is higher than the threshold value. If no, go to 6. 5. Formulate the question, remember it and send it to the agent. 6. Pass the received object to the SAM. Finding Answers to Previously Asked Questions. The messages are exchanged asynchronously. This means that when the system asks an agent a question, it does not stop to wait for the answer. It is not obligatory for the

Dialogue Organization in Polint-112-SMS

319

agent to answer every question. They may not know the answer and decide that it is not worth to write a message to inform the system about it. They may have already started typing the next message which will be sent before even reading the one containing the question. The asynchronicity allows for more natural communication, but poses diﬃcult tasks: the system has to be able to pair the incoming answers with the questions. The pairing operation is one of the key responsibilities of the DMM. Every formulated question is remembered in the DMM as an object with additional data such as: the users’s phone number, the time of sending the question, the number of messages that were exchanged since that time, the path to the attribute within the data object that was the subject of the question, information about whether the agent has already answered the question and whether the question is a ”yes/no” question. The ”simplest” case is that of ”yes/no” questions. Algorithm 2. Processing ”yes/no” Answers 1. Find the latest unanswered ”yes/no” question. 2. If too little time has passed between the current moment and the moment of asking the question (and so the agent could not have answered that quickly) discard the question and go to 1. 3. If there is no such question, give up. All known problems resulting from using this algorithm are discussed in 4.2. Algorithm 3. Processing ”I don’t know” Answers 1. The case with ”I don’t know” answers is very similar to algorithm 2, but all questions (and not only ”yes/no” ones) are taken into consideration in the ﬁrst step. The remaining group of questions and answers are regular value questions, asked when an attribute needs to be ﬁlled. The applied algorithm depends on the type of incoming information. If the incoming information is a proper syntactic-frame-like object, algorithm 4 is used. Algorithm 4. Pairing Value Answers with Their Questions 1. Retrieve all unanswered questions sent to the given agent. 2. Remove from the list questions that were asked in a moment too close to the current moment (where it is not possible to send an SMS message back so quickly). 3. Test if it is possible to merge (without any data conﬂicts) each question object with the incoming object. Remove questions that cannot be merged. 4. For the remaining question objects, check if after performing the merging (without actually performing it at this point!) the attribute that was the subject of the question will be ﬁlled. Remove questions that do not comply to this rule.

320

J. Walkowska

5. If there is more than one question left, choose the latest question. 6. If a question has been chosen, merge it with the incoming object, mark the question as answered, and pass the resulting object to the SAM. If no question has been found, send the incoming object to the SAM as new information. If the incoming information is just a text-string (a proper name), the DMM applies algorithm 5. Algorithm 5. Accepting Proper Name Answers 1. Retrieve all unanswered questions sent to the given agent. 2. Leave in the list only questions about attributes known to contain proper names (surname, name of a stadium sector, etc.) or about attributes that might be represented by proper names.1 3. If there is more than one question left, choose the latest question. 4. If a question has been chosen: (a) If the question is a question for just the proper name, put the name as the correct attribute’s value. (b) If the question is a question for a whole object of a certain type (frame class), create the object, inputting the proper name value into the correct attribute. Than place the whole object into the proper attribute of the question object, mark the question as answered. (c) Pass the result to SAM and ﬁnish. 5. Otherwise ignore the incoming information. One more DMM’s mechanism dedicated to dealing with questions and answers is the cleaning thread. It checks the set question validity threshold and marks older questions as answered, assuming the answer is "I don’t’ know". Choosing the Correct Text Interpretation. There are cases when the TAM cannot unambiguously decide whether a single word (or a sequence of words that do not form a sentence together) should be treated as a proper name or as an object. A good example are nicknames (possibly of stadium hooligans known to the security forces) that are also common adjectives. They are quite common in the Polish language (”łysy”, meaning ”bald”, ”silny”, meaning ”strong”, and so on). Even if the agents are required to use correct spelling, and thus starting proper names with a capital letter, the problem remains, because a single word in an elliptic sentence is also capitalized. In such cases the TAM presents diﬀerent interpretations to the DMM. The assumption (based on corpora observation) is that very short sentences of this type almost always appear as an answer to previously asked questions. The DMM is prepared to receive multiple interpretations. It uses algorithm 6 to choose the most probable one. 1

The attributes that might be represented by proper names are attributes of a complex (object) type, which themselves contain a proper-name attribute that can be used as an ID. For example, the type of the expected answer to the question ”Where is Kowalski?” is person, but the person may be referred to by one of the proper-name attributes, e.g. the surname. In this situation special merging rules apply (an implicit person object has to be created).

Dialogue Organization in Polint-112-SMS

321

Algorithm 6. Choosing the Best Semantic Interpretation 1. If there is only one interpretation, accept it and ﬁnish. 2. From all the interpretations choose only those that can be answers to previously asked questions. 3. If more than one interpretations might me possible answers, choose the interpretation(s) possibly answering the latest question (based on algorithms 4 and 5). If only one interpretation is left, accept it and ﬁnish. 4. If more than one interpretation is a possible answer to the same, latest question, choose the ﬁrst one containing proper names (and not objects), accept it and ﬁnish. 5. If no interpretation has been accepted yet (none answers any questions), choose the ﬁrst one containing objects (and not proper names), accept it and ﬁnish. 6. If no interpretation has been accepted yet (none answers any questions, none contains any objects), accept the ﬁrst one and ﬁnish. The result of applying the algorithm is the choice of the most probable interpretation: either one answering a recently asked question or, if no interpretation complies to this rule, one providing the most new information. Finding Anaphora Targets. SMS messages are a very concise form of communication. The senders want to save both time and eﬀort spent on typing on a limited keyboard. One of the easiest ways of reducing sentence length is by using anaphora (particularly eﬀective in languages like Polish, in which you can skip the subject pronoun: Wszedł do sektora 3 means [He] entered sector 3). The DMM uses the following algorithm to ﬁnd the target of an anaphoric reference. The TAM marks up anaphoric references in objects, detecting them by the presence of pronouns and (not yet implemented) certain words, or by the lack of sentence subject. There are no articles in Polish, so it is not possible to use them to determine if an object (e.g. a person) has been mentioned before. In the above example the object representing the performer (the performer attribute of the change_of_loc event) will be marked up as a reference. Algorithm 7. Solving Anaphora 1. Retrieve from the object at the DMM’s input all objects marked up as references. Remember them as {r1 ,...,rn}, each2 with the object in which it was nested. If there are no such objects, ﬁnish. 2. A special rule holds for groups. If one of the frames in {r1 ,...,rn} represents a group and one of the following is true: 2

Normally n = 0 or n = 1. The maximum for one sentence found in the SMS corpus were two anaphoric references. Theoretically, more complex scenarios are possible, as in the sentence ’She gave him that’ (n = 3), but they did not appear in the corpus. One of the reasons may be that the users do not trust a computer system to eﬃciently cope with anaphora overload.

322

3.

4. 5. 6. 7.

J. Walkowska

(a) in this agent’s dialogue history there are new (e.g. modiﬁed no more than 10 minutes ago) frames with more than one performers {p1 ,...,pm }, (b) less than 10 minutes ago a message appeared that led to the creation of a number of frames representing animate entities {a1 ,...,ak }, then based on frames {p1 ,...,pn} or {a1 ,...,ak } create a fake group that will be considered during the anaphora resolution process. The original frames are remembered as members of the group. For each object being a reference: (a) Retrieve all previous objects from the agent’s session (that is all objects this agent introduced to DMM), (b) Rate the objects, rewarding them (giving them points) for: – time proximity, – type (frame class) concordance, – nesting concordance (both objects are nested in other objects, and those other objects have the same or similar type), – contained data similarity. Choose the object with the highest score. When there are more than one, choose the newest (most recently modiﬁed) object. If there are no older objects or it is not possible to merge the reference object with the best candidate, delete the reference markup and give up. Otherwise merge the reference object with its target and remove the reference markup. Delete the fake group. If the fake group is the chosen anaphora target to be merged with the incoming object, then merge the object with group members instead of the fake group.

The main assumption on which this algorithm is based (again, taken from corpus analysis) is that the agents use references when they are sure that ﬁnding their targets is easy, which mostly means that the reference targets have been mentioned quite recently. The fake group scenario allows for a correct decision in situations when the user describes an event in which a number of named individuals participated, and later refers to them as to a group. Informing Agents’ Profiles. During system operation the DMM is constantly updating agents’ proﬁles. A proﬁle contains information about the agent’s actions: how many messages they sent, how many questions they answered, how much time on average they needed to answer a question, how many times they provided information contradicting what another agent had said. Based on this information (that can also be saved to a ﬁle or loaded from a ﬁle) the agent’s reliability is determined. The reliability is important during the process of dealing with contradictory data (described further in this paper). Other types of information are used to better adjust the dialoguing strategy to the agent. For example, the DMM restrains from asking new questions if another question has already been asked and the agent’s average answer time has no yet passed.

Dialogue Organization in Polint-112-SMS

323

Data Unification/Merging. The important decisions about merging data are made in the SAM. The DMM performs very careful merging, using only data from within one agent’s dialogue session. Besides the algorithms described above in this paper, it will only merge two objects if they were sent by the same agent, they are of the same type, and no data is contradictory. When the DMM obtains very positive data (e.g. a surname), it may contact the SAM to ﬁll the object with more information in order to avoid asking unnecessary questions. 3.3

Dialogue-Oriented Features of the Situation Analysis Module

As has been stated above, some areas of dialoguing strategy require access to data that is kept outside of the DMM, so the DMM delegates parts of the dialogueoriented reasoning to other modules. Two important tasks are performed by the SAM: dealing with contradictory data and providing answers to users’ questions. Contradictory Data. It may happen that the system receives information that without doubt concerns the same people or events, but contains contradictory facts. Reasoning over contradictions lies in the responsibilities of the SAM. There are two possible contradiction scenarios: – the data comes from one agent, – the data comes from two diﬀerent agents. Algorithm 8. Dealing with Contradictory Data from Diﬀerent Users 1. Having detected a contradiction, the SAM saves both versions of the data, but sets the newer version as the current one. It informs the DMM about sending a contradiction question, so that the DMM can correctly understand the agents’ answers. 2. It sends messages to the agents responsible for the contradiction. They are presented both versions and are asked if they still conﬁrm theirs. 3. If: (a) one of the agents conﬁrms their version and the other doesn’t, the conﬁrmed version is set as the current one. (b) both agents conﬁrm their versions, the system chooses the version of the agent with higher reliability value. (c) neither of the agents conﬁrms, the newer version is set. Finding Answers to User Questions. As has been mentioned before, Polint112-SMS is a knowledge management system with natural language interface. This means that the system not only asks questions (which has been laid out in detail above), but also provides answers to questions asked by the agents. The TAM parses user questions to the form of frame objects (same as normal data objects in the system), indicating the attribute that has been the subject of the question. The DMM ﬁlls such objects when it is necessary (for example ﬁnding anaphora targets in questions such as ”Where is he now?”), but the real

324

J. Walkowska

work of ﬁnding objects in the system’s knowledge that can be the answer to the questions lies completely within the competence of the SAM, which applies a wide range of reasoning rules to ﬁnd the answer. An example of such rule: if the user asks for information about meeting of individuals A and B, the SAM checks information about all types of events in which both of them participated, and not only meetings. Finally if sends the obtained answer objects to the DMM to formulate text and send it to the asking agent.

4

Evaluation

At this point of the development process the system is fully operative, even though the TAM’s language coverage still needs to be extended. Over the past few months a series of ﬁeld experiments with real security oﬃcers has been conducted. After the experiments the users were given surveys in which they could describe their experience with the system. More information about the current version of Polint-112-SMS system may be found in [8]. 4.1

User Surveys

The surveys presented to the agents after the ﬁeld experiments contained questions about the number of messages the users would like to receive and are able to read and react to in a given amount of time. They were also asked to prioritize diﬀerent types of information they want the system to process. They were supposed to point out the most annoying system behaviours (e.g. too long messages, too many questions, mistakes in interpretation). They were invited to make any complaints about the system’s operation. The survey results proved that the least favourite behaviour of the system was asking to many questions about details unimportant from the point of view of the oﬃcers. It has already inﬂuenced the system’s development: the initial attribute priority settings were changed (mostly lowered). The number of questions asked by the system has been limited also by adding the mechanism preventing the system from asking new questions if the average agent answer delay time has not yet passed since asking the latest one. The users agreed that the system facilitates the process of collecting and retrieving information, especially about the more static situations and events. When something urgent occurs, there is often no time to write messages. Still, the Visualization Module makes it possible to very quickly assess the concentration of dangerous events. 4.2

Known Problems

There is a group of problems that have not yet been fully solved. One of them are the ”yes/no” messages. Even though algorithm 2 has been thoroughly tested and has proved correct in almost all cases, there are situations when the results are wrong. The DMM has no means of ﬁnding out that an agent did not read an SMS message containing a question, so cannot possibly be answering it. This is one possible reason for errors.

Dialogue Organization in Polint-112-SMS

325

A problem that has not yet been addressed properly is the fact that sometimes two diﬀerent meanings of a word can make sense in the system context (e.g. of an article an agent spotted in the stadium). At this moment the TAM keeps a word-sense priority list, but this decision should probably be made by the DMM, which might ask the agent for clariﬁcation, for example choosing a diﬀerent literal (word) from the same WordNet synonym set, especially when the DMM already connects to WordNet while checking data uniﬁcability. Then again: too many questions have been recognized as the most irritating system behavior. One more problem is the (extremely rare) situation on which in algorithm 8 (contradictory data) both informing agents are sure about their versions of data. The implemented solution is that the version of the more reliable user will be chosen. In practice, the reason for the diﬀerence in opinion may be the system’s prior wrong decision about merging - the users may in fact be describing two diﬀerent entitites. A demerge algorithm has been proposed in [8].

Acknowledgements This work has been supported by Polish Ministry of Science and Higher Education, grant R00 028 02 (within the Polish Platform for Homeland Security). Polint-112-SMS design and implementation team is comprised of Zygmunt Vetulani (chief), Piotr Kubacki, Marek Kubis, Jacek Marciniak, Tomasz Obrębski, Jędrzej Osiński, Justyna Walkowska, and Krzysztof Witalewski.

References 1. Dunin-Keplicz, B.: Towards better understanding of anaphora. In: Proceedings of 1st Conference of the European Chapter of ACL, E-ACL 1983, Pisa, pp. 139–143 (1983) 2. Fairon, C., Paumier, S.: A translated corpus of 30,000 French SMS. In: Proceedings of LREC 2006, Genova (2006) 3. Fellbaum, C.D.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998) 4. Fillmore, C.J.: Frame semantics. In: Linguistics in the Morning Calm, pp. 111–137. Hanshin Publishing Co., Seoul (1982) 5. Milward, D., Beveridge, M.: Ontology-based dialogue systems. In: 3rd Workshop on Knowledge and Reasoning in Practical Dialogue Systems (IJCAI 2003), pp. 9–18. MIT Press, Cambridge (2003) 6. Minker, W., Bennacef, S.: Speech and Human-Machine Dialogue. The Kluwer International Series in Engineering and Computer Science. Kluwer Academic Publishers, Massachusetts (2004) 7. Vetulani, Z.: Question answering system for Polish (POLINT) and its language resources. In: Proceedings of the Question Answering - Strategy and Resources Workshop, LREC 2002, Las Palmas de Gran Canaria (2002) 8. Vetulani, Z., Dąbrowski, A., Obrębski, T., Osiński, J., Kubacki, P., Kubis, M., Marciniak, J., Vetulani, G., Walkowska, J., Witalewski, K.: Zasoby językowe i technologie przetwarzania tekstu. POLINT-112-SMS jako przykład aplikacji z zakresu bezpieczeñstwa publicznego, Wydawnictwo Naukowe Uniwersytetu im. Adama Mickiewicza w Poznaniu (2010)

326

J. Walkowska

9. Vetulani, Z., Marciniak, J.: Corpus based methodology in the study and design of systems with emulated linguistic competence. In: Christodoulakis, D.N. (ed.) NLP 2000. LNCS (LNAI), vol. 1835, pp. 346–357. Springer, Heidelberg (2000) 10. Vetulani, Z., Marciniak, J., Konieczka, P., Walkowska, J.: An SMS-based system architecture (logical model) to support management of information exchange in emergency situations. POLINT-112-SMS project. In: Intelligent Information Processing IV, 5th IFIP International Conference on Intelligent Information Processing, Beijing, China, October 19-22, pp. 240–253. Springer, Boston (2008) 11. Vetulani, Z., Walkowska, J., Obrębski, T., Konieczka, P., Rzepecki, P., Marciniak, J.: PolNet - Polish WordNet project algorithm. In: Vetulani, Z., Uszkoreit, H. (eds.) LTC 2007. LNCS, vol. 5603, pp. 369–381. Springer, Heidelberg (2009) 12. Walkowska, J.: Gathering and Analysis of a Corpus of Polish SMS Dialogues. In: Challenging Problems of Science. Computer Science. Recent Advances in Intelligent Information Systems, pp. 145–157. Academic Publishing House EXIT, Warsaw (2009)

Valuable Language Resources and Applications Supporting the Use of Basque I˜ naki Alegria, Maxux Aranzabe, Xabier Arregi, Xabier Artola, Arantza D´ıaz de Ilarraza, Aingeru Mayor, and Kepa Sarasola Ixa Group. University of the Basque Country [email protected] http://ixa.si.ehu.es

Abstract. We present some Language Technology applications and resources that have proven to be valuable tools to promote the use of Basque, a low density language. We also present the strategy we have followed for almost twenty years to develop those tools and derived applications as the top of an integrated environment of language resources, language tools and other applications. In our opinion, if Basque is now in a quite good position in Language Technology is because those guidelines have been followed. Keywords: Language resources, Language Technology applications, Strategy for Language Technology development.

1

Introduction

Basque is both a minority and a highly inﬂected language with free order of sentence constituents. Language Technology for Basque is thus both, a real need and a test bed for our strategy for developing language tools for Basque. Basque is an isolate language, and little is known about its origins. It is likely that an early form of the Basque language was already present in Western Europe before the arrival of the Indo-European languages. Basque is an agglutinative language, with a rich ﬂexional morphology. In fact for nouns, for example, at least 360 word forms are possible for each lemma. Each one of the grammar cases as absolutive, dative, associative has four diﬀerent suﬃxes to be added to the last word of the noun phrase. These four suﬃx variants correspond to undetermined, determined singular, determined plural and close determined plural. Basque is also an ergative-absolutive language. The subject of an intransitive verb is in the absolutive case (which is unmarked), and the same case is used for the direct object of a transitive verb. The subject of the transitive verb (that is, the agent) is marked diﬀerently, with the ergative case (shown by the suﬃx -k ). This also triggers main and auxiliary verbal agreement. The auxiliary verb, or periphrastic, which accompanies most main verbs, agrees not only with the subject, but with the direct object and the indirect object, if present. Among European languages, this polypersonal system (multiple Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 327–338, 2011. c Springer-Verlag Berlin Heidelberg 2011

328

I. Alegria et al.

verb agreement) is only found in Basque, some Caucasian languages, and Hungarian. The ergative-absolutive alignment is rare among European languages, but not worldwide. It remains alive but in last centuries Basque suﬀered continuous regression. The region in which Basque is spoken is smaller than what is known as the Basque Country, and the distribution of Basque speakers is not homogeneous there. The main reasons of this regression during centuries [5] were that Basque was not an oﬃcial language, that it was out of educational systems, out of media and out of industrial environments. Besides, the fact of being six diﬀerent dialects made diﬃcult the wide development of written Basque. However, after 1980, some of those features changed and many citizens and some local governments promote recovering of Basque Language. Today Basque holds co-oﬃcial language status in the Basque regions of Spain: the full autonomous community of the Basque Country and some parts of Navarre. However, Basque has no oﬃcial standing in the Northern Basque Country. In the past Basque was associated with lack of education, stigmatized as uneducated, rural, or holding low economic and power resources. There is not such an association today, Basque speakers do not diﬀer from Spanish or French monolinguals in any of these characteristics. Standard Basque, called Batua (uniﬁed) in Basque, was deﬁned by the Academy of Basque Language1 (Euskaltzaindia) in 1966. At present, the morphology is completely standardized, but the lexical standardization process is underway. Now Batua is the language model taught in most schools and used on the few media and oﬃcial papers published in Basque. We are around 700,000 Basque speakers, around 25% of the total population of the Basque Country, and we are not evenly distributed. But still the use of Basque in industry and especially in Information and Communication Technology is not widespread. A language that seeks to survive in the modern information society has to be present also in such ﬁeld and this requires language technology products. Basque as other minority languages has to make a great eﬀort to face this challenge [14,17].

2

Strategy to Develop HLT in Basque

IXA is a research group created in 1986 by 5 university lecturers in the Computer Science Faculty of the University of the Basque Country with the aim of laying foundations for research and development of NLP software mainly for Basque. We wanted to face the challenge of adapting Basque to language technology. Twenty three years later on, now IXA2 is a group composed by 28 computer scientists, 13 linguists and 2 research assistants It works in cooperation with more than 7 companies from Basque Country and 5 from abroad; it has been involved in the birth of two new spin-oﬀ companies; and there are several products of language technology we have built. 1 2

http://www.euskaltzaindia.net http://ixa.si.ehu.es

Valuable Language Resources and Applications for Basque

329

In recent years, several private companies and technology centers of the Basque Country have begun to get interested and to invest in this area. At the same time, more agents have come to be aware of the fact that collaboration is essential to the development of language technologies for minority languages. Fruits of this collaboration were the HIZKING21 project (2002-2005) and ANHITZ project (20062008). Both projects were accepted by the Government of the Basque Country as a new strategic research line called Language Info-Engineering. At the very beginning, twenty three years ago, our ﬁrst goal was to create just a translation system for Spanish-Basque, but after some preliminary works we realized that, being Basque so diﬀerent from their neighboring languages, instead of wasting our time in creating an ad hoc MT system with small accuracy, we had to invest our eﬀorts in creating basic tools and resources for Basque (morphological analyzer/generator, syntactic analyzers...) that could be used later on to build not just a more robust MT system but also any other language application. This thought was the seed to design our strategy to make progress in the adaptation of Basque to Language Technology. This way we could face up to the scarcity of the resources and tools, and could make possible the development in Language Technology for Basque at a reasonable and competitive rate. We presented an open proposal for making progress in Human Language Technology [1]. Anyway, the steps proposed did not correspond exactly with those observed in the history of the processing of English, because the high capacity and computational power of new computers allowed facing problems in a diﬀerent way. Our strategy may be described in two points: 1. Need of standardization of resources to be useful in diﬀerent researches, tools and applications 2. Need of incremental design and development of language foundations, tools, and applications in a parallel and coordinated way in order to get the best beneﬁt from them. Language foundations and research are essential to create any tool or application; but in the same way tools and applications will be very helpful in the research and improvement of language foundations. Following this, our steps on standardization of resources brought us to adopt TEI and XML standards and also to the deﬁnition of a methodology for corpus annotation [7]. In the same way, taking as reference our experience in incremental design and development of resources/tools, we propose four phases as a general strategy for language processing: 1. Foundations. Corpus I (collection of raw text without any tagging mark). Lexical database I (the ﬁrst version could be just a list of lemmas and aﬃxes). Machinereadable dictionaries. Morphological description.

330

I. Alegria et al.

2. Basic tools and applications. Morphological analyzer. Lemmatizer/tagger. Spelling checker and corrector (although in morphologically simple languages a word list could be enough, in Basque we can not take this approach). Speech processing at word level. Corpus II (word-forms are tagged with their part of speech and lemma). Lexical database II (lexical support for the construction of general applications, including part of speech and morphological information). Statistical tools for the treatment of corpus. 3. Advanced tools and applications. An environment for tool integration. Web crawler. A traditional search machine that integrates lemmatization and language identiﬁcation. Surface syntax. Corpus III (syntactically tagged text). Grammar and style checkers. Structured versions of dictionaries (they allow enhanced functionality not available for printed or raw electronic versions). Lexical database III (the previous version is enriched with multiword lexical units, semantic information). Integration of dictionaries in text editors. Lexical-semantic knowledge base. Creation of a concept taxonomy (e.g.: Wordnet). Word-sense disambiguation. Speech processing at sentence level. Computer Aided Language Learning (CALL) systems. 4. Multilingualism and general applications. Information retrieval and extraction. Question/Answering. RBMT and SMT Machine Translation System development and Translation aids (integrated use of multiple online dictionaries, translation of noun phrases and simple sentences). Corpus IV (semantically tagged, annotation of senses, argumentstructure of sentences). Extraction of information based on semantics. Anaphora resolution and study of discourse markers. We complete this strategy with some suggestions about what shouldn’t be done when working on the treatment of minority languages. a) Do not start developing applications if linguistic foundations are not deﬁned previously; we recommend following the above given order: foundations, tools and applications. b) When a new system has to be planned, do not create ad hoc lexical or syntactic resources; you should design those resources in a way that they could be easily extended to full coverage and reusable by any other tool or application. c) If you complete a new resource or tool, do not keep it to yourself; there are many researchers working on English, but only a few on each minority language; thus, the few results should be public and shared for research purposes, for it is desirable to avoid needless and costly repetition of work. There are other interesting works related to general policies to develop resources and applications for low-density languages [15,9].

3

Useful Applications and Resources

In this section we describe four eﬀective applications and four language resources already created by our group.

Valuable Language Resources and Applications for Basque

3.1

331

Spelling Checker/Corrector

Because the use of Basque was forbidden during many years in schools and also because of its late standardization3 , adult speakers nowadays did not learn it at school, and so they write it imperfectly. For example, when someone goes to write the word zuhaitza (tree), the many possible spellings (zuhaitz ? zugaitz ? zuhaitx ? zuhaitsa? sugatza?) may cause the writer to hesitate, often leading to an easy solution: Give up, and write the whole text in Spanish or French! The spelling checker Xuxen [2] is a very eﬀective tool in this kind of situation, giving people more conﬁdence in the text they are writing. In fact, this program is one of the most powerful tools in the ongoing standardization of Basque. The spelling checker is more complex than equivalent software for other languages, because most of those are based on recognizing each word in a list of possible words in the language. However, because of the rich morphology of Basque, it is diﬃcult to deﬁne such a list, and consequently, possible morphological analysis must be included. Xuxen is publicly available from http://www. euskara.euskadi.net, where there have been more than 20,000 downloads. There are versions for Oﬃce, OpenOﬃce, Mozilla, PC, Mac, and also an online web service (http://www.xuxen.com). The version for Oﬃce includes morphological analysis, but, what happens if we want to use the speller in the ”free world” (OpenOﬃce, Mozilla, emacs, LaTeX, ...)? ispell and similar tools (aspell, hunspell, myspell) are the usual mechanisms for these purposes, but they do not ﬁt with the two-level model [11] we have to use to be able to describe Basque morphology. In the absence of two-level morphology, our solution was to adapt the two-level description to hunspell in a (semi)automatic way. With the stems and two sets of suﬃxes, corresponding to the paradigms at ﬁrst and second level, which have been obtained all the information we needed for the hunspell description was ready. Only a format conversion was necessary for delivery the spelling checker/corrector for OpenOﬃce, and other tools integrating hunspell (http://www.euskara. euskadi.net) In addition, we also adapted the description to myspell that is useful for open source programs like Firefox that have not yet integrated hunspell (http://www.librezale.org/mozilla/firefox); to do this we combine the main paradigms (here with restricted generation power for each one) and the inclusion of the word forms appearing in a big corpus, after eliminating forms rejected by the original complete spelling checker. Although those approaches for the free world have lesser coverage for Basque morphology, they are very useful spelling checkers. As a reference of its use we can mention that more than 115.000 downloads4 have been done since 2007 for this Firefox add-on. 3 4

The academy of Basque defined the morphology and verbs of Unified Basque in 1966, but the lexical standardization process is still going on. http://addons.mozilla.org/en-US/firefox/addon/4020

332

I. Alegria et al.

Fig. 1. Lemmatization-based on-line dictionary consulting

3.2

Lemmatization-Based On-Line Dictionaries

The main product created for this kind of application is a plug-in for MS Word that enables looking up a word in several dictionaries; but, in order to make it more useful for a language like Basque with its rich morphology, the dictionary is enhanced with lemmatization. This means that morphological analysis is ﬁrst performed, and then possible lemmas of the word are matched with the dictionary. In the example shown in Fig. 1, the user asks for the meaning in Basque of the Spanish word cupi´eramos. That word-form cant be found in paper dictionaries because it is a ﬁnite verb form, but the application recognizes that it corresponds to the verb caber (Basque for to ﬁt), and shows ﬁve diﬀerent equivalents in Basque for that verb. At the moment this plug-in works with three dictionaries: Spanish-Basque, French-Spanish and a dictionary of synonyms. The Spanish-Basque version is publicly available in http://www.euskara.euskadi.net.

Valuable Language Resources and Applications for Basque

333

Fig. 2. Lemmatization based document search

3.3

Lemmatization-Based Search Machine

We have developed a search machine to be used with text documents.. This program ﬁrst performs morphological analysis of the word, and then searches relevant documents containing the lemmas corresponding to these possible morphological decompositions. In the example shown in Fig. 2 the user is searching in the Elhuyar5 science divulgation journal for documents related to the Basque word form saguarekin (with the mouse). The search machine looks for documents containing words whose lemma is just sagu (mouse): saguen, saguaren, sagua, saguetan... The principal search machines available nowadays do not have this ability; therefore, if you want to ﬁnd sagu (mouse), you will only ﬁnd occurrences of that exact word, or alternatively, when searching for any word beginning with that word (sagu*), many irrelevant documents will be found that contain words such as saguzar (bat) which do not correspond to the desired lemma. Consequently, lemmatization-based search machines give users better results. 3.4

Transfer-Based Machine Translation System

When we have faced a diﬃcult task such as Machine Translation into Basque, our strategy has worked well. In 2000, after years working on basic resources 5

http://www.zientzia.net

334

I. Alegria et al.

Fig. 3. Matxin MT system

and tools, we decided it was time to face the MT task. Our general strategy was more speciﬁcally deﬁned for Machine Translation, and we had in mind the following concepts: 1. Reusability of previous resources, especially lexical resources and morphology description. 2. Standardization and collaboration: at least, using a more general framework in collaboration with other groups working in NLP. 3. Open-source: this means that anyone having the necessary computational and linguistic skills will be able to adapt or enhance it to produce a new MT system, even for other pairs of related languages or other NLP applications. We have gotten good results in a short time by just reusing previous work, reusing other open-source tools, and developing only a few new modules in collaboration with other groups.6 In addition, we have produced new reusable tools and deﬁned suitable formats. We created Matxin using a transfer rule-based MT approach. It translates text from Spanish into Basque, and two results produced in the machine translation track are publicly available: 6

Opentrad project: http://www.opentrad.org

Valuable Language Resources and Applications for Basque

335

– http://matxin.sourceforge.net for the free code of the Spanish-Basque system and – http://www.opentrad.org for the online version. Now we are working in the construction of EBMT and SMT systems and a multiengine system including three subsystems based on diﬀerent approaches to MT: rule-based machine translation, statistical machine translation and examplebased machine translation [4,13]. 3.5

EDBL: Lexical Database for Basque

EDBL is a general-purpose lexical database of Basque, and so it constitutes an essential foundation for any task in the ﬁeld of automatic processing of the language. It was ﬁrst developed as lexical support for the spelling checker, but nowadays it constitutes also the basis for the lexical component of diﬀerent tools such as a morphological analyzer, a lemmatizer, a multiword lexical units’ recognizer, a named entities’ recognizer, and so on. It has proved to be a multipurpose resource, from which tailored lexicons can be exported. Following a mass enrichment process carried out two years ago, its content increased a 25%, thanks to the collaboration with UZEI7 and Elhuyar8, in the frame of a corpus project named The Observatory of the Lexicon9 and leaded by Euskaltzaindia, the Academy of the Basque Language. So, with the addition of new entries coming from the lexicographic databases of the above mentioned organizations, EDBL contains currently near 120,000 dictionary entries, more than 20,000 inﬂected forms (mostly verb ﬁnite forms) and about 700 non-independent morphemes, among others. Concerning the information stored in the database, a full-ﬂedged two-level morphology system for Basque is contained in it. Although EDBL contains mostly the general lexicon of standard Basque, lots of normative information are also stored in the database. EDBL contains many non-standard words and morphemes, such as dialectal forms, typical errors, etc., along with indications leading to their correct and standard use. The database is designed to gather other types of information of syntactic or semantic nature, such as subcategorization information, semantic features, etc., and it actually contains some of this kind of information in many cases. Currently, EDBL resides under the ORACLE DBMS, on UNIX, and it may be consulted via the Internet (http://ixa2.si.ehu.es/edbl). Exportations from the database are made into XML documents, that are useful to create speciﬁcally formatted lexicons. 3.6

BasWN: Basque WordNet

The Basque WordNet is a lexical knowledge base that structures word meanings around lexical-semantic relations. It follows the speciﬁcations of WordNet [10], 7 8 9

http://www.uzei.com http://www.elhuyar.org/ http://lexikoarenbehatokia.euskaltzaindia.net/cgi-bin/kontsulta.py

336

I. Alegria et al.

as well as on its multilingual counterparts EuroWordNet and the Multilingual Central Repository (MCR). The Basque WordNet has been constructed with the expand approach [16], which means that the English synsets have been enriched with Basque variants. Besides, we also incorporate new synsets that exist for Basque but not for English. Due to EuroWordNet and the MCR frameworks, the Basque WordNet is already linked to the Spanish, Catalan, English and Italian wordnets, and it can also be linked to any other wordnet tightly linked to the English WordNet. The contents can be viewed using an interface which directly accesses the Basque, Spanish, Catalan and English WordNets10 . It comprises 93.353 word senses and 59.948 words. Up to now, the Basque WordNet has been focused on general vocabulary leaving aside specialized language and terminology. Nowadays we are creating a new resource called WNTERM (from WordNet and Terminology) with the aim of enriching the Basque WordNet with terminological information. 3.7

EPEC: Syntactically Annotated Text Corpus

EPEC Corpus (Reference Corpus for the Processing of Basque) is a 300,000 word corpus of standard written Basque [3] which aim is to be a training corpus for the development and improvement of several NLP tools [8]. EPEC has been manually tagged at diﬀerent levels: morphosyntax, syntactic phrases, syntactic dependencies (BDT Basque Dependency Treebank) and WordNet word senses for nouns. In the course of the last few months, we started working to tag it with semantic roles. The ﬁrst version of this corpus (50,000 words) has already been used for the construction of some tools such as a morphological analyzer, a lemmatizer, or a shallow syntactic analyzer. This ﬁrst version is publicly available in two websites: – Ancora project11 . This corpus can be downloaded and consulted with a friendly graphic interface. – Natural Language Toolkit12 . 3.8

ZTC: Morphosyntactically Annotated Text Corpus

Today statistical tools for text processing are so powerful in language technology, that the number of words compiled and organized as text corpora could be used as a measure of the position of a language in the area. The ZTC corpus [6] has been built by compiling text on the subject of Science and Technology. A previous inventory of years 1990-2002 registered 20 million words on this subject. The ZTC corpus compiled 10 millions words of standard written Basque. All those words were automatically annotated, and up to 1.8 million were manually revised and disambiguated. A speciﬁc interface for 10 11 12

http://ixa2.si.ehu.es/mcr/wei.html http://clic.ub.edu/ancora http://www.nltk.org

Valuable Language Resources and Applications for Basque

337

advanced query of the corpus was also built. The result is a public resource: http://www.ZTcorpusa.net. The creation of this resource would have been impossible without reusing the lemmatizer. We built a new tool for corpus compilation and annotation. The massive use of the lemmatizer was necessary. The ZTC corpus is still far away from the size of the corpora for other languages; e.g., the BNC corpus13, that is becoming a standard corpus resource, has 100 million words. However, the ZTC corpus is a very useful resource for manual study of Basque, as well as for machine learning techniques.

4

Conclusions

A language that seeks to survive in the modern information society requires language technology products. ”Minority” languages have to do a great eﬀort to face this challenge. Ixa group has been working since 1986 in adapting Basque to language technology, having developed several applications that are eﬀective tools to promote the use of Basque. Now we are planning to deﬁne the BLARK for Basque [12]. From our experience we defend that research and development for less resourced languages should to be faced following this points: high standardization, open-source, reusing language foundations, tools, and applications, and incremental design and development of them. We know that any HLT project related with a less privileged language should follow those guidelines, but from our experience we know that in most cases they do not. We think that if Basque is now in an good position in HLT is because those guidelines have been applied even though when it was easier to deﬁne ”toy” resources and tools useful to get good short term academic results, but not reusable in future developments.

References 1. Aduriz, I., Agirre, E., Aldezabal, I., Alegria, I., Ansa, O., Arregi, X., Arriola, J.M., Artola, X., D´ıaz de Ilarraza, A., Ezeiza, N., Gojenola, K., Maritxalar, M., Oronoz, M., Sarasola, K., Soroa, A., Urizar, R.: A framework for the automatic processing of Basque. In: Proceedings of Workshop on Lexical Resources for Minority Languages (1998) 2. Aduriz, I., Alegria, I., Artola, X., Ezeiza, N., Sarasola, K., Urkia, M.: A spelling corrector for Basque based on morphology. Literary and Linguistic Computing 12(1), 31–38 (1997) 3. Aduriz, I., Aranzabe, M., Arriola, J.M., Atutxa, A., D´ıaz de Ilarraza, A., Ezeiza, N., Gojenola, K., Oronoz, M., Soroa, A., Urizar, R.: Methodology and steps towards the construction of epec, a corpus of written basque tagged at morphological and syntactic levels for the automatic processing. In: Archer, D., Rayson, P., Wilson, A., McEnery, T. (eds.) Proceedings of the Corpus Linguistics 2003 Conference, March 28-31, vol. 16 (1), pp. 10–11. Lancaster University, UK (2003) 13

http://www.natcorp.ox.ac.uk

338

I. Alegria et al.

4. Alegria, I., D´ıaz de Ilarraza, A., Labaka, G., Lersundi, M., Mayor, A., Sarasola, K.: Transfer-based MT from spanish into basque: Reusability, standardization and open source. In: Gelbukh, A. (ed.) CICLing 2007. LNCS, vol. 4394, pp. 374–384. Springer, Heidelberg (2007) 5. Amorrortu, E.: Bilingual Education in the Basque Country: Achievements and Challenges after Four Decades of Acquisition Planning. Journal of Iberian and Latin American Literary and Cultural Studies 2(2) (2002) 6. Areta, N., Gurrutxaga, A., Leturia, I., Alegria, I., Artola, X., D´ıaz de Ilarraza, A., Ezeiza, N., Sologaistoa, A.: ZT Corpus: Annotation and tools for Basque corpora. In: Corpus Linguistics, Birmingham (2007) 7. Artola, X., D´ıaz de Ilarraza, A., Soroa, A., Sologaistoa, A.: Dealing with Complex Linguistic Annotations within a Language Processing Framework. IEEE Transactions on Audio, Speech, and Language Processing 17(5), 904–915 (2009) 8. Bengoetxea, K., Gojenola, K.: Desarrollo de un analizador sintctico estadstico basado en dependencias para el euskera. Procesamiento del Lenguaje Natural 1(39), 5–12 (2007) 9. Borin, L.: Linguistic diversity in the information society. In: SALTMIL 2009 Workshop: IR-IE-LRL Information Retrieval and Information Extraction for Less Resourced Languages. University of the Basque Country (2009) 10. Fellbaum, C. (ed.): WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998) 11. Koskenniemi, K.: Two-level morphology: a general computational model for wordform recognition and production, University of Helsinki (1983) 12. Krauwer, S.: The Basic Language Resource Kit (BLARK) as the First Milestone for the Language Resources Roadmap. In: International Workshop Speech and Computer, Moscow, Russia (2003) 13. Labaka, G.: EUSMT: Incorporating Linguistic Information into SMT for a Morphologically Rich Language. Its use in SMT-RBMT-EBMT hybridation. PhD thesis. University of the Basque Country (UPV-EHU), Donosita, Basque Country (2010), http://ixa.si.ehu.es/lxa/Argitalpenak/Tesiak/1271852575/ publikoak/GorkaLabaka.Thesis.pdf 14. Petek, B.: Funding for research into human language technologies for less prevalent languages. In: Second International Conference on Language Resources and Evaluation (LREC 2000), Athens, Greece (2000) 15. Streiter, O., Scannell, K., Stuflesser, M.: Implementing nlp projects for noncentral languages: instructions for funding bodies, strategies for developers. Machine Translation 20(4), 267–289 (2006) 16. Vossen, P. (ed.): EuroWordNet: a multilingual database with lexical semantic networks. Kluwer Academic Publishers, Norwell (1998) ´ oinin, D., Petek, B.: Speech and Language Tech17. Williams, B., Sarasola, K., OCr´ nology for Minority Languages. In: Proceedings of Eurospeech 2001 (2001)

Clues to Compare Languages for Morphosyntactic Analysis: A Study Run on Parallel Corpora and Morphosyntactic Lexicons Helena Blancafort1,2 and Claude de Loupy1 1

Syllabs, 15, rue Jean-Baptiste Berlier, 75013 Paris, France 2 Universitat Pompeu Fabra, Roc Boronat, 138 08018 Barcelona, Spain

Abstract. The aim of the present work is to find clues on how to compare the difficulties of five languages for morphosyntactic analysis and the development of lexicographic resources running a corpora and lexical comparative study on multilingual parallel corpora and morphosyntactic lexicons. First, we ran some corpus-based experiments without any other type of knowledge, following classical measures used in lexical statistics. Then we carried out further experiments on the corpora using morphosyntactic lexicons. Finally, we plotted given diagrams using different clues to offer an overview of the difficulty of a language for the development of morphosyntactic resources. Keywords: morphosyntactic lexicons, multilingual resources, lexical statistics, language typology for NLP.

1 Introduction As the importance of multilingual processing grows, it is becoming urgent, for the development of linguistic resources for NLP, to better understand the differences between languages. To enhance the development of linguistic resources in several languages, we work on finding clues to first compare languages that we already know and then try to apply those clues on other languages. In the long run, our aim is to set up a roadmap that will construct lexicographic resources from scratch using only corpora and a little linguistic knowledge. To do so, the first step was to find clues to compare the difficulties of five languages (English, German, Spanish, French, and Italian) for morphosyntactic analysis and the development of lexicographic resources [4]. Here we have repeated the experiments run in [4] using another corpus (Europarl instead of JRC Corpus, because the JRC contained too many proper names and, therefore, was not useful for our experiment. Moreover, we have found other clues to compare languages using lexical statistics on corpora. Finally, we have assembled all the clues to generate some plots visualizing the differences between languages based on the clues. The obtained results show that it is possible to foresee the types of difficulties of a language for the development of lexicographic resources using the classical measures of vocabulary and bigram growth. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 339–350, 2011. © Springer-Verlag Berlin Heidelberg 2011

340

H. Blancafort and C. de Loupy

The present article is organized as follows: Section 2 briefly reviews the state of the art; Section 3 describes the resources used and discusses some problems. Next, we report on a series of corpus-based experiments run on parallel corpora with and without lexicons. Finally, we draw our conclusions and discuss further work.

2 State of the Art Traditional morphological typological studies [18]describe several linguistic phenomena for the comparison of languages, but do not provide any quantitative information about them. Experts in IR such as Pirkola [15] claim the need of knowing more about morphology for NLP. Pirkola suggests a language typology using the indexes of fusion and synthesis of traditional morphology to measure morphological phenomena, and gives interesting fine-grained indexes for distinguishing derivation, compounding and inflection. However, this typology is difficult to implement, as word boundaries are not clear-cut and so words cannot be segmented. [12] and [10] suggest automatic methods to compare languages and their proximity. [12] proposes a method to measure the proximity between languages that relies on the structure of the vocabularies of the languages considered by computing the Dice coefficient of the number of analogies between words, which are common, through translation, to two languages. They applied this measure to a multilingual lexicon of nine European languages compiled automatically from the parallel corpus JRC and computed a proximity matrix for these nine languages. This matrix corresponds in general with the knowledge about the relative proximity of these nine languages. On the other hand, [10] uses a file compression program as an analysis tool to compare the complexity of the 21 official EU languages on lexical, morphological and syntactic levels. This approach is capable of showing relationships between languages on these levels. Obtained results are mainly in accordance with linguistic descriptions of the languages. An interesting field for our research is lexical statistics, which deals with the study of frequency distributions of words and provides clues used for our research, such as the presence of hapax legomena [2], vocabulary size and token/type ratio, often applied to measure vocabulary richness and lexical variety in different fields as stylometry. A more sophisticated technique for measuring vocabulary richness is vocabulary growth, i.e., the development of the vocabulary V for increasing corpus size N. [2] suggested a model to measure vocabulary growth included in the open source zipfR library [6]. [7] ran lexical statistics based experiments on small manually PoS tagged corpora (about 2000 words) to compare languages for developing new languages from existing resources (Russian and Portuguese, using existing resources in Czech and Catalan respectively). They provide interesting clues as entropy to predict the corpus size needed for training the tagger. In [5], we report on an experiment to learn morphology automatically from raw corpora in several Romance and Germanic languages as well as the Slavic language Polish, and use the results to compare the complexity and richness of their morphology, especially regarding inflection and derivation. For this task, we used the opensource tool Linguistica from Goldsmith [8] for inducing morphology automatically. Inducing morphological information directly from corpora without previous resources

Clues to Compare Languages for Morphosyntactic Analysis

341

seems an interesting approach to compare languages and for building lexicographic resources. However, this automatic approach has some limitations, as it cannot handle correctly allomorphy or make the distinction between inflection and derivation processes. Nevertheless, unsupervised induction of morphology remains a promising field for building morphosyntactic lexicons from scratch with a resource-light approach.

3 Description of Resources As lexical studies on corpora are biased by the type of discourse represented in the corpus, we decided to use two different sets of corpora: Europarl [11] and the Bible [16]. To minimize the bias introduced by the lexicons, we used morphosyntactic lexicons with similar features. This is the case of MulText [9] but, as showed in [4], MulText lexicons score a lower coverage than the other lexicons used and present some incoherencies. This is why we only present results obtained with FreeLing lexicons v2.0 [1] and, as FreeLing is not available for French or German, with other large-coverage lexicons: the Lefff [17] for French, and Morphy [13] for German. The table below provides information about the size of the lexicons. Table 1. Size of the lexicons

Lang. en es it de fr

Lexicon FreeLing FreeLing FreeLing Morphy Lefff

Nb of lemmata 40,219 76,201 40,277 91,311 56,917

Nb of entries 67,213 668,816 437,399 4,055,789 472,582

4 Experiment and Analysis of the Results 4.1 Corpus-Based Study We used different indicators to compare the variability of the vocabulary and the syntax between languages: corpus and vocabulary size, token/type ratio, percentage of hapax legomena, behavior of bigrams, and vocabulary growth. In other words, we tried to find out which language has a greater variability (vocabulary and syntax) by just using parallel corpora. Greater variability could be seen as an indicator of the difficulty to process a language for NLP tasks, as it may imply more work to develop basic resources as lexicons and taggers, and could indicate which language has productive morphological processes such as composition and rich inflection. Thus, the probability of encountering unknown words will be greater for these languages and special processing will be necessary (e.g., compound splitting). 4.1.1 Corpus and Vocabulary Size As shown below, German has a more variable vocabulary than other languages. While in Europarl each token in English is repeated 122 times, in German a given

342

H. Blancafort and C. de Loupy Table 2. Vocabulary and corpus size, and token/type ratio

Corpus Lang. V size (types) ratio token type Corpus Language V size (types) ratio token type

de 26380 25

en 14679 56

de 90011 45

en 34634 122

Bible es 25238 33 Europarl es 58688 76

fr 21385 43

it 30498 28

fr 45832 102

it 54523 77

tokenonly appears 45 times. The more variable the vocabulary of a language, the more difficult it will be to reach a similar coverage for lexicons of the same size. 4.1.2 Presence of Rare Words: Hapax Legomena The percentage of hapax legomena (words occurring once) in each corpus provides additional information on the variability of the vocabulary. In consonance with the results concerning the token/type ratio presented above, experiments showed that German has the greatest percentage of hapax legomena, which means that there are more words used just once: 48% in German Europarl compared with around 35% in French and English in the same corpus. Baroni’s experiments [3] show that a corpus has usually 50% of hapax legomena. However, in our corpora, all languages with the exception of German are far below 50%. For example, in the English Bible, the percentage of hapax legomena is only 33%, which indicates that the vocabulary is quite repetitive. In Europarl, Spanish is the second language with the richest vocabulary, but the difference with Italian is minimal, however, in the Bible Italian is very close to German. In the Bible, Spanish has a much more varied vocabulary than English and French. Some questions arise, such as if translation type and quality might introduce a non-negligible bias and how to evaluate the parallelism and/or the comparability of a corpus. 4.1.3 Vocabulary Growth Vocabulary growth is another indicator of the variability of the vocabulary: the steeper the rise of the curve, the greater the vocabulary. Figure 1 presents the vocabulary growth of German (the higher curve), Spanish, Italian, French and English (the lower curve). We observe that German vocabulary increases faster than other languages. English increases more slowly, followed by French. The same results were obtained on the Bible corpus. Again, we observe that English and French had the least variable vocabulary. Italian has a different behavior depending on the corpus: in the Bible, Italian vocabulary grows much faster than Spanish and is closer to German. However, in Europarl, it is situated behind Spanish. The behavior of bigrams is slightly different (cf. Figure 2). Italian has a bigram vocabulary growth much more important than Spanish while English and French have a similar behavior, though English is far behind French for single words. This could be an indicator of a greater difficulty to construct language models for POS tagging or syntax analysis.

Clues to Compare Languages for Morphosyntactic Analysis

343

Fig. 1. Vocabulary growth of Europarl

Fig. 2. Vocabulary growth of bigrams in Europarl

4.1.4 Concluding Remarks on Corpora Statistics According to these results, we plotted a diagram for each language corresponding to the results obtained for the Bible using five clues. Our clues consider the following: vocabulary size (the number of different words), vocabulary growth (the slope of the curve at 100,000 tokens), hapax legomena percentage (number of hapax divided by the vocabulary size), token/type ratio (the average number of occurrences of a type) and bigram occurrences (the number of different bigrams). Each axis represents the value for each clue. The graphs are shaded in grey depending on the scored value: more grey expresses more difficulties to process the language. In order to normalize the scales, we used the highest value scored for each clue as a reference. Then we divided the score obtained by each language by the highest score used as a reference. For the clues concerning token/type ratio and bigram occurrences, we reversed the scale, as the lowest score stands for greater difficulty. These figures allow a quick analysis of the difficulties of a language. A large vocabulary size and a steep vocabulary growth slot, together with a low token/type ratio as well as a high number of hapax legomena, indicate that the language is more variable and thus, that a larger lexicon is required. Moreover, the lower the number of

344

H. Blancafort and C. de Loupy

de

en

vocabulary size

vocabulary size

vocabulary growth bigram occurrences

bigram occurrences

token/type ratio

bigram occurrences

token/type ratio

vocabulary growth

token/type ratio

hapax percentage

hapax percentage

es

fr

vocabulary size

vocabulary size

vocabulary growth bigram occurrences

hapax percentage

token/type ratio

vocabulary growth

hapax percentage

It vocabulary size

bigram occurrences

token/type ratio

vocabulary growth

hapax percentage

Fig. 3. Diagrams showing the degree of difficulty based on corpora statistics

bigram occurrences, the more difficult it is to learn the syntactic behavior of bigrams and the more difficult it should be to train a tagger, which implies that a larger corpus is necessary. Figure 3 shows that German is a difficult language to process, since it has the maximum score for almost all clues. English is the language with a lowest score for the five clues and thus, should be easier to process. Spanish seems to have a more variable vocabulary than French and English. French seems to be the easiest language to process after English. Italian is surprisingly almost as difficult to process as German. According to these diagrams, Italian should pose many problems for the creation of resources and for syntactic analysis. As mentioned earlier, these conclusions were unexpected, especially for Italian and for French. Moreover, those results also raise the question about the usefulness of

Clues to Compare Languages for Morphosyntactic Analysis

345

parallel corpora. Even if they seem to be adequate for comparing tasks, the bias introduced by translation might pose some problems. In the case of Italian, we wonder if the fact that it achieves results closer to German than to Romance languages is directly related to the quality or style of the translation. On the other hand, it seems strange to us that French shows one of the poorest vocabulary and syntax. 4.2 Using Morphosyntactic Lexicons Next, we tried to find out which other clues can be learned using corpora and morphosyntactic lexicons. 4.2.1 Dynamic and Static Coverage Dynamic coverage (dyn cvg) concerns the number of types covered by the lexicons, while static coverage (stat cvg) is based on tokens [14]. In the following table, the cells shaded in light grey represent the lowest coverage scores, while the ones shaded in dark grey highlight best results. Table 3. Dynamic and static coverage

Lexicon Lang.

Morphy de

Dyn cvg Stat cvg

0.39 0.89

Dyn cvg Stat cvg

0.58 0.87

FreeLing FreeLing en es Europarl 0.51 0.70 0.95 0.96 Bible 0.58 0.78 0.90 0.94

Lefff fr

FreeLing it

0.79 0.97

0.71 0.97

0.83 0.95

0.70 0.91

According to our results, in German it is more difficult to achieve a good coverage than in other languages. The Morphy lexicon scores a dynamic coverage of 0.39 in Europarl, as well as a static coverage of 0.89. The highest score for dynamic and static is in French (0.79 of dynamic coverage in Europarl). 4.2.2 Coverage Results and Vocabulary Growth If we go back to the vocabulary growth results described in subsection 4.1., where German had the steepest growth, it is not surprising that German gets the lowest coverage score, as it has a much richer vocabulary. Nevertheless, if we consider that the German Morphy lexicon is by far the largest lexicon with more than 90.000 lemmata and more than 4.000.000 forms, we can conclude that even a vast lexicon does not suffice to get satisfactory coverage results. As a result, we could conclude that German requires some special NLP processing for morphosyntactic analysis in addition to lexicographic information. It seems necessary to have some extra morphological analysis. When looking for differences between Romance languages, vocabulary growth results revealed that Spanish had a greater vocabulary than French. These might be confirmed by the coverage results. The Spanish FreeLing lexicon is larger han the French Lefff lexicon (76,201 to 56,917 lemmata), but the static and dynamic coverage

346

H. Blancafort and C. de Loupy

are lower (0.70 dynamic coverage in Spanish Europarl versus 0.79 dynamic coverage in French Europarl). However, this study is not sufficient to claim that Spanish has a more varied vocabulary than French, we are conscious that we need further tests on different corpora to confirm these results. 4.2.3 Length of Hapax Legomena Next, we wanted to find out whether hapax legomena are longer than other words, for this could be a clue of composition or agglutination. We observed that the length of hapax legomena increases in all languages with respect to the average length of the rest of the words in the vocabulary. In German, the number of words with more than 12 characters increases from more than 53% to 72%, almost 20 points, while the number of words with more than 20 characters increases from 8,1% to 17,6%. This could be an indicator of higher composionality in hapax legomena than in the rest of the vocabulary. It is coherent with the fact that composition in German is productive and not always lexicalized. 4.2.4 Comparing Morphosyntactic Ambiguities Concerning morphosyntactic ambiguity, Table 4 lists the average number of possible tags and PoS for a given form in the lexicon (sum of the number of possible tags for all the forms divided by the number of forms) as well as in Europarl corpus (sum of the number of possible tags for each occurrence according to the lexicon divided by the number of tokens – an unknown word is considered as a single tag). Table 4. Morphosyntactic ambiguity rates on Europarl

Morphy de in lexicon in corpus

12.9 8.0

in lexicon in corpus

1.4 1.7

FreeLing FreeLing en es Average tags per form: 1.5 1.1 2.4 1.9 Average PoS per form: 1.2 1.2 2.0 1.7

Lefff fr

FreeLing it

1.7 2.7

1.5 1.7

1.2 1.6

1.2 1.5

Clearly, German has the highest scores concerning morphosyntactic ambiguities. In addition to this, it can be observed that the average number of tags and PoS per form in corpus is lower than in the lexicon, i.e., there are more ambiguities in the lexicon than in the corpus. This behavior is unlike other languages, where ambiguity rates are higher in the corpus than in the lexicon. Moreover, it seems that in German the ambiguity is higher intracategorically than intercategorically, i.e., a form can have several tags that share the same PoS. This fact indicates that the language is highly inflected and that inflected forms are ambiguous. Furthermore, we should not disregard the bias introduced by the encoding philosophy of the lexicon. Morphy, e.g., shows greater ambiguity, partially due to the fact that adjectives have been nominalized and added to the lexicon as nouns by changing the case and the tags. This means that each adjective has at least two different tags

Clues to Compare Languages for Morphosyntactic Analysis

347

when we do not consider the case. Another observation is that Spanish seems to be the least ambiguous language, especially concerning the morphosyntactic ambiguities in the lexicon. French seems to be less ambiguous than German but much more ambiguous than Spanish. 4.2.5 Comparing Inflection Paradigms In this part, we compare the inflectional complexity of the languages by studying their paradigms. To compare the lexicons on the same basis, we generated new lexicons from the original ones to cover 80% of the tokens in each corpus. These lexicons were generated by extracting all the lemmata that could be associated with a given form (e.g., in French, portes is associated to the noun porte and the verb porter). Then we derived all the inflected forms associated with those lemmata. Finally, we automatically induced inflection paradigms from the obtained lexicon. The table below summarizes the number of paradigms per language and PoS category to cover 80% of the corpus and the number of inflections per paradim per language and PoS category, as well as the average number of forms per paradigms. To our surprise, in Spanish a smaller number of paradigms are needed than for other languages; depending on the corpus and lexicon used, Spanish is equivalent to English. We expected that only English would show a small number of paradigms. Moreoever, we can see that English requires a higher number of verbal paradigms than French and Spanish due to the large number of irregular verbs, but that theaverage number of inflected forms per category is very low. Table 5. Statistics on inflection paradigms (Bible corpus) Number of Paradigms Total Adj Noun Verb De En Es Fr It

529 90 98 107 108

20 13 11 20 15

155 15 34 37 33

354 62 53 50 60

Number of Inflections per Paradigm Average Adj Noun Verb

106 5 36 26 37

183 2 3 5 4

13 2 2 3 2

142 7 65 51 64

Yet another interesting observation concerns the complexity of inflection processes and how a lemma is modified to get an inflected form calculated by the number of deleted characters at the end of the lemma and the average length of the ending that is added. The average number of characters to be deleted in Spanish (1.7) is lower than in French (2.3) and in Italian (2.3), but English is still the language with the lowest average (0.5). Another interesting comparison was amonh the productivity of inflection paradigms, i.e., to evaluate how many lemmata are covered by a certain number of inflection paradigms. Thus, we evaluated the percentage of paradigms that are used once, that is, how many paradigms might not be very productive. In English, 90% of the adjectives are inflected with only three paradigms. French turned out to be the language with the most complex adjectival inflection: only 55% of the lemmata are inflected using the first three paradigms while more than 26% of the paradigms are just used just once.

348

H. Blancafort and C. de Loupy

4.2.6 Concluding Remarks on Lexicon and Corpus-Based Studies Finally, we used the results presented above to define clues to predict the cost of writing inflection rules and developing lexicons. Clues such as the number of inflections per paradigm and number of total paradigms, as well as the productivity of most frequent paradigms are valuable indicators to gather information about the complexity of the inflection system of each language and its impact on lexicon development. Ending length as well as the number of deleted letters are additional interesting indicators concerning the complexity of inflection processes. In addition, the percentage of ambiguous PoS per form in corpus, together with the percentage of ambiguous tags per form in the lexicon, provide valuable information about morphosyntactic ambiguities, contributing to the measurement of the complexity of the disambiguation task for the PoS tagger. We also included a clue concerning word length, i.e., the percentage of words with more than 20 characters that indicate a higher probability of compounding or complex derivation in a language. To normalize the scales, we used the same methodology as explained in Section 4.1. With all these clues, we plotted a diagram for each language corresponding to the results obtained for the Bible. de word length > 20 letters 3 most productive paradigms coverage

en word length > 20 letters

tags per form in lexicon

paradigms used only once

deleted letters

3 most productive paradigms coverage

PoS per form in corpus

paradigms used only once

nb of paradigms

deleted letters

inflections per paradigm

ending length

tags per form in lexicon PoS per form in corpus

nb of paradigms inflections per paradigm

ending length

nb of endings

nb of endings

es word length > 20 letters

word length > 20 letters

3 most productive paradigms coverage

fr

tags per form in lexicon

tags per form in lexicon

3 most productive paradigms coverage

paradigms used only once

PoS per form in corpus

paradigms used only once

PoS per form in corpus

deleted letters

nb of paradigms

deleted letters

nb of paradigms

inflections per paradigm

ending length

inflections per paradigm

ending length nb of endings

nb of endings it word length > 20 letters 3 most productive paradigms coverage

tags per form in lexicon

paradigms used only once

PoS per form in corpus

deleted letters

nb of paradigms inflections per paradigm

ending length nb of endings

Fig. 4. Diagrams showing the degree of difficulty based on lexicon and corpora statistics

Clues to Compare Languages for Morphosyntactic Analysis

349

5 Conclusions and Further Work In this work, we tried to find clues to compare the difficulties for morphosyntactic analysis for five languages using parallel corpora and morphosyntactic lexicons. The main findings are that lexical statistics run on parallel corpora provide several clues to evaluate the difficulty of building resources as well as clues to preview the complexity of syntactic behaviour. Clues based on vocabulary size, vocabulary growth, percentage of hapax legomena, and token/type ratio provide information about the variability of the vocabulary, while bigram occurrences are valuable in predicting syntax variability. Morphosyntactic lexicons run on parallel corpora are useful to calculate the cost of writing inflection rules and developing morphosyntactic lexicons. Statistics regarding inflection paradigms provide information about the complexity of the inflection system of a language and its impact on lexicon development. Moreover, the study of morphosyntactic ambiguities in corpus and lexicon enables the measure of the disambiguation task complexity for the PoS tagger. These results have to be validated with ther languages. Moreover, as noted before, we were confronted with several hurdles, especially the coherence and consistency of morphosyntactic lexicons and the comparability of parallel corpora. This is why our further research will focus on the evaluation of existing resources and the construction and use of multilingual comparable corpora. In addition, we will continue to explore how unsupervised induction of morphology techniques presented in the state of the art [5] can contribute to the development of lexicographic resources for morphosyntactic analysis using a resource-light approach. Acknowledgements. We would like to thank Toni Badia at Universitat Pompeu Fabra for his valuable comments and suggestions during the whole study presented here.

References 1. Atserias, J., Casas, B., Comelles, E., González, M., Padró, L., Padró, M.: FreeLing 1.3: Syntactic and semantic services in an open-source NLP library. In: Proceedings of LREC 2006. ELRA, Genoa (2006) 2. Baayen, H.: Word Frequency Distributions. Kluwer Academic Publishers, Dordrecht (2001) 3. Baroni, M.: Distributions in text. In: Lüdeling, A., Kytö, M. (eds.) Corpus linguistics: An International Handbook, vol. 2, pp. 803–821. Mouton de Gruyter (2009) 4. Blancafort, H., de Loupy, C.: Comparing languages from vocabulary growth to inflection paradigms: a study run on parallel corpora and multilingual lexicons. Procesamiento del lenguaje natural 41, 113–120 (2008) ISSN 1135-5948 5. Blancafort, H.: Learning Morphology of Romance, Germanic and Slavic Languages with the Tool Linguistica. In: LREC 2010, La Valetta, Malta (2010) 6. Evert, S., Baroni, M.: ZipfR: Working with words and other rare events in R. In: R User Conference (2006)

350

H. Blancafort and C. de Loupy

7. Feldman, A., Hana, J.: A resource-light approach to morpho-syntactic tagging. In: Mair, C., Meyer, C.F., Oostdijk, N. (eds.) Language and Computers. Studies in Practical Linguistics, vol. 70. Rodopi Press, Amsterdam (2010) 8. Goldsmith, J.: Unsupervised learning of the morphology of a natural language. Computational Linguistics 27(2), 153–198 (2001) 9. Ide, N., Véronis, J.: MULTEXT: Multilingual Text Tools and Corpora. In: Proceedings of the 15th International Conference on Computational Linguistics, COLING 1994, Kyoto, Japan, pp. 588–592 (1994) 10. Kettunen, K., Sadeniemi, M., Lindh-Knuutila, T., Honkela, T.: Analysis of EU languages through text compression. In: Salakoski, T., Ginter, F., Pyysalo, S., Pahikkala, T. (eds.) FinTAL 2006. LNCS (LNAI), vol. 4139, pp. 99–109. Springer, Heidelberg (2006) 11. Koehn, P.: Europarl: A Parallel Corpus for Statistical Machine Translation. MT Summit (2005) 12. Lepage,Y., Lardilleux, A, Gosme, J.: Commonality across vocabulary structures as an estimate of the proximity between languages. In: 4th Language & Technology Conference (LTC 2009), Poznań, Poland. (2009) 13. Lezius, W.: Morphy-German Morphology, Part-of-Speech Tagging and Applications. In: Heid, U., Evert, S., Lehmann, E., Rohrer, C. (eds.) Proceedings of the 9th EURALEX International Congress, Stuttgart, Germany, pp. 619–623 (2000) 14. Mérialdo, B.: Multilevel decoding for very-large-size-dictionary speech recognition. IBM Journal of Research and Development 32(2), 227–237 (1988) 15. Pirkola, A.: Morphological Typology of Languages for IR. Journal of Documentation 57, 330–348 (2001) 16. Resnik, P., Broman, O., Diab, M.: The Bible as a parallel corpus: Annotating the “Book of 2000 Tongues. Computers and the Humanities 33(1-2), 363–379 (1999) 17. Sagot, B., Clément, L., Villemonte de la Clergerie, E., Boullier, P.: The Lefff 2 syntactic lexicon for French: architecture, acquisition, use. In: Proceedings of LREC 2006 (2006) 18. Whaley, L.J.: Introduction to typology: the unity and diversity of language. Sage Publications, Thousand Oaks (1997)

Corpus Clouds - Facilitating Text Analysis by Means of Visualizations Chris Culy and Verena Lyding European Academy Bozen/Bolzano Viale Druso 1, 39100 Bolzano, Italy {christopher.culy,verena.lyding}@eurac.edu

Abstract. Large text corpora are a main language resource for the humandriven analysis of linguistic phenomena. With the ever increasing amount of data, it is vital to find ways to help people understand the data, and visualization techniques provide one way to do that. Corpus Clouds is a program which provides visualizations of different types of frequency information dynamically derived from a corpus via a standard query system, integrated with a standard KWIC display. We apply established principles from information visualization to provide dynamic, interactive representations of the query results. The selected design principles and alternatives to the implementation will be discussed and a preview on what other types of information connected to corpora can be visualized in similar ways are provided. Corpus Clouds can thus be seen as answer to the call by Collins et al. [1] to design in a principled way new visualization tools for linguistic data. Keywords: corpus linguistics, visualization.

1 Introduction People have been analyzing languages for thousands of years, and many tools have been developed to aid in linguistic analysis. The output of those tools has, until very recently, been some type of text, perhaps indexed, marked up, or formatted for input into a database. However, with the maturing of the field of information visualization ([2], [3]), the NLP community is starting to realize that linguistic data can be visualized in ways that aid the analyst in the task of making sense of what can be tremendous amounts of data ([4], [1]). In visualizing linguistic data for the analyst, we have several general goals: • • •

Enable faster understanding of the data, e.g. seeing things at a glance Enable better understanding of the data, e.g. seeing unexpected patterns Enable easier manipulation of the data representation, e.g. quickly alternating views

What all these goals have in common is improving how the linguist analyst interacts with the data generated by other computational tools. We can thus consider visualization to be another key element in the NLP tool chain, equally important for human Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 351–360, 2011. © Springer-Verlag Berlin Heidelberg 2011

352

C. Culy and V. Lyding

analysis, we would argue, as other tools such as taggers, parsers, and the like. The increasing amounts of data available are useful only to the extent that we can create new information from them. Visualization tools greatly facilitate that knowledge creation. However, there are still relatively few NLP applications that make use of the great potential of visualization techniques for data presentation and analysis. In particular, little research has been targeted to the use of visualizations in corpus query tools. Among the general projects on the visualization of linguistic information that have emerged over the past years, WordTrees [5], TileBars [6], and Arc Diagrams [7] are the ones most closely connected to corpus analysis tasks. WordTrees presents a technique for the visualization and interactive exploration of keyword in context lines as tree structures. TileBars visually presents document length and frequency of the query term together with its distribution across the text. Arc Diagrams are a visualization method for representing patterns of repetition in string data. Many other efforts are concerned with clustering of terms and documents (cf. e.g. [8]; [9]), often in connection with information retrieval tasks. There are other projects still, like DocuBurst [10], a method for visualizing document content based on lexical hierarchies from WordNet, a method for mapping document content to shape by Rohrer et al. [11], and many more. Some other recent efforts have incorporated a cloud-type representation for displaying corpus search results, including TAPoR [12] and MONK [13]. While interesting in their own right, these tools typically use relatively simple queries, and/or a small number of texts (often just one). In contrast, we are using a sophisticated corpus query engine (see below), and our testbed corpus contains over 60 million words. Cloud-type representations have become increasingly common in the intervening time since this paper was first presented, but rarely, if ever, with any of the considerations of Corpus Clouds. The work presented here applies standard principles from information visualization to the design of visualization of linguistic data (e.g. “overview first, zoom and filter, then details on demand” [14]). We use some common visualization techniques (e.g. graphs, sparklines [15]), re-purpose others (e.g. tag clouds), and combine them with standard corpus query presentations (e.g. keyword in context (KWIC)) to create a novel tool for corpus analysis.

2 Corpus Clouds 2.1 Corpus Query Tool The way users work with corpora is characterized by the “ability to use computers to find, sort, analyze and quantify linguistic features and processes in huge amounts of text” [16]. The primary tools for searching and analyzing corpora are corpus query engines such as Corpus Workbench [17], WordSmith Tools [18], SketchEngine [19], and online query systems like DWDS [20]. Corpus query tools mainly support two basic functionalities: the extraction of lists of words or word combinations (including lists of all words in the corpus, lists of n-grams, lists of key terms, lists of collocations) and the creation of concordances for specified search strings. Considering the characterization above, it is striking that the analysis step in corpus analysis is the

Corpus Clouds - Facilitating Text Analysis by Means of Visualizations

353

processing stage that is still least supported by current query tools – given that we don't consider outputting large numbers of text lines and lists of words as significant analysis support. Analysis is about adding meaning to data, or revealing characteristic and recurrent structures by making hidden information explicit. The concordancing task consists of two parts: the retrieval of results based on a query and the display of those results. The retrieval part is well supported in most query tools by powerful query languages for the retrieval of specific text examples (KWIC lines). In contrast the support for displaying the results is limited to basic operations like counting and sorting, expanding and collapsing the context of the search string, accessing the full text document and showing or hiding a restricted set of extra-textual information. Results sets are presented as simple text with little or no graphical markup. Due to the simple text format, differences between hits cannot be detected easily. 2.2 Corpus Inquiry Tasks and the Aim of Corpus Clouds Corpus Clouds aims at extending the power of concordancing functionalities towards better support for the analysis of query results. We claim that sophisticated visualizations and information supplemental to the immediate textual context are required to allow for a thorough understanding of the linguistic quality and relevance of information extracted from text corpora. Corpus Clouds brings together several levels of textual information, thus augmenting the explanatory power/ informative value of single hits (concordance lines) We furthermore stress the importance of presenting these information-rich concordance lines in an easy to comprehend form, to allow the user to gain a quick grasp of the data. The current version of Corpus Clouds focuses on frequency and distributional information derived from the corpus. The design of Corpus Clouds is driven by requirements derived from common tasks of language analysts. A small scale user study1 revealed that a large part of many corpus-based studies is the manual classification of often huge sets of search results. Orientation within large query results, and manual annotation and classification of results all require a good and quick understanding of the data, which, we believe, can be provided by integrating frequency information into KWIC lines, thus combining qualitative and quantitative information in one display. In addition, it turned out and is confirmed by the literature that exploratory phases, characterized by undirected corpus querying to get a feel for the data, are a common part of corpus based analysis [21]. When an analyst approaches a corpus for the first time, or starts a new inquiry, the first phase involves getting a sense of the corpus, and of the particular phenomena of interest. Two key aspects of the data that are relevant for exploratory search are frequency and distribution: the analyst is concerned with how frequent a phenomenon is and how that phenomenon is distributed across the corpus as a whole. For example when inspecting the sequence verb + preposition it might be helpful to the analyst to instantly get an understanding which prepositions the verb occurs most often with. Getting an indication of the distribution of a linguistic structure or a specific term over different text types or years is relevant to understand usage contexts or inspect language change. Summing up, the program is targeted at users who investigate 1

Qualitative interviews with 14 corpus linguists that were interviewed concerning their habits in the analysis of corpus data.

354

C. Culy and V. Lyding

linguistic phenomena on lexical and structural levels. In particular, Corpus Clouds is designed to facilitate exploratory search and the orientation in and analysis of large results sets by providing a quick understanding of the data. 2.3 Design Overview There are five parts of Corpus Clouds. [Fig. 1, clockwise from top left] 1. 2. 3. 4. 5.

The area for specifying the corpus query A distribution graph, showing the distribution of tokens and results over the corpus A results pane displaying all query results A KWIC display for a selected result type A pane showing the extended context for one KWIC line

The starting point is the specification of a corpus query (1 in Fig. 1). While we currently use the CQP corpus query engine [17], other query engines could be used to the extent that they provide the same types of information. The query area also provides further options concerning the filtering and visualization of the results. The results of a search, that is all the different strings matching the search query, are displayed according to the user specified options. While a typical corpus query tool by default displays the search results as a KWIC list, our default view is a “word cloud”, (3 in Fig. 1) modeled on tag clouds, and inspired by “Fast Faust” [22]. The key idea of the word cloud is to scale the size of words according to their frequencies. Over the last few years the cloud representation has become quite popular. While clouds on web sites are typically used for a small set of predefined tags or keywords, our tool dynamically creates clouds for results of corpus queries, scaled according to their frequencies in the corpus. Since frequencies of linguistic phenomena typically have a power law distribution (cf. Zipf's Law [23]), using raw frequency as the scaling factor usually leads to most terms being scaled too small to read. Therefore, the default scaling factor is the logarithm of the frequencies, with raw frequency as an option. Note that tag clouds do not have this scaling issue, since the number of tags is restricted to relatively few high frequency items. Another difference between a word cloud and a tag cloud is that the default order in a word cloud is descending frequency, while tag clouds are typically in alphabetical order, which is provided as an option in Corpus Clouds. Finally, a list view of the results is also available. All views of the frequencies provide instant access to the actual frequencies, either in-line in the list view, or on hovering in the word cloud view. The distribution panel (2 in Fig. 1) offers a more detailed view of the frequencies of the search results by simultaneously showing three aspects of distribution. The first aspect is the size (in number of tokens) of each sub-part of the corpus (in this case, by year, since the corpus is one of newspaper articles; other possibilities would be by author, genre, topic, etc.). The second aspect is the distribution of all of the search results (e.g. “Aussprache”, “Muttersprache”, etc. as in Fig. 1) across the sub-parts of the corpus. The final aspect of distribution is the distribution of an individual result selected in the word cloud (e.g. “Hochsprache” as in Fig. 1).

Corpus Clouds - Facilitating Text Analysis by Means of Visualizations

355

Fig. 1. Corpus Clouds Interface

By comparing these three distributions, the analyst can get a quick idea of whether a particular sub-part of the corpus deserves particular attention, and whether a particular result has a potentially interesting distribution. The distributions are shown as pie charts, histograms, or text tables, and are updated dynamically. Besides the word cloud view and connected distribution graphs, Corpus Clouds has a KWIC view (4 in Fig. 1), accessed by selecting a result in the cloud. When selecting a line of the KWIC display, fuller context is shown in the expanded context pane (5 in Fig. 1), a standard option provided by most corpus query tools. The primary innovation in our KWIC view is the optional incorporation of word frequency information into the KWIC via small bars indicating the order of magnitude of the frequencies of the words. This visualization allows for a quick detection of high frequency words (e.g. all function words, but also “deutsche”) and low frequency words (e.g. “Mundarten”). The visualization is toggleable and the technique is similar to the use of inline graphics in sparklines [15]. 2.4 Challenges with the Visualizations In choosing and implementing visualizations for Corpus Clouds we faced several partly theoretically, partly practically motivated challenges. 2.4.1 Validity of Tag Clouds There has been some discussion in the HCI literature about flaws with tag clouds, including misleading impressions of frequencies and difficulties with navigation ([24]; [25]), as well as discussions of alternatives [26]. However, in the case of

356

C. Culy and V. Lyding

Corpus Clouds, the purpose of the cloud is to give a quick impression, with actual numbers available on demand (unlike in tag clouds, where actual frequencies are not available). Navigation is not an issue, partly because the analyst is precisely interested in exploring by frequency, and partly because the analyst can quickly switch to an alphabetically ordered view. In other words, making the frequency view interactive and customizable addresses usability issues found in static tag clouds. 2.4.2 Clouds View and Complex/Large Search Patterns Currently the cloud view is optimal for searches that yield results of up to three to four words. Different representations would probably have to be provided for queries that return phrases or long sequences of words. A possible solution would be to show results by line. But user studies might also reveal that yet another representation is needed or that a scaled representation is of no use at all in this case. 2.4.3 Distribution Graphs Initially we chose to display the frequency distributions for the three groups (all tokens in the corpus, all results to a query and a specific query result) as histograms grouped by year. Our intent was to convey differences in the proportions among groups per year. Some “guerrilla” testing showed that the histogram display was confusing to most people, as they did not understand that the bars indicated percentages. We responded to this situation by providing pie charts as the default view of the distributions, as they are a common visualization for expressing percentages. The disadvantage with pie charts is that they make the comparison of distributions within years harder. Thus we kept the histogram view and let the user choose between the two options. 2.4.4 Word Sizes in Clouds View The cloud view adjusts the size of the results according to their frequencies. As there is only so much space in the cloud panel a maximum font size needs to be set. Calculating the font sizes according to the frequencies leads to very infrequent words getting too small to be readable, given the large magnitudes of the token frequencies. This is true even when using the logarithmic scale. To be able to stick to the correct scaling, the results that are too small to be readable are hidden and replaced by a notice of how many results are hidden. Providing the list view besides the cloud view enables the user to inspect the very infrequent results as well. 2.4.5 Word Sizes in KWIC View The scaling problem described above applies also to the KWIC view when incorporating small graphics to indicate word frequencies. The solution chosen for the clouds view is not appropriate in the KWIC view, as it would be odd to have “invisible” words in the KWIC line. Instead, we use vertical bars scaled to the log-frequencies of the words. Since readability is not an issue for the bars, they can be much smaller than a word and still be useful. While the issues of scaling words would be made simpler for our test corpus if we had a larger display area, allowing for a larger range of (readable) sizes, a larger display would not solve the problem in general, since we would soon be confronted with an even larger corpus overwhelming the display size again.

Corpus Clouds - Facilitating Text Analysis by Means of Visualizations

357

2.5 Some Design Principles A primary design principle of Corpus Clouds is to provide different views on the data simultaneously. Multiple types of information, including distribution, frequency, and the underlying data, are all available at once. Interconnections between data in different panels are visualized by coordinated color and highlighting. In the distribution panel color indicates years (pie chart view) or data group (histogram view). A second design feature is the common information visualization technique of providing an overview first (the word cloud) then zooming in (the KWIC view) and details on demand (the incorporation of frequency information into the KWIC view or the full context) [14]. Another fundamental design principle of Corpus Clouds is to give the analyst flexibility in the presentation of the data, since different investigations may be better served by different presentations. In addition, people have personal preferences as discussed above. All of the options are immediately available.

3 Evaluation and Future Work The current version of Corpus Clouds is a mature prototype of an interface to a corpus query tool with extended concordancing functionalities. Its design has benefitted from the guerilla testing mentioned above, as well as from feedback from a local user who used it extensively in project. Before proceeding further, we are planning to carry out user studies on selected features of the tool. Designed as a tool for the exploratory analysis of search results, its utility depends on how well it supports the completion of an analysis/exploratory task in terms of (1) the effectiveness and (2) the efficiency in carrying out a specified task. By combining user investigations with usage experiments three aspects of Corpus Clouds will be evaluated separately: 1.

The utility of general design principles we followed, that is having simultaneous different views on the data and being able to select what information to inspect more closely (zoom and details-on-demand) 2. The utility of individual components of Corpus Clouds, i.e., the distribution panel, the results cloud, the KWIC display with frequency bars, etc. 3. The utility of alternative visualization techniques applied to each component, for example for the distribution panel, pie charts vs. histograms or for the results panel, clouds view vs. list view

More concretely we aim at gaining insights into the following questions: • • •

What levels of textual information are particularly helpful (token level, sentence level, document level, etc.)? What types of linguistic information are particularly helpful (categorical information, positional information, structural information, etc.)? Does the integration of different information types and levels have an impact on the resulting interpretation of the data?

358

C. Culy and V. Lyding

• •

What additional information would have been needed (e.g. detailed semantic, syntactic, discourse, pragmatic information)? Are the provided navigation paths useful (e.g. from search term in context (KWIC) to search term in document)? What other paths would be useful?

Besides directions that will derive from the evaluation phase there are four paths for the future development of Corpus Clouds. First, we plan to try other types of visualizations of frequency information in the KWIC, and using other types of information, e.g. morpho-syntactic (part of speech), syntactic and semantic categories as well as positional information. Second, Corpus Clouds needs to be extended to be more general, e.g. automatically configuring the distributional information for sub corpora, more user customization, etc. Third, Corpus Clouds is meant primarily for exploratory search. As such, it should be integrated with a tool which is designed for more extensive, in-depth search. Fourth, Corpus Clouds is targeted to the linguist analyst. It could be evaluated to what degree it could be adapted for other target groups, like language learners or translators. An additional area for exploration is to what extent Corpus Clouds could become a visual component as opposed to a stand-alone application. There are already cloudtype visual components (e.g. via the Google Visualization API [27] ), but they do not typically have the flexibility of Corpus Clouds. Furthermore, they are, rightfully, designed to be a single visualization, while Corpus Clouds incorporates several visualizations. Perhaps a better question is how to make the visualizations that make up Corpus Clouds into components that could then be easily integrated.

4 Conclusion Corpus Clouds provides a first step towards using techniques from information visualization to facilitate the exploration, analysis and understanding of language data. By focusing on the visual presentation of language data it approaches an area that is often neglected in the NLP tool chain. Besides corpus query tools there are several NLPbased applications that output (often large amounts of) language data for postprocessing by the user (e.g. systems of information retrieval, translation aids, proofing tools). Corpus Clouds gives an indication of how visualization techniques can be turned into an analytical aid to the user. Finally, in this paper we have attempted to make explicit the motivation for our design choices, both in terms of user needs and the affordances of different representations of frequency information in the face of the power-law nature of that information. This kind of discussion, while common in the HCI literature, has not generally been seen in discussions of linguistic analysis programs. We believe that the field of linguistic visualization can only benefit from more such discussions.

References 1. Collins, C., Penn, G., Carpendale, S.: Interactive Visualization for Computational Linguistics. In: ACL 2008: HLT Tutorials (2008), http://www.cs.utoronto.ca/~ccollins/acl2008-vis.pdf 2. Card, S.K., Mackinlay, J., Shneiderman, B.: Readings in Information Visualization: Using Vision to Think. Academic Press, San Diego (1999)

Corpus Clouds - Facilitating Text Analysis by Means of Visualizations

359

3. Ware, C.: Information Visualization, 2nd edn. Perception for Design. Elsevier, Inc., San Francisco (2004) 4. Collins, C.: A Critical Review of Information Visualizations for Natural Language. PhD qualifying exam paper, University of Toronto (2005), http://www.cs.utoronto.ca/~ccollins/publications/docs/ depthPaper.pdf 5. Wattenberg, M., Viégas, F.B.: The Word Tree, an Interactive Visual Concordance. IEEE Trans. on Visualization and Computer Graphics 14(6), 1221–1228 (2008) 6. Hearst, M.A.: Tilebars: Visualization of Term Distribution Information in Full Text Information Access. In: CHI 1995, Denver, Colorado, pp. 56–66 (1995) 7. Wattenberg, M.: Arc Diagrams: Visualizing Structure in Strings. In: IEEE Symposium on Information Visualization, pp. 110–116. IEEE Computer Society Press, Washington (2002) 8. Widdows, D., Cederberg, S., Dorow, B.: Visualisation Techniques for Analysing Meaning. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2002. LNCS (LNAI), vol. 2448, pp. 107–114. Springer, Heidelberg (2002) 9. DeCamp, P., Frid-Jimenez, A., Guiness, J., Roy, D.: Gist Icons: Seeing Meaning in Large Bodies of Literature. In: IEEE Symposium on Information Visualization. IEEE Computer Society Press, Washington (2005) 10. Collins, C.: Docuburst: Radial Space-filling Visualization of Document Content. Technical Report KMDI-TR-2007-1, Knowledge Media Design Institute, University of Toronto (2007) 11. Rohrer, R.M., Sibert, J.L., Ebert, D.S.: The Shape of Shakespeare: Visualizing Text Using Implicit Surfaces. In: IEEE Symposium on Information Visualization, pp. 121–129. IEEE Computer Society Press, Washington (1998) 12. TAPoR, http://portal.tapor.ca/ 13. MONK, http://www.monkproject.org/ 14. Shneiderman, B.: The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations. In: IEEE Symposium on Visual Languages, pp. 336–343. IEEE Computer Society Press, Washington (1996) 15. Tufte, E.: Beautiful Evidence. Graphics Press, Cheshire (2006) 16. Kennedy, G.: An Introduction to Corpus Linguistics. Longman, London (1998) 17. Christ, O.: A Modular and Flexible Architecture for an Integrated Corpus Query System. In: 3rd Conference on Computational Lexicography and Text Research, Budapest, pp. 23–32 (1994) 18. Scott, M.: Developing WordSmith. In: Scott, M., Pérez-Paredes, P., Sánchez-Hernández, P. (eds.) Software-aided Analysis of Language, special issue of International Journal of English Studies, vol. 8(1), pp. 153–172 (2008) 19. Kilgarriff, A., Rychly, P., Smrz, P., Tugwell, D.: The Sketch Engine. In: EURALEX 2004, Lorient, pp. 105–116 (2004) 20. Sokirko, A.: DDC – A Search Engine for Linguistically Annotated Corpora. In: Dialogue (2003) 21. Lemnitzer, L., Zinsmeister, H.: Korpuslinguistik. Eine Einführung. Gunter Narr, Tübingen (2006) 22. Müller, B.: Fast Faust (2000), http://www.esono.com/boris/projects/faust/ 23. Zipf, G.K.: Human Behavior and the Principle of Least-effort. Addison-Wesley, Cambridge (1949)

360

C. Culy and V. Lyding

24. Hearst, M.A., Rosner, D.: Tag Clouds: Data Analysis Tool or Social Signaller? In: 41st Annual Hawaii international Conference on System Sciences, p. 160. IEEE Computer Society, Washington (2008) 25. Hassan-Montero, Y., Herrero-Solana, V.: Improving Tag-clouds as Visual Information Retrieval Interfaces. In: InSciT 2006, Mérida (2006) 26. Kaser, O., Lamire, D.: Tag-Cloud Drawing: Algorithms for Cloud Visualization. In: WWW 2007 Workshop on Tagging and Metadata for Social Information Organization, Banff, Alberta (2007) 27. Google Visualization API, http://code.google.com/apis/visualization/documentation/ gallery.html

Acquiring Bilingual Lexica from Keyword Listings Filip Grali´ nski1 , Krzysztof Jassem1 , and Roman Kurc2 1

Adam Mickiewicz University Faculty of Mathematics and Computer Science Umultowska 87, 61-614 Pozna´ n, Poland {filipg,jassem}@amu.edu.pl 2 Wroclaw University of Technology Faculty of Computer Science and Management Wybrze˙ze Wyspia´ nskiego 27, 50-370 Wroclaw, Poland [email protected]

Abstract. In this paper, we present a new method for acquiring bilingual dictionaries from on-line text corpora. The method merges rulebased techniques for obtaining dictionaries from structuralised data, such as paper dictionaries (in electronic form) or on-line glossaries, with methods used by aligning tools, such as GIZA. The basic idea is to search for anchor words such as abstract or keywords followed by their equivalents in another language. Text fragments that follow anchor words are likely to supply new entries for bilingual lexica.

1

Introduction

The origins of the idea to use large text corpora for ﬁnding translations of words cab be traced back to Weaver’s famous memorandum [10], where the author predicted that the meaning (and proper translation of a word) can be determined on the analysis of its context in a text – the longer the context, the less unambiguity. However, his ideas had not been fully put into practice until the breakthrough work of Brown et al [1]. Since then the idea of Statistical Machine Translation (SMT) has gradually overcome former ideas on automatic translation. The main assumption of SMT is that given a bilingual corpus of source texts and their translations (target texts) we may presume the most likely translation of a word (phrase) to be the word (phrase) that occurs most frequently in the similar position in the target text. This idea is often used in the creation of bilingual dictionaries, in particular domain-speciﬁc ones, when the spectrum of texts under consideration is limited to a certain topic. In order to obtain a corpus-based bilingual dictionary, three phases are executed: i) collection of bilingual corpora, ii) paragraph and sentence alignment, iii) word alignment — to obtain mappings between words, i.e. a raw bilingual dictionary. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 361–370, 2011. c Springer-Verlag Berlin Heidelberg 2011

362

F. Grali´ nski, K. Jassem, and R. Kurc

Each step poses speciﬁc issues. i) Resnik [8] as well as Xiaoyi and Liberman [11] suggested algorithms for ﬁnding parallel texts on the Internet. The papers report very good results for pairs of languages such as English and German (the latter reports 97% recall and 99% precision). However, the same experiments carried out for a language pair less frequently represented on the Internet [9]. ii) There exist tools for sentence alignment such as described in [5], for which the precision over 95% was reported. The up-to-date open source implementation of the Moore aligner has been described in [3]. However, all algorithms give signiﬁcantly worse results when applied to texts that are not well structured (which is often the case with texts acquired from the Internet). iii) The most widely used tool for word alignment is GIZA and its successors [7]. The main problem is that GIZA, as a purely statistical tool, requires a corpus suﬃciently large to yield a reliable alignment. SMT is rather robust as far as word-alignment noise is concerned [4], whereas lexicon acquisition is more demanding. The combination of three imprecise tools for each of the three steps makes the SMT approach not fully satisfactory if used for the creation of bilingual dictionaries. This is why other corpus-based methods are searched for. The usage of monolingual domain-speciﬁc resources and the Internet was described in [6]. For a given word they look for multiword terms including the word that occur most frequently in the collected monolingual corpus. Next, they search Internet target texts for the same multiword terms – it is assumed that target language texts may have source language insertions. The translation of the given word is the most frequent meaningful word occurring in the target texts that include multiword terms. The idea would not give high recall for generaldomain words but it works well for domain-speciﬁc texts and specialised terms which often are cited in texts together with their translations. A method for acquiring bilingual dictionaries solely from monolingual web pages was presented in [2]. The authors of that paper observed that “many Chinese terms (e.g., named entities that are not stored in a conventional dictionary) are accompanied by their English translations in the Chinese web pages”. The method described in this paper is similar to that of Cao: the algorithm works on a single document rather than on a pair of documents. The document may be only partly bilingual, for example a scientiﬁc paper might be written in Polish with only the abstract and the list of keywords given in Polish and English. In order to ﬁnd the bilingual fragments we introduce a new factor for searching: the anchor. For the sake of this paper we assume the following deﬁnition: Definition 1. Anchor is a pair of words or multiword terms that indicates two positions in a text: the beginning of the text which should be rich in source terms; the beginning of the target text fragment which should contain their translations. For instance, the following anchors could be used for acquiring a Polish-English dictionary: {streszczenie, summary}, {streszczenie, abstract }, {abstrakt, abstract },

Acquiring Bilingual Lexica from Keyword Listings

363

{slowa kluczowe, keywords} {skladniki, ingredients}1 . In what follows, we present the idea implemented for the ﬁrst anchor ({slowa kluczowe, keywords}). The goal is to extract all terms and their translations that are deﬁned as keywords in a text (usually a scientific paper). Our method is intermediate between algorithms for extraction of terms from structuralised materials (like on-line lexica, glossaries) and algorithms for extraction of terms from raw, free texts. Note that keywords are usually listed in their basic form2 , which makes the extraction more reliable compared to that executed on raw texts. Furthermore, a phrase listed as a keyword is more likely to make sense as a lexicon entry (than a phrase extracted from a GIZA alignment). Another advantage of using terms deﬁned as keywords as dictionary entries is a relatively high reliability of translation, since keywords are usually used in scientiﬁc or reliable business articles. The techniques described in this paper show how the vast amount of text resources on the Internet makes it possible to acquire a useful amount of data (here: bilingual dictionaries) by using “tricky” methods. The paper is organised as follows: We start with some brief remarks on how the raw corpus was collected (Sect. 2). In Sect. 3, algorithms for extraction of bilingual dictionaries on the basis of anchors are presented. The evaluation by means of comparison to GIZA is given in Sect. 4 We end with conclusions in Sect. 5

2

Collecting the Corpus

A great advantage of using anchors is that they can be used not only for the extraction of desired text fragments in a given document, but also for seeking out the document itself on the Internet. In other words, documents with text fragments in question might represent a tiny percentage of the whole Web, however, if they are amassed from the Internet, a textual material of considerable value can be obtained. In order to collect the corpus of papers with bilingual keyword listings, we started with manually querying the Google search engine with queries like "s´lowa kluczowe" keywords abstract summary3. A tentative list of websites was obtained in this manner. Each of these websites was crawled by our in-house web robot. We obtained 20945 documents, mostly PDF ﬁles, containing the anchors. Not all of these documents contained a bilingual keyword listing (more on this in Sect. 4.2), however, some of them contained more than one keyword listing. 1 2

3

Slowa kluczowe, streszczenie, abstrakt, skladniki are simply Polish equivalents of respectively: keywords, summary, abstract, ingredients. With the exception of, for example, the names of chemicals, which are sometimes given in plural form in keyword listings. However, even in this case, the keywords are in the nominative, which is an important advantage considering the rich declension system of the Polish language. We used less frequently used synonyms of slowa kluczowe (e.g. hasla kluczowe, hasla indeksowe) as well.

364

3

F. Grali´ nski, K. Jassem, and R. Kurc

Procedure for the Keywords Extraction

The procedure for the keywords extraction consists of two phases: the scan of the document in search for keyword lists and the alignment of keywords in the lists found. The scan is executed by means of regular expressions with anchors. The alignment applies various heuristic techniques. The pseudo-code representation of the whole procedure is as follows: foreach document in documents do a. choose anchor schema; b. while (kwdsSRClist, kwdsTRGlist) = anchor schema.scan and split(document) do c. if correct(kwdsSRClist, kwdsTRGlist) then d. aligned pairs = align(kwdsSRClist, kwdsTRGlist); end end end A general run of the procedure will be shown by Example 1. Example 1. Piotrowo 3, 60-965 Pozna´ n, tel. (61)6652065 Slowa kluczowe: w´ ozek inwalidzki aktywny, wyb´ or w´ ozka inwalidzkiego, konﬁguracja w´ ozka inwalidzkiego Streszczenie W´ ozek inwalidzki z napedem recznym w du˙zym ... mo˙zliwo´sci socjalizacyjne osoby niepelnosprawnej ruchowo. 1 Chosen problems of manual wheelchair selection and conﬁguration Keywords: wheelchair for active use, wheelchair selection, wheelchair conﬁguration Abstract A manual wheelchair highly determines possibility of adaptation of a person with permanent dysfunction of locomotion for daily living. Selecting type of the wheelchair, geometrical sizes, adjustments regulations and choosing additional equipments determine regular choosing and selection of the wheelchair to individual user The paper characterizes wheelchairs for activ4 Re a.: The chosen anchor schema for the passage is: Slowa kluczowe: srckeyword delimiter src-keyword delimiter . . . other text Keywords: trg-keyword delimiter trg-keyword delimiter . . . Re b.: Both keyword blocks are cleaned and split into keyword lists. In this phase the source block Slowa kluczowe: w´ ozek inwalidzki aktywny, wyb´ or w´ ozka inwalidzkiego, konﬁguracja w´ ozka inwalidzkiego Streszczenie W´ ozek is converted into: kwdsSRClist = [ w´ ozek inwalidzki aktywny, wyb´ or w´ ozka inwalidzkiego, konﬁguracja w´ ozka inwalidzkiego ] Let us notice that the last two words of the block are removed in this step. Accordingly Keywords: wheelchair for active use, wheelchair selection, wheelchair conﬁguration Abstract A manual is converted into: 4

A fragment of a paper by Maciej Sydor and Marek Zablocki taken from http:// www.au.poznan.pl/sdwtd/sydor/sydor_zablocki_wybrane_probl_2006.pdf

Acquiring Bilingual Lexica from Keyword Listings

365

kwdsTRGlist = [ wheelchair for active use, wheelchair selection, wheelchair conﬁguration ] Re c.: At this point the structures of keyword lists are recognised. The goal is to delete ill-balanced list pairs, i.e. pairs of lists in which the number of elements diﬀer signiﬁcantly. This step will qualify the exemplary pair of lists as both of the lists have an equal number of elements (3). Re d.: The last step is the alignment of keywords. As the result of this step, lists of aligned pairs are obtained: aligned pairs = [ (w´ ozek inwalidzki aktywny, wheelchair for active use), (wyb´ or w´ ozka inwalidzkiego, wheelchair selection), (konﬁguracja w´ ozka inwalidzkiego, wheelchair conﬁguration)] Sections 3.1 and 3.2 focus on technical details of scan and alignment respectively. 3.1

Scan and Split

Two issues have to be considered here: 1. how to locate keywords; 2. how to extract anything but keywords, skipping unnecessary text before and after keyword blocks. We solve problem (1) by using regular expressions. Let us discuss two general patterns: Pattern 1 – Two lists are placed one after the other as in Example 1: src-anchor src-keyword delimiter src-keyword delimiter . . . some text . . . trg-anchor trg-keyword delimiter trg-keyword delimiter Pattern 2 – Each keyword is followed by its translation (e.g. written in brackets): src-anchor trg-anchor src-keyword trg-keyword delimiter srckeyword trg-keyword delimiter . . . Pattern 2, extracts keywords form text given as Example 2. Example 2. Slowa kluczowe (key words): kamica nerkowa (renal stone disease), szczawian wapnia (calcium oxalate), . . . The second issue (cleaning) is handled by two heuristic techniques: First, we set a maximum length of the keyword to 5, i.e. if a fragment between keyword separators (usually commas) is composed of more than 5 words then it is treated as a beginning of the non-keyword block. Second, we use regular expressions to delete unwanted strings from keywords, e.g. strings in braces or brackets or certain strings at the end of the keyword lists such as those starting with abstract or summary. In Example 3 a list of keywords is presented and the strings that are removed during cleaning are highlighted. Example 3. Keywords: wheelchair for active use, wheelchair selection (since 1990), wheelchair conﬁguration Abstract A manual Once the keywords are found and cleaned, they are split into lists.

366

3.2

F. Grali´ nski, K. Jassem, and R. Kurc

Alignment

In order to align elements of keyword listings we consider the following cases, reﬂecting the possible ways in which authors of scientiﬁc papers list and translate keywords: 1. At least one list is sorted (we assume that the author of a given paper wrote the keywords in alphabetical order on purpose5 and therefore there may be no correspondence between the order in which source language keywords were given and the order in which their translations were listed). 2. Both lists are unsorted and the lists are of diﬀerent length. 3. Both lists are unsorted and the lists are of the same length (presumably because the author of a given papers listed the translations of keywords in the same order as the original keywords). Each case is treated by in a diﬀerent way, which is illustrated by the following pseudo-code: 1.

2. 3.

align(kwdsSRClist,kwdsTRGlist):aligned pairs; begin if kwdsSRClist or kwdsTRGlist is sorted ascending then align similar(kwdsSRClist, kwdsTRGlist, true) else if length(kwdsSRClist) = length(kwdsTRGlist) then align similar(kwdsSRClist, kwdsTRGlist, false) else simple align(kwdsSRClist, kwdsTRGlist); end end end

In the ﬁrst and the second case (align similar with diﬀerent values of the third argument) the ﬁrst step of the alignment is the search for translation pair candidates. All elements of the ﬁrst list are compared with all elements of the second list. A pair {src-keyword, trg-keyword} is a translation candidate if one of the two conditions is met: – the keywords comprise of words that are equivalent according to an external bilingual lexicon (a very large, albeit noisy, lexicon of over 2.5 million translation pairs was used; words are lemmatised before the equivalence check); – the keywords are similar as far as their surface form is concerned, which is determined among others by the Levenshtein distance. Example 4 shows an exemplary linkage between Polish and English keywords. Example 4. [Bloto pochromowe, chrom {1}, chromian sodu {2}, ekstrakcja {3}, ﬁltracja {4}, kalcynacja {5}, lugowanie, metody otrzymywania chromianu sodu, obr´ obka surowc´ ow chromono´snych, recyrkulacja {6}, ruda chromowa {7} ]; 5

Obviously, in some cases keywords occur in alphabetical order just by accident.

Acquiring Bilingual Lexica from Keyword Listings

367

[Calcination {5}, chromic mud, chromic ore {7}, chromium {1}, extraction {3}, ﬁltration {4}, methods of sodium chromate production, processing of chromium materials, recirculation {6}, separation, sodium chromate {2}]; Links from {1} to {7} were detected by lexicon look-up. For example, translation pair chrom = chrome (link {1}) was simply found in the lexicon. Obtaining {7} required some more processing. Namely, the Polish word ruda is homonymous (it is either a feminine form of adjective rudy or the basic form of noun ruda) and the following translations can be found for the two interpretations of the word: 1. (adj. rudy) redhead ; rusty; red ; reddish-brown; foxy; ginger ;red-haired; rufous, ginger-haired, russet; ruddy; brownish-red ; bay 2. (noun ruda) ore; mineral ore The Polish word chromowy (the lemma for the feminine form chromowa) have a few equivalents as well: chromic, chromium, chrome, chromie. Similar type of information may be found in the English-to-Polish direction, where the word ore have a few Polish equivalents, such as: rudowy, rudny, kruszec, kopalina, ruda metalu, ruda. In order to ﬁnd the link between ruda chromowa and chromic ore all combinations of equivalents have to be processed. Once the translation candidate pairs are found, the alignment algorithm tries to determine other equivalents. If one of the keyword lists is sorted alphabetically, it is assumed that the order of elements gives no further clue for the alignment: thus only translation candidate pairs (found in the external lexicon) are aligned, leaving remaining elements unaligned (this is the case in Example 4). However, if exactly one pair of keywords remains unaligned, the algorithm assumes the keywords to be equivalent and aligns them as well. When keyword lists have diﬀerent lengths and both are unsorted, the procedure align similar aligns sublists of keywords. The algorithm looks for translation candidate pairs. If any two candidate pairs are separated by the same number of keywords, the algorithms aligns all the keywords between them, as shown in Example 5. Example 5. [przetwarzanie rozproszone {1}, uklady programowalne, FPGA {2}, kompresja obrazu {3}, sie´c w ukladzie {4}, sie´c NoC {5}, AVC {6}, VC-1 {7}]; [scattered processing {1}, FPGA {2}, video compression {3}, Network on Chip {4}, NoC network {5}, AVC {6}, VC-1 {7}] Let us assume that {1}, {2}, {3}, {6} and {7} are candidate pairs (i.e. obtained by lexicon look-up). There is a redundant keyword between the ﬁrst and the third pair. It is removed. {3} and {6} are translation candidates and keywords between them ({4} and {5}) form continuous sequences of the same length. Therefore we assume that they are parallel translations and we join them together. If the lists are of the same length and both of them are unsorted the algorithm assumes that they are parallel translations of each other. All elements of the list are paired according to their positions on the lists. This case produces the highest amount of new keywords. The precision of this lexically unsupervised case is surprisingly high (see Sect. 4).

368

4 4.1

F. Grali´ nski, K. Jassem, and R. Kurc

Evaluation Alignment with GIZA as a Baseline

GIZA [7] is a well-known tool for word alignment. We decided to use GIZA (IBM Model 4) on the same material on which our algorithm had worked, for comparison. Lists of keywords were organised as a parallel corpus. Since GIZA uses statistical mechanisms, we decided to add the entries of the external lexicon that were used in the procedure align similar (see Sect. 3.2) and/or abstracts that follow keywords to the parallel corpus, thus supplying GIZA with more data. (A parallel corpus of Polish and English abstracts/summaries is an interesting by-product of our experiment. We managed to extract 1656 abstract/summary pairs.) An in-house program for extracting Polish/English translation pairs from GIZA output ﬁles was used. 4.2

Recall from Documents

The recall from documents was designed to prove a high quality of the ﬁrst part of the algorithm: the scan. The results of the scan are the base for an alignment and thus for the comparison between our method of alignment and GIZA. In order to estimate the recall from documents we used documents that were rejected during the scan. Out of 20945 documents, 8513 were rejected. We identiﬁed which of them were rejected by mistake and the reason behind it. A sample of 385 rejected documents was inspected. In this way, we estimated the documentlevel recall (the ratio of the number of successfully scanned documents to the number of all documents containing parallel Polish/English keyword listings) to be 0.87%. Such an outcome is not surprising, since the texts were searched on the Internet using the elements of the anchors. Most of the rejections resulted from a malformed encoding, garbled text (e.g. the result of unsuccessful PDFto-text conversion), papers containing only keywords in one language, etc. Some mistakes came from the fact that we had missed some anchor patterns. 4.3

Precision

Our anchor algorithm found 17825 diﬀerent translation pairs (see Tab. 1), 14142 of which were obtained with the simple align procedure (i.e. without a lexicon look-up, see Sect. 3.2). In order to estimate the precision, we checked the number of translation pairs conﬁrmed in the lexicon (the same lexicon as used in the align similar procedure) – 4621 pairs were found in the lexicon. Then a sample of 390 unconﬁrmed translation pairs was manually checked, only 18 translation pairs were marked as incorrect. Consequently, the estimation of the overall precision of our method is 0.97%. 25730 translation pairs were obtained with GIZA run on keyword listings and appropriate lexicon entries (interestingly, adding summaries/abstracts reduces the number of obtained translation pairs). It is more than the number of pairs

Acquiring Bilingual Lexica from Keyword Listings

369

Table 1. Results of various methods of extraction of keyword pairs Anchors GIZA [abstracts] # of translation pairs 17825 15065 # of conﬁrmed pairs 4621 1423 % of conﬁrmed pairs 0.259 0.094 precision (estimated) 0.97 # of correct translation 17290 pairs (estimated)

GIZA [lexicon] 25730 2374 0.092 0.52 13380

GIZA [ab stracts+lexicon] 23866 1935 0.081 -

obtained with anchors, but the estimated precision (the same estimation procedure was used as for our method) is much lower (0.52%). It is quite obvious that the corpus (even with abstracts) processed by GIZA was too small to give reasonable results for statistical methods.

5

Conclusions

This paper shows a new technique of extracting translation pairs from Web texts. The method relies on the existence of special words (called anchors) that indicate fragment of texts likely to contain words together with their translations. The described method has been used for extracting keyword pairs from scientiﬁc papers. It turns out that such tailored, “tricky” techniques signiﬁcantly outperform standard statistical methods for speciﬁc data. Obviously, extracting keyword translations from scientiﬁc papers is not enough to create a complete lexicon, it may, however, make a valuable contribution to (or conﬁrmation of) lexical material obtained with other techniques.

Acknowledgment The paper is based on the research funded by the Polish Ministry of Science and Higher Education (Grant No 003/R/T00/2008/05).

References 1. Brown, P.F., Cocke, J., Pietra, S.D., Pietra, V.J.D., Jelinek, F., Laﬀerty, J.D., Mercer, R.L., Roossin, P.S.: A statistical approach to machine translation. Computational Linguistics 16(2), 79–85 (1990) 2. Cao, G., Gao, J., Nie, J.Y.: A system to mine large-scale bilingual dictionaries from monolingual web pages. Machine Translation Summit XI, 57–64 (2007) 3. Lipski, J.: Ur´ ownoleglanie tekst´ ow dwujezycznych na poziomie zdania. Master’s thesis, Adam Mickiewicz University in Pozna´ n (2007) 4. Lopez, A., Resnik, P.: Word-based alignment, phrase-based translation: What’s the link? In: Proceedings of the 7th Conference of the Association for Machine Translation in the Americas, pp. 90–99 (2006)

370

F. Grali´ nski, K. Jassem, and R. Kurc

5. Moore, R.C.: Fast and accurate sentence alignment of bilingual corpora. In: Stephen D, pp. 135–244. Springer, Heidelberg (2002) 6. Nazar, R., Wanner, L., Vivald, J.: Two step ﬂow in bilingual lexicon extraction from unrelated corpora. In: Proceedings of the EAMT(European Association for Machine Translation) 2008 Conference, Hamburg, Germany, September 22-23 (2008) 7. Och, F.J., Ney, H.: A systematic comparison of various statistical alignment models. Computational Linguistics 29(1), 19–51 (2003) 8. Resnik, P.: Parallel strands: A preliminary investigation into mining the web for bilingual text. In: Farwell, D., Gerber, L., Hovy, E. (eds.) AMTA 1998. LNCS (LNAI), vol. 1529, pp. 72–82. Springer, Heidelberg (1998) 9. Rosi´ nska, M.: Collecting Polish-German parallel corpora in the Internet. In: Proceedings of the International Multiconference on Computer Science and Information Technology, XXIII Autumn Meeting of Polish Information Processing Society, vol. 2 (2007) 10. Weaver, W.: Translation. In: Mimeographed, pp. 15–23. MIT Press, Cambridge (1949) 11. Xiaoyi, M., Liberman, M.: BITS. a method for bilingual text search over the Web. Machine Translation Summit VII, September 13 (1999)

Annotating Sanskrit Corpus: Adapting IL-POSTS Girish Nath Jha1, Madhav Gopal2, and Diwakar Mishra3 1,3

Special Center for Sanskrit Studies, J.N.U., New Delhi-110067 [email protected], [email protected] 2 Center of Linguistics, School of Language Literature & Culture Studies JNU, New Delhi-110067 [email protected]

Abstract. In this paper we present an experiment on the use of the hierarchical Indic Languages POS Tagset (IL-POSTS) (Baskaran et al 2008 a&b), developed by Microsoft Research India (MSRI) for tagging Indian languages, for annotating Sanskrit corpus. Sanskrit is a language with richer morphology and relatively free word-order. The authors have included and excluded certain tags according to the requirements of the Sanskrit data. A revision to the annotation guidelines done for IL-POSTS is also presented. The authors also present an experiment of training the tagger at MSRI and documenting the results. Keywords: POS Tagset, IL-POSTS, MSRI, EAGLES, hierarchical tagset, schema, type, category, attribute, value, Sanskrit, Pā ini, A ādhyāyī, subanta, ti anta, k danta, taddhita, samāsa, strī pratyaya, dhātupā ha, ga as, ātmanepadī, parasmaipadī, lakāra, Dravidian, Indo Aryan, vaidikī, laukikī.

1 Introduction Sanskrit, the oldest classical language of India, is also the oldest documented language of the Indo-European family. gveda (1500 BCE) is the oldest text of this family contains a sophisticated use of the pre-Pā inian variety also called vaidikī. Pā ini variously calls his mother tongue as bhā ā or laukikī. His grammar has two sets of rules – for vaidikī (variety used in the vedas) and for laukikī (variety used by the common people). The term ‘Sanskrit’ (meaning ‘refined’) is given to the standard form of laukikī (current language) which emerged after Pā ini’s grammar A ādhyāyī (AD) (700 BCE) (Jha et al 2007). Structurally Sanskrit is relatively free word-order and inflected language with amazing capacity to synthesize new sounds and morphemes at the word/sound junctures. Morphologically very elaborate, the rules of Sanskrit grammar are precisely done by Pā ini. However, these rules are not always easy to completely solve computationally (Jha et al 2009). The viability and usefulness of POS annotation in Sanskrit has been often argued that a good morphological analyzer is what you would need for it. But resolution of ambiguous labels in Sanskrit, as in other languages, cannot be done by morph analyzer alone. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 371–379, 2011. © Springer-Verlag Berlin Heidelberg 2011

372

G.N. Jha, M. Gopal, and D. Mishra

The relatively large computational linguistics community in India does not have a single standard for annotating linguistic data. There are in fact many ‘standards’. Five language families with diversity being a norm than exception, the feasibility of a single framework was considered un-attainable until 2008, when Microsoft Research India initiated a collaborative initiative in developing a common framework for Indian languages based on EAGLES guidelines. The framework which resulted is called ILPOSTS (Indic Languages-POS Tag Set). (Baskaran et al 2008 a&b). This framework has since been successfully tried on many Indian languages across many families. The present research is a report on adapting this framework on Sanskrit.

2 POS Tagging in Sanskrit While POS tagging is not a new research topic, it is, indeed, a new field as far as Sanskrit is concerned. The usefulness of annotated corpora for natural language analyses is well known. Unfortunately, so far, there have been no such annotated corpora available for Sanskrit. An even greater problem is the lack of training and testing data. The POS information is very important for language processing as it gives significant information about the word and its neighbours. For the major grammatical categories this is obviously true (i.e. verb pa hati versus participle pa hati: the verb pa hati expects a noun in nominative but a participle pa hati expects a noun in locative), but is also true for many finer distinctions. For example, the declension sub-tags (attributes) and gender-number sub-tags can help distinguish words in different categories (as in nara gacchanti the number tag in the verb gacchanti demands nara to be the nominative plural of the base n , and not the nominative plural of the word nara). By knowing the POS of a word we can tell what words with what POS are likely to occur in its vicinity. In linguistic items such complexity is found across languages. POS ambiguities in Sanskrit can be enormous. Chandrashekar (2007) has found eight types of ambiguity in the context of POS tag application. In Sanskrit, the nominal base (prātipadika) is inflected for multiple information based on the endcharacter, gender (li ga), vibhakti and number (vacana) information. The nominal forms (subanta-rūpas) having vibhakti markings -bhyām, -bhyas, -os are ambiguous in all endings and genders of a nominal bases (prātipadika). Sometimes different nominal bases have similar forms (for e.g., vibhava [1.1] when the prātipadika is vibhava / vibhava [1.3] when the prātipadika is vibhu). (Chandrashekar 2007). In a verb, the verb root, along with the optional prefixes information like pada (ātmane, parasmai), transitivity (karmatva), tense (kāla), mode (artha), voice (vācya), person (puru a) and number (vacana) are clubbed together. Adding to the complication, the nominal and verbal bases may be derived bases. Such being the complexity of Sanskrit morphology, there is ample scope for ambiguous word forms. The complexity of ambiguity in Sanskrit can be demonstrated using an example bhavati. It can be a verb or pronoun or a participle. It can still have many more ambiguous forms within the above mentioned categories, if we take inflectional features into consideration as well. Some forms of the third person pronoun bhavat in

Annotating Sanskrit Corpus: Adapting IL-POSTS

373

three genders can have similar forms as that of forms in the present participle of the verb root bhu. Though actual usage of the bhavati in the sense of participle and pronoun in neuter gender is often not seen, but grammaticality of the usage cannot be ruled out.

3 Sanskrit Morphology In Sanskrit, a syntactic unit is called pada. Cordona (1988) posits the formula for Sanskrit sentence (N-En)p…(V-Ev)p. A pada can be nominal (subanta) or verbal (ti anta). Padas with sup (nominal) inflections constitute the NPs (subanta-pada), and those with ti (verbal) can be called constituting the VPs (ti anta-pada). In the former, the bases are called prātipadikas which undergo sup affixations under specifically formulated conditions of case, gender, number, and also the end-characters of the bases to yield nominal syntactic words. The rules for subanta padas are found scattered in AD mostly in chapters 7-1, 7-2, 7-3, 6-1, 6-4. However, these rules have been treated in the subanta chapter of Siddhānta Kaumudī from rule number 177 to 446. (Jha 2004 b) The derivational morphology in Sanskrit studies primary forms (k danta) and secondary forms (taddhita), compounds (samāsa), feminine forms (strī pratyaya) etc. (Subash 2006). These can be inflected for 21 case (7 cases x 3 number) affixes to generate 21 inflected forms. The verb morphology (ti anta) is equally complex. Sanskrit has approximately 2014 verb roots including ka vādi according to Pā inian dhātupā ha classified in 10 ga as to undergo peculiar operations (Jha 2004 a), it can also be sub-classified in 12 derivational suffixes. A verb root conjugates for tense, mood, number and person information. Further, these can have ātmanepadī and parasmaipadī forms in 10 lakāras and 3x3 person and number combinations. There are 12 secondary suffixes added to verb roots to create new verb roots. A verb root may have approximately 2190 (tense, aspect, number etc.) morphological forms. Mishra and Jha (2005) have done a rough calculation of all potential verb forms in Sanskrit to be more than 1029,60,000.

4 MSRI Hierarchical Tagset Schema MSRI in collaboration with linguists and NLP experts has developed a common POS Tagset framework for Indian languages (especially for Indo-Aryan and Dravidian languages) following the hierarchical and decomposable tagset schema similar to that of EAGLES. This framework facilitates the sharing and reusability of scarce resources in Indian languages and ensures cross-linguistic compatibility. The rationale behind concentrating on Dravidian and Indo Aryan language families has been that of the 22 official languages in India a large majority is associated with these two language families.

374

G.N. Jha, M. Gopal, and D. Mishra

The Dravidian and IA language families have very different morpho-syntactic features at every level of linguistic analyses, but they also have a number of typological similarities that facilitate a common framework. Unlike flat tagsets, hierarchical tagset is structured making a hierarchy among categories. This implies that instead of having a large number of independent categories, a hierarchical tagset accommodates a small number of categories at the top level, each of which has a number of sub-categories in a tree structure. The associated morpho-syntactic features are packed in the different layers of hierarchy beginning from the major categories in the top and gradually progressing down to cover morpho-syntactic features for making it suitable to any Indian language, thereby keeping the framework a common standard across languages. The hierarchical tagset requires another feature called ‘decomposability’. It allows different features to be incorporated in a tag by separate sub-strings. Decomposable tags help in better corpus analysis (Leech, 1997) by allowing to search with an underspecified search string. The IL-POSTS This framework has a hierarchy at three levels: A. Categories are the highest level lexical classes. All categories are obligatory, i.e., are generally universal for all languages. B. Types are subclasses of categories and are assumed to be significant sub-classes common to a majority of languages. Some types may also be optional for certain languages. C. Attributes are morpho-syntactic features of types. Attribute tags contain the information like gender (male, female, neuter), number (singular, dual, and plural), case (nominative, accusative etc.), person (first, second, third) etc. All attributes are optional, though in some cases they may be recommended. The framework consists of 11 categories (including the punctuations and residual categories) that are recognized as universal categories for all ILs and hence, these are obligatory for any tagset derived from IL-POSTS. Barring punctuations, all categories have sub-classes called types which can have a number of attributes belonging to each of them. There are 17 attributes defined in the IL-POSTS framework. The attributes can be either binary or multi-valued.

5 Adaptations for Sanskrit For finalizing our tagset we have used the Sanskrit tagset by Chandrashekar (2007) and the Hindi specific tagset of MSRI. There are some changes in the tagset that we have adapted for Sanskrit data. These are at the category, subcategory, and attribute levels. In the subcategories of noun, we have only common and proper nouns. Verbal nouns have been clubbed under common nouns and spatio-temporal nouns under

Annotating Sanskrit Corpus: Adapting IL-POSTS

375

adverbs. Hindi and Sanskrit nouns do not have similar attributes. The verb has only finite form in Sanskrit and there is no auxiliary. The upapada ‘sma’ behaves like an auxiliary but we have put it under particle (avyaya). There is no change in the subtypes of pronoun. However, they differ in their attributes. The pronominal takes gender, number, person, case, nominal declension, honorificity, and distance attributes. The reflexive and reciprocal carry gender, number, case, and nominal declension attributes. The relative pronoun has gender, number, person, case, and nominal declension. And finally, the Wh-pronoun takes gender, number, person, case, and nominal declension. The subtypes of the demonstrative are same as in Hindi. The adverb category in which a number of Sanskrit indeclinables fall has no attribute like Hindi adverbs. The postposition is not found in Sanskrit. Their role has been replaced by different declensions. In Sanskrit, we have participles (k danta) with subtypes as participle proper and participle gerundive. They are marked for gender, number, case, and nominal declension attributes. Under the particle category we have two extra subtypes- negative and emphatic. Particles have no attribute. Punctuation and residual have the same conditions in our tagset.

6 Proposed IL-POSTS for Sanskrit Following is the tagset we propose for annotating Sanskrit corpus. Using this tagset, we have annotated a corpus of simple Sanskrit text including articles and stories which are online available in our website at http://sanskrit.jnu.ac.in/corpora/ annotated/MSRIndic-JNUTagsetTaggedCorpora.txt. Currently, annotation of several Sanskrit story-collections is in progress. Table 1. Categories, types and their respective attributes for Sanskrit Category Noun (N)

Verb (V) Pronoun (P)

Type Common (NC)

Attributes gender, number, case, nominal declension

Proper (NP)

gender, number, case, nominal declension

Pronominal (PPR) Reflexive (PRF) Reciprocal (PRC) Relative (PRL) Wh (PWH)

Nominal Modifier (J)

Adjective (JJ) Quantifier (JQ)

pada, number, person, tense\mood, honorificity gender, number, person, case, nominal declension, honorificity, distance gender, number, case, nominal declension gender, number, case, nominal declension gender, number, person, case, nominal declension gender, number, person, case, nominal declension gender, number, case, nominal declension, negative, honorificity gender, number, case, nominal declension, numeral, negative

376

G.N. Jha, M. Gopal, and D. Mishra Table 1. (continued)

Demonstrative (D)

Absolutive (DAB)

Participle (KDP)

gender, number, person, case, nominal declension, distance, honorificity gender, number, person, case, nominal declension, distance, honorificity gender, number, person, case, nominal declension, distance, honorificity gender, number, case, nominal declension

Gerundive (KDG)

gender, number, case, nominal declension

Relative (DRL) Wh- (DWH) Kridant (KD)

Particle (C)

Punctuation (PU) Residual (RD)

Coordinating (CCD) Subordinating (CSB) Gerundive (CGD) Interjection (CIN) Negative (CNG) Emphatic (CEM) Interrogative (CNT) Adverb (CAD) Ambiposition (upapada) (CPP) Quotative (CQT) Comparative (CCM) Reduplicative (CRD) Other (CX)

Foreign word (RDF) Symbol (RDS) Others (RDX)

Table 2. Attributes and their values for Sanskrit No. 1. 2. 3. 4.

Attributes Gender (Gen) Number (Num) Person (Per) Case (Cs)

5.

Nominal declension vibhakti (Vbh)

6.

Tense/Mood (Tns/Mood)

7.

Numeral (Nml)

Values Masculine (mas), Feminine (fem), Neuter (neu) Singular (sg), Dual (du), Plural (pl) First (1), Second (2), Third (3) Nominative (nom), Accusative (acc), Instrumental (ins), Dative (dat), Ablative (abl), Genetive (gen), Locative (loc), vocative (voc), prathamā (i), dvitīyā (ii), tritīyā (iii), caturthī (iv), pa–camī (v), a hī (vi), saptamī (vii), vocative (viii) Present (prs), Aorist (aor), Imperfect (imprf), Perfect (prf), Periphrastic Future (phf), General Future (gft), Imperative (imp), Potential (pot), Benedictive (ben), Conditional (cnd) Ordinal (ord), Cardinal (crd), Non-numeral (nnm)

Annotating Sanskrit Corpus: Adapting IL-POSTS

377

Table 2. (continued) 8. 9. 10. 11.

Distance (Dist) Negative (Neg) Honorificity (Hon) Pada (Pd)

Proximal (prx), Distal (dst) Yes y, No n Yes y, No n parasmaipada (ppd), ātmanepada (apd)

We have also used the following common attributes that MSRI Tagset contains: • •

Not-applicable (0); when any other value is not applicable to the category or the relevant morpho-syntactic feature is not available. Undecided or doubtful (x); when the annotator is not sure about the exact attribute.

7 POS Results and Current Status The initial experiment of tagging 10 K data of ordinary Sanskrit and subsequent training of the tagger at MSRI is documented below Training data: 200 sentences ( ~5.8K words) Test Data: 50 Sentences (~1.2K words) Word Level Acc: 75.35% Sentence Level Acc: 29.3% In the similar experimental set up, the accuracy of Sanskrit was much better than Bangla (71%) and lower to Hindi (77%). Currently, we are tagging about 50 K data from two story collections - Pa–catantra and Hitopadeśa. This task will finish in about two months time. The subsequent training and automatic tagging results will be reported.

8 Conclusion In this paper we have presented a Sanskrit specific tagset framework for annotating Sanskrit corpus. At experiment level a certain amount of data has been manually tagged and we have revised our tagset again and again. This framework follows the guidelines of the IL-POSTS framework for Indic Languages as much as possible. We have tried our level best to be near to this hierarchical framework. It has also been observed that IL-POSTS framework is adaptable for Sanskrit as well. We hope that Sanskrit corpus tagged with this tagset would be more useful as it will ensure the maximal use and sharing of it, following a well-established standard POS tagset framework. This Sanskrit Tagset along with the annotation guidelines (that we ourselves have designed for tagging Sanskrit text) and tagged corpus is available on our website: http://sanskrit.jnu.ac.in.

378

G.N. Jha, M. Gopal, and D. Mishra

References AU-KBC tagset. AU-KBC POS tagset for Tamil, http://nrcfosshelpline.in/smedia/images/downloads/ Tamil_Tagset-opensource.odt Baskaran, S., Bali, K., Bhattacharya, T., Bhattacharyya, P., Choudhury, M., Jha, G.N., Rajendran S., Saravanan K., Sobha L., Subbarao, K.V.S.: A Common Parts-of-Speech Tagset Framework for Indian Languages. In: LREC 2008 - 6th Language Resources and Evaluation Conference, Marrakech, Morocco, May 26-June1 (2008) Baskaran, S., Bali, K., Bhattacharya, T., Bhattacharyya, P., Choudhury, M., Jha, G.N., Rajendran, S., Saravanan, K., Sobha, L., Subbarao, K.V.S.: Designing a Common POS-Tagset Framework for Indian Languages. In: The 6thWorkshop on Asian Language Resources, Hyderabad (January 2008) Cardona, G.: Pānini: His work and its traditions, Motilal Banarasidass, Delhi (1988) Chandrashekar, R.: POS Tagger for Sanskrit, Ph.D. thesis, Jawaharlal Nehru University (2007) Cloeren, J.: Tagsets. In: van Halteren, H. (ed.) Syntactic Wordclass Tagging. Kluwer Academic, Dordrecht (1999) Jha, G.N.: Generating nominal inflectional morphology in Sanskrit. In: SIMPLE 2004, IIT-Kharagpur Lecture Compendium, Shyama Printing Works, Kharagpur, WB (2004) Jha, G.N., Sobha, L., Mishra, D., Singh, S.K., Pralayankar, P.: Anaphors in Sanskrit. In: Johansson, C. (eds.) Proceedings of the Second Workshop on Anaphora Resolution (2008), vol. 2. Cambridge Scholars Publishing (2007) ISSN 1736-6305 Jha, G.N., Mishra, S.K.: Semantic processing in Panini’s karaka system. In: Huet, G., Kulkarni, A., Scharf, P. (eds.) Sanskrit Computational Linguistics. LNCS, vol. 5402, pp. 239–252. Springer, Heidelberg (2009) Greene, B.B., Rubin, G.M.: Automatic grammatical tagging of English. Department of Linguistics, Brown University, Providence, R.I (1981) Hardie, A.: The Computational Analysis of Morphosyntactic Categories in Urdu. PhD Thesis submitted to Lancaster University (2004) IIIT-Tagset. A Parts-of-Speech tagset for Indian Languages, http://shiva.iiit.ac.in/SPSAL2007/iiit_tagset_guidelines.pdf Gerard, H.: The Sanskrit Heritage Site, http://sanskrit.inria.fr/ Kale, M.R.: A Higher Sanskrit Grammar. MLBD Publishers, New Delhi (1995) Leech, G., Wilson, A.: Recommendations for the Morphosyntactic Annotation of Corpora. EAGLES Report EAG-TCWG-MAC/R (1996) Leech, G., Wilson, A.: Standards for Tag-sets. In: van Halteren, H. (ed.) Syntactic Wordclass Tagging. Kluwer Academic, Dordrecht (1999) Leech, G.: Grammatical Tagging. In: Garsire, R., Leech, G., McEnery, A. (eds.) Corpus Annotation: Linguistic Information for Computer Text Corpora. Longman, London (1997) Sudhir, M., Jha, G.N.: Identifying verb inflections in Sanskrit morphology. In: Proceedings of SIMPLE 2004, IIT Kharagpur (2005) NLPAI Contest-2006, http://ltrs.iiit.ac.in/nlpai_cntest06 Hellwig, O.: A Stochastic Lexical and POS Tagger for Sanskrit. In: Huet, G., Kulkarni, A., Scharf, P. (eds.) Sanskrit Computational Linguistics. LNCS, vol. 5402, pp. 266–277. Springer, Heidelberg (2009)

Annotating Sanskrit Corpus: Adapting IL-POSTS

379

Ramkrishnamacharyulu, K.V.: Annotating Sanskrit Texts Based on Sabdabodha Systems. In: Kulkarni, A., Huet, G. (eds.) Sanskrit Computational Linguistics. LNCS (LNAI), vol. 5406, pp. 26–39. Springer, Heidelberg (2009) Santorini, B.: Part-of-speech tagging guidelines for the Penn Treebank Project. Technical report MS-CIS-90-47, Dept. of Computer and Information Science, University of Pennsylvania (1990) Subash, C.: Sanskrit Subanta Recognizer and Analyzer, M.Phil dissertation submitted to Jawaharlal Nehru University (2006)

Effective Authoring Procedure for E-learning Courses’ Development in Philological Curriculum Based on LOs Ideology Galina Kedrova, Anatoly Yegorov, and Maria Volkova Moscow State Lomonosov University, GSP-1, Leninskije Gory, 119991 Moscow, Russian Federation {kedr,yegorov,vochok}@philol.msu.ru

Abstract. An efficient model of hyper textual authoring processes and guidelines for electronic multi-media educational and scientific resources’ development in Philology is being discussed. The model was successfully applied for construction of the interactive hyper-media e-learning Web-resource “Russian Dialectal Phonetics”. The electronic public educational and research resource in Russian Dialectology is the first and unique one fully based on SCORM ideology. Basic authoring algorithms were adopted from the object-oriented programming (OOP) and Rapid Application Design (RAD) ideology and practice for application. The e-course comprises two major components: a theoretical overview and practical sections (on-line self-tests, exercises). An interactive glossary of linguistic and other special terms is also appended through the network of hyperlinks. As essential multi- and hypermedia product the course comprises texts, image illustrations (dialect maps included), authentic dialectal spoken language recordings from the archive of dialectological expeditions. Keywords: e-learning resources, hypermedia authoring principles, SCORM, Russian Phonetics, Russian Dialectology.

1 Introduction Any national or international modern system of education is now facing three global imperatives generated by the new – computerized – era’s challenges in the domain of educational technologies’ development. They are: (1) openness of educational institutions; (2) accessibility of the individually tailored education (anytime – anywhere; ondemand; just in time); (3) continuity of the personal learning’s processes – life-long learning. Openness of the educational space provides equal possibilities to every member of a human society to become educated in a certain chosen field/area. Accessibility of education – “anytime – anywhere” – is of special importance for countries with big territory and dispersed population. Continuity, or life-long learning is tightly related to a constant development and changeover in information technologies being today a key source of modern world’s transitions. Investigation and understanding of the passage of educational interactions from a real world’s natural communication patterns and models to a virtual communication environment is crucial for understanding and adopting the Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 380–387, 2011. © Springer-Verlag Berlin Heidelberg 2011

Effective Authoring Procedure for E-learning Courses’ Development

381

latest progressive trends operating today all the modern educational processes. Thus, a problem of “computerizing”, or “virtualizing” educational resources has become of common primary concern. The technology of e-learning resources authoring is tightly connected with a new concept of e-learning materials (electronic manuals, e-assessments, e-libraries and other e-learning resources – for example, data bases and corpuses). And here a new fundamental concept in Educational Technology: a notion of “learning object” (LO) [1] comes to the front. Learning objects, as defined by the IEEE's Learning Technology Standards Committee, are “any entity, digital or non-digital, which can be used, re-used or referenced during technology supported learning” [2]. Generally, the learning object model could be characterized by the belief that one can create independent chunks of educational content that provide a certain educational experience for some pedagogical purpose. Based on the object-oriented programming (OOP) model, this approach asserts that these chunks are self-contained, though they may contain references to other objects; and they may be combined or sequenced to form longer educational interactions. These chunks of educational content may be of any mode (interactive, passive) and of any format or media type. Main benefits of LO approach to education content authoring is usually defined as so-called RAID package (Reusability, Accessibility, Interoperability, Durability) [3]. The main problem of the worldwide educational community at the moment is that there are no commonly acknowledged and adopted guidelines for effective learning objects’ authoring hitherto. Therefore, any significant “lessons learned” from various latest experiences in e-learning materials’ authoring, production and dissemination are not incurious and could be of some practical importance. Thus, some basic ideas and practical algorithms, tested within an international project of development of the interactive hyper-media e-learning Web-resources “Russian Phonetics” and “Russian Dialectal Phonetics”, are presented for discussion below.

2 Theoretical Background It’s well known that the most significant challenge in LO-based approach to elearning materials authoring is a problem arising from the fact that the reusable learning objects represent in many ways an alternative approach to educational content development. This approach could be characterized as a certain dualism in information processing procedures. On the one part, an educational content needs to be fragmented into meaningful “chunks” irrelevant to their size, physical nature and perception mode. From the other part, taking into account the pedagogical perspective, each “chunk” must play a specific role within a certain instructional design methodology. Linkage of these two concepts is often considered as path to successful elearning resources’ authoring and application. An efficient and promising approach to the problem could be adopted from a software logic engineering practices [4]. From a certain perspective, an online course might be considered as another program application, therefore educators would need to apply design techniques already well-known in the software industry, and in particular, that of Rapid Application Design (RAD). The application of RAD for software development allows a designer to select and apply a set of predefined subroutines from a menu, or

382

G. Kedrova, A. Yegorov, and M. Volkova

selection set within a certain programming environment. Thus, an e-learning course, viewed as a software entity, may be also considered as a collection of reusable subroutines and applications. Simultaneously, an online course, viewed as a system of learning objectives, may be seen as an ordered set of reusable learning materials. It is supposed that merging these two concepts, i.e. conceiving of reusable learning materials as reusable subroutines and simultaneously as reusable pedagogical applications, is a keystone of any successful e-learning resources’ authoring activity. Another productive concept, we would like to offer for adoption from the software programming theory and practice, is the notion of object-oriented design elaborated within OOP theory [5]. The core idea in object-oriented design is that of reproduction, or “cloning”: prototypical software entities, once defined, could be easily cloned, and used as a piece of software when and where needed.

3 Authoring Practice and Guiding Principles We have tried to apply the both mentioned above basic approaches to the construction of e-learning course of Russian Dialectal Phonetics on the Web [6]. The open-source electronic public educational and research resource in Russian Dialectal linguistics was programmed as the pilot one based on SCORM ideology in the field. In the functional perspective the course’s structure comprises two major components: (1) a comprehensive theoretical overview of the domain, and (2) interactive practical sections (on-line self-tests, exercises, etc.). An interactive glossary of all encountered in texts linguistic and other special terms should be also provided through the dense network of hyperlinks. As essential multi- and hypermedia product, the linguistic e-course normally encompasses information in various perception modes: texts, image illustrations (dialect maps included), authentic dialectal spoken language recordings. According to the pre-planned authoring procedures a “crude” course’s general prototype has been defined beforehand, and a specific e-learning course (namely, e-learning Webmanual of the Russian Phonetics [7]) has been created online using prototyping and cloning ideology. In the previous authoring activity we have tried to formulate some more detailed precursory parameters of a linguistic course-prototype (for more details see also [8]). It is worth mentioning that since the year 2001 these presuppositions have been evaluated through practical use of the Russian Phonetics e-course in various distant education programs of teaching and learning Russian linguistics. This project being one of the most successful and proven as an authoritative source on the subject by Wikipedia, has initiated a series of similar educational and scientific eresources both in Russia and worldwide. Therefore, we were encouraged to apply the same procedures and programming patterns, though revised and re-developed according to new theoretical concepts, in construction of the original educational and scientific Web-resource on the Russian Dialectal Phonetics. 3.1 Course Prototype Structures Designers of computer interfaces have already long ago understood that the starting point of any good interface design – that is user-friendly and corresponding to users’ intuitive layout schemes program interface, should be a metaphor. The metaphor

Effective Authoring Procedure for E-learning Courses’ Development

383

principle demands that both monitor screen layout and basic means of interaction with the computer system should appeal to a situation familiar to the user. The metaphor principle determines many other valuable parameters of the hypertext design providing information integrity, information consistency and information density. Various aspects of the underlying metaphor could help to outline a formal system of all basic interaction patterns at the earliest stage of designing processes, and to determine thus an inventory (‘alphabet’) of reusable design elements, each of them being assigned special, context-dependent function and meaning. Normally the system of reusable design elements should coincide with system of basic LOs prototypes. This approach has proved its unique efficacy in construction of vast Internet-portals, normally containing hundreds of interconnected Web pages, each of them enriched with multimedia and hypermedia information. To outline a hierarchy of LOs prototypes one needs to classify and categorize basic constituents of the whole semantic space of instructional content keeping in view their didactical roles. It has become already a common sense statement that any online learning content typically contains: 1. 2. 3. 4. 5. 6.

Text, sound, graphics, and (optional) movies; Controlled (interactive) graphic and animations [9]; A navigation facilities (a scheme, a table of contents and/or buttons); Drills and self-assessment facilities; Assessments and evaluation forms; External/internal hyperlinks. Thus, a prototypical lesson in our course of Russian Dialectology consists of:

− Text properties of two principle kinds (title, paragraph); − Sound illustrations’ properties (for dialectal / standard speech digital recordings’ import); − External / internal (cross-reference) hyperlinks’ properties (glossary entries included); − Graphical illustrations’ properties (for sonogram, oscillogram, intonogram, scheme, etc. import); − Animation (video) illustrations’ properties (for simulations and videos); − Exercises of two principle patterns (self-assessments and control tests). When an informational “chunk” (a lesson) prototype should be defined for the first time, it makes no sense to specify for this prototype alone each and all of the above listed properties. Rather, what happens in object-oriented design is that the most basic prototype is constructed first – for example, in our case, in a Lesson prototype – the most generic Title & Paragraph & Illustrative media & Hyperlink prototype, or in a model Exercise prototype – the most generic Question & Answer & Check box prototype. An internal (external) hyperlink prototype inherits more general hyperlink properties, but has a special value: it is targeted to internal data base subroutine (in our case a Glossary), or an URL. Then, the next more detailed prototype, for example, a graphics’ prototype is defined. The graphics prototype “inherits” the lesson prototype; that is, we say that all the properties a lesson can have, a graphics item can have as well. Thus, when we

384

G. Kedrova, A. Yegorov, and M. Volkova

create the graphics prototype, we need only to create – to add-on those properties and behaviors that are unique to graphical illustrations. And so this defining continues on down the hierarchy. When we create a scheme prototype, we define a scheme as inheriting all the properties of a graphics, and define only those properties that are unique to scheme. Thus programmers can quickly and efficiently create a new type of entity – a special class of illustrations, for example, a new sub-class of graphics (Dialectal maps, for example) – by simply inheriting the necessary properties from more generic entities. A sample Lesson page with marked prototype attributes for menu bars, text, audio, video, image items, exercises and self-assessments, as well as referential and crossreferential links is exposed on Fig. 1. A sample Lesson page with marked prototype attributes for graphics exemplars is exposed on Fig. 2. Learning object prototypes may also define prototypical actions or behaviors for their clones. For example, a behavior we might expect from a student doing self-assessments, or control tests is to click the right answer (for multiple answer question type assessment) and to push the re-do button (or activate any similar icon facility). The assessment prototype has this behavior predefined as a function; when a clone is created – for example, a multiple answer control test, it comes complete with this behavior. Hence, we can make our clone do things by referring to these predefined functions.

Fig. 1. Lesson sample page with LOs’ marking: 1 – Modules’ menu bar; 2 – Paragraph; 3 – Sound speech example; 4 – Sonogram entry link; 5 – Glossary entry window; 6 – Exercise; 7 – Lessons’ menu column; 8 – Self-assessment activity

Effective Authoring Procedure for E-learning Courses’ Development

385

Fig. 2. Lesson sample page with Graphical illustrations’ patterns: 1 – Dialectal map window; 2 – Sonogram illustration window

Objects may interact, or more generally, be related to each other, in many ways. The most useful and common form of interaction is the containing interaction. The course comprises modules, into which lessons were encompassed as another learning object. The containing interaction of the LOs’ represented in their titles (or metadata) taken in hierarchic order, forms navigational menu’s prototype(s). Thus, a well-ordered sequence of navigation interactive keys could be fast formed quite automatically, according to the author’s current intentions for educational content’s granulation and (re)structuring. Another type of interaction is a hyperlinked association either with external objects (to other web-sites, for example), or to internal objects – cross-references and links to the Glossary items’ list (see Fig. 1-5). 3.2 Course Design Requirements The LOs-based approach has proved its unique efficacy in construction of vast Internet-portals, normally containing hundreds of interconnected Web pages, each of them enriched with multimedia and hypermedia information. It is important to emphasize that the latter capacities require additional special designing patterns and procedures, while psychology research in human perception has revealed that activation of any new perceptive modality – a new perceptive experience (for example, dynamic, animated, or sounding objects incorporated into a text fragment or placed inside graphic

386

G. Kedrova, A. Yegorov, and M. Volkova

elements, and vice versa), would disturb continuity of perceptive processes, dissipate user’s attention and could thus produce strong distracting effect in general. Therefore, correct design representation of conventional (textual) and/or multi- and hypermedia LOs (i.e. text, image, animation, video and sound mixture) is the key element for adequate information processing, as well as for preserving psychological safety and stability of user, as well as her/his mental comfort. Synthetic nature of the hypertext’s perception and non-linear processing procedures impose special demands to semantics of the color palette for every LO, harmony and conformity of a color scale on a page and throughout the whole Web-site, significative balance of hyperlinks’ color markers and corresponding LOs: sound, image, animation, or video-clip palette, etc. Within the both e-learning resources it’s the meaningful interplay of all organizing and designing features (i.e. logo and navigation bars, background colors and text character palette, font type, general text formatting schemes, format of hyperlink markers, etc.) that play special role and function to mark various LOs, and their interaction patterns. Worth to add that aesthetically the overall color palette of the Web site should also be thoroughly harmonized within the general Web-page layout, as well as across the whole set of Web pages embodying certain information hyperspace. These principles of the Internet design should be also based on the current LOs inventory and content structuring, enabling therefore to ‘melt’ text and multimedia elements into a perfect whole either within an integrated communicative message on a single Web page, or across the pages on an Internet-site. Only the systematic integrity of the kind allows to reach such information and aesthetic value when all consists of everything, every constituent is in harmony with the other ones, any new element could be easily built-in into the whole comprehensive structure and would get support from other elements of the system [10].

4 Results Nowadays we state that the above described theoretical assumptions and elaborated within the project’s activity practical algorithms have proved ever since its effectiveness in time- and labour-savings. Below is some numerical data of the estimated size of the Web-site e-learning resource “Russian Dialectal Phonetics”, which has been authored in accordance with the above-described principles within the time period of two years. Currently the site consists of: − 57 presentation chapters (in 4 main modules); − 212 contextual as well as separate interactive and multimedia self-assessments’ modules; − 122 control integrative interactive multimedia tests; − 2090 speech illustrations (sound-) files; − 515 graphic illustrations (image-) files; − 52 interactive dialectal maps; − 57 sonograms; − 161 glossary terms with definitions.

Effective Authoring Procedure for E-learning Courses’ Development

387

As an open-source product elaborated upon detailed study of the most effective SCORM-technologies and appropriate ideology, the e-course “Russian Dialectal Phonetics” could be further enlarged and outspread to other levels of linguistic description of the Russian dialectal system, i.e. morphology, syntax, lexicology, etc. Thus, our results endorse an assumption of the effectiveness of the technologically based approach to LOs’ construction and application. As Stephen Downes wrote in [4]: “Courses developed along previously used models – the Bates model in the first hand – were expensive because of two major (and related) design features. First, all course material is created from scratch, and second, this material is applied only to the limited number of students taking this particular course. In order to lower costs, therefore, a course development program must enable educators to avoid creating everything from scratch, and to allow created course content to be applied to a much larger number of students”. The elaborated authoring practical guidelines and recommendations could be considered as a convincing example of these principles’ efficiency.

References 1. Advanced Distributed Learning Initiative, http://www.adlnet.org 2. Learning Technology Standards Committee, Institute of Electrical and Electronics Engineers, http://ltsc.ieee.org 3. Quinn, C., Hobbs, S.: Learning Objects and Instruction Components. Educational Technology and Society 3(2) (2000) 4. Downes, S.: Learning Objects: Resources For Distance Education Worldwide. International Review of Research in Open and Distance Learning 2(1) (2001) 5. Montlick, T.: What is object-oriented software? Software Design Consultants, http://www.softwaredesign.com/objects.html. 6. Russian Dialectal Phonetics, [http://dialect.philol.msu.ru, Фонетика русских диалектов] 7. Russian Phonetics, http://fonetica.philol.msu.ru, [Русская фонетика] 8. Kedrova, G.: Actual problems of e-learning resources’ authoring processes for distance education. The Quality Bulletin 12(72), 26–31 (2006), [Кедрова Г.Е. Актуальные вопросы проектирования электронных учебных материалов для дистанционного обучения. Вестник качества, 12(72), 26–31 (2006)] 9. Pankov, P.S., Alimbay, E.: Virtual Environment for Interactive Learning Languages. In: Vetulani, Z. (ed.) Proseedings 2nd LTC for Human Language Technologies as a Challenge for Computer Science and Linguistics, Poznan, Poland, pp. 357–360 (2005) 10. Kirsanov, D.: Web-design: a book of Dmitrij Kirsanov. Symbol Press, St. Petersburg (2003), [Кирсанов Д. Веб-дизайн: книга Дмитрия Кирсанова. Изд-во Символ-Плюс. С-Пб, 2003]

Acquisition of Spatial Relations from an Experimental Corpus Jędrzej Osiński Adam Mickiewicz University, Department of Computer Linguistics and Artificial Intelligence, ul. Umultowska 87, 61-614 Poznań, Poland [email protected]

Abstract. Qualitative techniques are important in spatio-temporal reasoning and in artificial intelligence in general. The Cardinal Direction Calculus (CDC) is one of the classic formalisms which represents the spatial relations between objects whose positions are described in the reference to the geographical directions. This particular model may be used in many systems also in the applications which are based on natural language input. In this paper we discuss the problem of generating an intuitive, simple and quickly understood answer to specific question concerning spatial relations asked by a user of a system. We present the results of the linguistic experiment which was performed to analyse human language competence in describing space. Keywords: natural language processing, qualitative knowledge, cardinal direction calculus.

1 Introduction Qualitative techniques, which make abstraction from quantitative knowledge, can be successfully used for spatial reasoning including the description of spatial relations between objects, computation of positions or for assessing similarity between spatial scenes. Representing qualitative relation between intervals as originally proposed by Allen in [1] was one of the first formalisms to describe intuitive temporal relations without exact duration. Allen introduces thirteen temporal relations together with a “transition” table presenting composition rules for any two of them. Qualitative reasoning about relations in 2D space was comprehensively discussed by Frank ([4]) who analyses the necessary properties of formal cardinal direction systems. In [6] a model was proposed for analysing cardinal directions between points: the space around the reference point is divided into a number of mutually exclusive areas. The rectangle calculus originally presented in [2] focuses on relations between two rectangular objects whose sides are parallel to the axes of an orthogonal basis in a two-dimensional Euclidean space (169 permitted relations were introduced). The next important step in the development of qualitative techniques was introduction of the Cardinal Direction Constraints (CDC) calculus discussed in [5] where cardinal directions were adopted for reasoning about spatial relations between regions (without predefined shape or Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 388–399, 2011. © Springer-Verlag Berlin Heidelberg 2011

Acquisition of Spatial Relations from an Experimental Corpus

389

size) instead of points. Recently we introduced in [8] an extension of this technique to a temporal dimension for representing complex spatio-temporal entities (XCDC). Examples of the usage of the XCDC were presented in [7] where the problems with the monitoring of complex events were discussed. For simplicity, during the experiment we focused the attention of the participants only on the spatial aspect of the analysed situation. Consequently, in the present article we make an abstraction from the temporal span (time interval) and focus on the classical CDC model.

2 Description of the Problem The key idea of the Cardinal Direction Constraints formalism (originally proposed in [5]) is based on dividing the plane around the reference object (i.e. the object from which the direction relation is determined) into nine regions named after the geographical directions: NW, N, NE, W, O (central region meaning the same location), E, SW, S and SE. These areas, called direction tiles, are closed, unbounded (except for O), their interiors are pairwise disjoint and their union is the whole plane. Directions between the reference object A and target object B are represented in a 3 x 3 matrix denoted by dir(A,B). It is worth noticing that this classical model is often extended in different aspects, e.g. in [3] we can find reference to the four bordering lines on A at the top, bottom and both sides. However we define the direction-relation matrix as follows:

Let us consider a system based on natural language processing which uses the described technique for reasoning about spatial relations between objects. The system should be able to answer the question asked by a user and the answer should be generated as a sentence in a natural language. Suppose the relation between two objects (denoted by A and B) of the analysed scene is represented by the following direction-relation matrix dir(A,B):

Now suppose a user sends a question about this relation. In response the user receives a message similar to the sentence: B is to the north of A. That appears as an intuitive and natural result and we can assume it is the kind of answer the user would expected. However not all the possible relations are stored in the system. Some need to be calculated using a composition algorithm e.g. the one presented in [9]. Let us consider

390

J. Osiński

the following example. Suppose there is information about two relations between objects X, Y and Z stored in the system:

Now let us calculate the relation between X and Z (for simplicity the result is only one of disjunctive relations; this relation contains all possible direction tiles):

This time the sentence in a natural language to be generated in a response is not so obvious. A very precise answer would be similar to the following: Z is to the northwest, north and north-east of X. On the other hand the sentence Z is to the north of X seems to be more natural and easier for a user to interpret, especially while working under stress and time pressure. The presented aspect is also important if there are requirements for the length of a sentence (e.g. in a text message sent to a mobile phone) or the number of sentences to be analysed. It is crucial to decide what kind of statements are the ones expected by a user. According to the definition of the ‘soft artificial intelligence’ and the Turing test, a system can be called ‘intelligent’ if under the same conditions it answers similarly to the way a human would. That was the main reason why we decided to check how people describe spatial relations.

3 Experiment The aim of the experiment was to collect the specific corpus of statements describing spatial relations between objects. The relations chosen for the experiment were the ones which can be expressed in various ways in a natural language. It was important to verify which sentences for the particular relations are the most frequent and how wide is the diversity of the statements. It is important to mention that we were interested only in the spatial semantics of this statements. We made abstraction from all the other language aspects. The second aim of the experiment was to compare the collected results by taking into consideration the participants’ profiles. In particular we check whether the education level influence the way the statements are formulated. The experiment was prepared to fulfill the following assumptions: 1. The collected corpus should cover most of the possible statements that can be used to describe the defined relation. 2. The experiment should be easy for participants to avoid distortions caused by misunderstanding of the instructions. 3. The influence on the participants should be minimized. In particular, no examples or suggestions of expected sentences should be presented. 4. The answers of participants should be precise and focused on the spatial relations.

Acquisition of Spatial Relations from an Experimental Corpus

391

Fig. 1. The general pattern of the experiment tasks

The experiment was performed via Internet, without a direct contact between the participants and the organizers. All the information necessary to take part in the experiment was presented on a web page. The information about the experiment was distributed among the students of the Adam Mickiewicz University and within social network services. The only constraints for the participants were: age over 18, an individual e-mail box and Windows as the operating system. The first two requirements were included to reduce the number of statements sent by internet users who could treat the experiment more like fun rather than a research project. The necessity of providing a private e-mail address entailed a partial loss of anonymity on the Web. The age limit guarantees that the participants are people with established language competences. The instruction for the participants was the following: 1. 2. 3.

Download the experiment application. Run the experiment application. Fill in the short questionnaire typing your name, age and choosing your educational level from the list. 4. In the next window a picture will appear (the draft of a map – a bird’s eye view). Under the picture there will be a question (about the location of an event) which you are asked to answer in the Polish language, in a precise but also concise form. 5. There are 25 tasks. Each question is the same but the pictures are different. 6. After typing each statement press ‘next’. After the last question a file will be generated. Send it as an attachment of an e-mail message.

392

J. Osiński

Fig. 2. The legend (translated into English) for the tasks of: (a) Type 1, (b) Type 2

As we can see, the instruction is simple and prepared carefully to avoid possible mistakes by a user. Each task is based on the same pattern (Fig. 1.). The main area of the window contains the schematic picture describing a relation between two objects in the context of a stadium: one representing the place of a fight and the other – the location of playing pitch. Because we focused our attention only on the spatial semantics of the statements, we use these two objects for all the pictures. Also the question asked does not change: ‘Gdzie ma miejsce bójka?’ (‘Where is the fight?’). We divided all the tasks into two groups: Type 1 and Type 2. In Type 1 the legend contains an arrow which points in an northern direction (Fig. 2a.). The arrow appears also on the situation scheme (the picture to be described). We expected to collect the corpus of relations expressed by giving absolute directions. In Type 2 the legend contains the pictogram of a human face which marks the location of a person to whom the statement is addressed (Fig. 2b.). There is also additional information added below: Attention: This time you do not know where the north is! We expected to collect interesting sentences expressing subjective relations. There are 18 tasks of Type 1 and 7 of Type 2. After the experiment was finished the collected statements were grouped by the tasks. Then from each of the sentences the expression that describes the spatial relation was extracted. This process resulted in obtaining the set of expressions which correspond to spatial relations. On that set a statistic analysis was carried out.

4 Results A total of 71 people took part in the experiment which produced a number of 1776 statements written in a natural language to describe the spatial relations. Only 18 sentences were marked as ‘N/A’ because they were not related to the topic of the experiment (describing non-spatial aspects of a presented situation), did not answer the question or the text was empty. That means over 98.9 % of the collected statements fulfill the requirements. This value is surprisingly high for internet experiments which are usually not very reliable due to the dynamic, impossible to control and hard to define environment.

Acquisition of Spatial Relations from an Experimental Corpus

393

Fig. 3. The relative growth of the number of new sentences

As we mentioned before, one of the aims of the experiment was to collect a corpus which covers most of the possible statements. To verify whether this requirement was met, we counted the number of new expressions (which correspond to spatial relations) added to the corpus in steps after analysing sentences sent by 10, 20, 30, 40, 50, 60, 71 people. Fig 3. shows the number of new expressions in percentage:

where E0 is the old number of different expressions, E1 – the new number of different expressions and NE is percentage relative growth. The growth was reducing significantly for every subsequent group of participants and the last growth (between 60 and 71 participants) was just 5.2%. That result fulfils the requirement stated above (see chapter 3, requirement 1). Now let us discuss the results for the particular tasks. Fig. 4. shows the pictures presented to the participants in each of 25 tasks. 4.1 Type 1 Task 1: 97.2% of the participants defined the relation as ‘to the north of’. There was only 1 statement which says ‘outside’ which is much less precise. This example shows that a matrix in a CDC formalism containing only one value 1 can be directly translated into a natural language and would fulfill the user expectation. Task 2: This example is exactly the situation described in the introduction as a motivation for the experiment. 84.5% of the people said ‘to the north of’ in comparison with 9.9% for ‘to the north-west, north and north-east of’. This shows that a user prefer more intuitive way of describing real-world situations. Task 3: 64.8% described it as ‘to the north-west of’, 14.1% - ‘to the north-west and north of’, 7.0% - ‘to the north of and a little to the west’, 5.6% - ‘to the north of’. Task 4: 70.0% of the participants defined the relation as ‘to the north-west of’, 16.9% - ‘to the north-west, north and west of’, 8.5% - ‘to the north and west of’, and only 2.8% ‘around the corner of’.

394

J. Osiński

Fig. 4. The pictures presented to the participants of the experiment

Task 5: 45.1% described the relation as ‘to the north and west of’, 18.3% - ‘to the north-west and west of’ and only 7.0% - as ‘to the north, north-west, west and south-west of’, which is the most precise sentence possible for the CDC model. Task 6: 23.9% of the internet participants described the relation as ‘to the south, west and north of’ and it was most frequent answer. 14.1% respond ‘around except the east’, 12.7% - ‘around’, 9.9% - ‘everywhere except the north-east’, 8.5% - ‘around except the north-east and east’ and 8.5% - ‘to the south, west and north-west of’. It is worth mentioning that this particular task is the first in which we can divide all the statements into two classes: the sentences describing where the object is (denoted by Class A) – 66.2% and the sentences describing a relation by complement – everywhere except X, denoted by Class B (here used by 33.8% of the participants). Task 7: 38.0% of the participants defined the relation as ‘to the north-west and north of’, 18.3% - as ‘to the north-west and south-east of’, 14.1% - ‘to the north-west, south and

Acquisition of Spatial Relations from an Experimental Corpus

395

south-east of’ and 7.0% - ‘to the north, north-west, south and south-east of’. There were only 2 statements of Class B. Task 8: 56.3% described the relation as ‘to the east, south and north-west of’, 19.7% ‘around’ (increase by 7.0% compared to the task 6), 5.6% - ‘to the north, north-west, south, south-east, north-east and east of’. There was no statement of Class B. Task 9: 59.1% of the participants defined the relation as ‘to the east, west and south of’, 8.4% - as ‘to the south, west and north-west of’, 7.0% - ‘around except the north’ and 5.6% - ‘around’. Task 10: This task was the first of seven prepared to verify human language competence in describing inside relations. 42.3% of the participants described the relation as ‘inside’, making abstraction from the very exact position of the fight. 33.8% said ‘in the northeast part of’, 11.3% - ‘in the centre and the north-east part of’ and 8.5% - ‘in the centre of’. Task 11: In this picture the difference in the size of the objects are more significant, causing different perception of the picture. In result 90.1% of the people described the relation as ‘in the north-west part of’ and just 2.8% - as ‘inside’ (which means decrease by almost 40% compared to the task 10). Task 12: 76.0% of the participants defined the relation as ‘in the north part of’, 8,5% - ‘in the north-west and north part of’ and 4,2% - as ‘in the north part of except the northeast’. Task 13: In this picture the reference object is much bigger compared to the task 12. The position and the size of the target object (the area of the fight) remain the same. 45.1% described the relation as ‘in the north-west part of’, 9.9% - ‘in the north and west part of’ and 7.0% - as ‘in the north-west and north part of’. Task 14: 81.7% of the internet participants described the relation as ‘in the centre of’ (increase by more than 70% compared to the task 10), 8.5% - ‘in the centre and a little in the north part of’ and 5.6% - as ‘in the centre and a little in the north-west part of’. Task 15: In this picture the qualitative relation remains the same as in the task 14, however the size of the target object was reduced. All the participants (100% of the statements) described this relation as ‘in the centre of’. Task 16: 45.1% described the relation as ‘in almost all the area of’, 35.2% - ‘in the all area of’ and 14.1% - ‘inside’. As we can see comparing tasks 14-16, the answers are totally different when manipulating the size of the objects of the described scene.

396

J. Osiński

Task 17: 43.7% of the participants defined the relation as ‘to the north-east of’, 19.7% - as ‘to the north of’, 15.5% - ‘to the north and north-east of’ and 9.9% - as ‘to the north and east of’. This picture is almost symmetric to the one presented in the task 3. Also the results are similar. That may lead to the conclusion that an analogous relation between objects is described in a similar way and that is highly independent from the actual absolute direction. Task 18: 88.7% described the relation as ‘to the west and east of’, but also 7.0% said ‘to the left and right of’, which is very subjective and imprecise. 2.8% of participants said it is placed ‘symmetric to’ but without pointing the absolute direction. 4.2 Type 2 Task 19: 47.9% of the internet participants described the relation as ‘behind’, 23.9% - ‘on the opposite side’ and 19.7% - ‘on the other side’. Task 20: 22.5% described the relation as ‘behind’, 18.3% - ‘on the other side’, 16.9% - ‘behind and to the right of’, 11.3% - ‘on the opposite side and to the right of’ and 7.0% - ‘on the opposite side’. Task 21: 80.3% of the participants defined the relation as ‘to the right of’, 4.2% - ‘on the sidelines of’. It is also worth noticing that 5.6% said ‘to the left of’ what points out the aspect of the orientation of the interlocutor (whether he/she looks at the playing pitch or not). Task 22: 33.8% described the relation as ‘behind and to the right of’, 14.0% - ‘on the opposite side and to the right of’, 14.0% - ‘on the other side and to the right of’ and 7.0% - ‘on the back of’. That shows there is no definite majority of any kind. We can compare it to the task 4 in which the picture is analogous with additional pictogram defining the north direction (the most frequent answer covers 70% of the sentences). Task 23: 52.1% described the relation as ‘inside’, 15.5% - ‘in the centre of’, 11.3% - ‘in the centre and the right part of’, 5.6% - ‘in the right part of’. The most popular answer is the same as in the task 10, so the inside topological relation is independent from the objectivity level. Task 24: 49.3% of the internet participants described the relation as ‘inside and to the right of’, 26.8% - ‘in the centre and to the right of’ and 5.6% - as ‘in the right part and to the right of’. The least precise answer was the most frequent. Task 25: 25.4% described the relation as ‘in front of the interlocutor’, 23.9% - ‘behind the interlocutor’, 16.9% - ‘on the opposite side of’, 5.6% - ‘the interlocutor is between

Acquisition of Spatial Relations from an Experimental Corpus

397

the objects’, 4.2% - ‘across from’ and 4.2% - ‘on the other side of’. As we can see there was no answer with definite majority.

5 Results and the Participants’ Profiles As we have mentioned above before the participants started to describe the pictures of the spatial relations they were asked to fill in a short questionnaire. In particular they needed to choose their educational level from the following list: 1. 2. 3. 4. 5. 6. 7.

elementary education (no participants), secondary education in technical fields (no participants), secondary education in other fields (1.4% of participants), student of a scientific field (40.0%), student of another field (38.6%), higher education in a scientific field (7.1%), higher education in another field (12.9%).

For the purpose of the analysis we have divided all the participants into two groups depending on the fields studied: Alpha (education types: 2, 4 and 6), Beta (education types: 1, 3, 5 and 7). Then we have calculated the statistics of the average number of words and characters in the participants’ statements (Tab. 1.). As can be seen the people classified as the Alpha group used about 4-5 characters less than in the Beta group (if we compare the value of quartile ¾ we can notice that the real difference is even larger). Similarly their statements are on average one word shorter. We can come to the conclusion that the participants whose education is connected with technical and scientific fields are more concise in their way of describing spatial relations. It coincides with the widespread opinion that scientists are more economical with their words in comparison with specialists in the arts. Table 1. The number of words and characters

Total number Average Median Quartile ¾

Number of characters Alpha Beta 32192 40666 37.87 43.96 32 36 46 55

Number of words Alpha Beta 5264 6657 6.19 7.20 5 6 8 9

We also tried to analyse whether the type of education influences the precision of the statements. We have compared the types of expressions used for the selected tasks versus the type of education (Tab. 2.). For this analysis we have chosen mainly the tasks in which there was no majority answer (see the results presented in the previous section). As we can see in the results of the task 6, the difference appears while analysing the statements of Class B (complement) – the direction is more precise within the Alpha group. Similarly their sentences are more exhaustive in describing the absolute relation presented in the task 7. On the other hand task 17 and 23 show there in no significant correlation between the precision of information and the type of education.

398

J. Osiński Table 2. The type of expression versus the type of education Expressions Task 6 S+W+N around, exc. E around around, exc. NE Task 7 NW + S NW + SE NW + S + SE Task 10 inside NE part centre + NE centre Task 6 S+W+N around, exc. E around around, exc. NE Task 7 NW + S NW + SE NW + S + SE

Participants (%) Alpha Beta 24.2 9.1 15.1 15.1

24.3 18.9 10.8 5.4

30.3 24.2 18.1

45.9 13.5 10.8

45.5 39.4 6.1 12.1

40.5 29.7 16.2 5.4

24.2 9.1 15.1 15.1

24.3 18.9 10.8 5.4

30.3 24.2 18.1

45.9 13.5 10.8

Expressions Task 10 inside NE part centre + NE centre Task 16 in almost all area in all the area inside Task 17 NE N N + NE Task 23 inside centre

Participants (%) Alpha Beta 45.5 39.4 6.1 12.1

40.5 29.7 16.2 5.4

30.3 54.5 15.2

59.5 18.9 13.5

48.5 24.2 9.1

40.5 16.2 21.6

57.5 18.2

48.7 13.5

Results in the task 16 may lead to the conclusion that the people connected with science interpret the diagrams (e.g. the bordering lines of the playing pitch presented in the picture) in a different way than the participants who are connected with arts. This analysis shows that there is no serious difference in the way of describing spatial relation by the people with a different educational background. However this aspect of the human language competence should be studied by a further analysis.

6 Conclusion The presented results of the experiment illustrate human competences and preferences in the way of describing spatial relation using a natural language. They also show what kind of phrases and the level of granularity is usually preferred by a user. This results was used to prepare the converter module used for translating the model knowledge (stored in the system as direction-relation matrices describing the world in the CDC formalism) into the language sentences which are likely to be expected and understood by a user. Also more careful analysis will be carried to compare the differences in the answers with the pictures’ properties. Moreover the collected corpus will be analysed from the point of view of syntax and lexicon.

Acquisition of Spatial Relations from an Experimental Corpus

399

Acknowledgements This research was partially covered by the Polish Government grant R00 028 02 “Text processing technologies for Polish in application for public security purposes” (2006-2009) headed by Z. Vetulani, within the Polish Platform for Homeland Security. I would like to thank Z. Vetulani (UAM, Poland), T. Obrębski (UAM, Poland) and D. Byrne (Athlone IT, Ireland) for their helpful comments. Also great thanks to all the participants of the experiment for their time and commitment.

References 1. Allen, J.F.: Maintaining Knowledge about Temporal Intervals. Artificial Intelligence and Language Processing 26(11), 832–843 (1983) 2. Balbiani, P., Condotta, J., Fariñas del Cerro, L.: A new tractable subclass of the rectangle algebra. In: Proceedings of 16th International Joint Conference on Artificial Intelligence (1999) 3. Cicerone, S., Di Felice, P.: Cardinal directions between spatial objects: the pairwiseconsistency problem. Information Sciences 164(1-4), 165–188 (2004) 4. Frank, A.: Qualitative Spatial Reasoning about Cardinal Directions. In: Proceedings of the Tenth International Symposium on Computer-Assisted Cartography (Auto-Carto 10) (1991) 5. Goyal, R.K., Egenhofer, M.J.: Cardinal directions between extended spatial objects. IEEE Transactions on Knowledge and Data Engineering (2001) 6. Ligozat, G.: Reasoning about Cardinal Directions. J. Visual Languages and Computing 9, 23–44 (1998) 7. Ligozat, G., Vetulani, Z., Osiński, J.: Spatio-temporal aspects of the monitoring of complex events. In: Proceedings of Workshop on Spatio-Temporal Reasoning at IJCAI 2009, Pasadena, USA (2009) 8. Osiński, J.: Extending the cardinal direction calculus to a temporal dimension. In: Proceedings of FLAIRS 2009, Sanibel Island, USA (2009) 9. Skiadopoulos, S., Koubarakis, M.: Composing cardinal direction relations. In: Jensen, C.S., Schneider, M., Seeger, B., Tsotras, V.J. (eds.) SSTD 2001. LNCS, vol. 2121, pp. 299–317. Springer, Heidelberg (2001)

Which XML Standards for Multilevel Corpus Annotation? Adam Przepiórkowski1,2 and Piotr Bański2 1

2

Institute of Computer Science PAS, ul. Ordona 21, 01-237 Warszawa, Poland [email protected] University of Warsaw, Krakowskie Przedmieście 26/28, 00-927 Warszawa, Poland [email protected]

Abstract. The paper attempts to answer the question: Which XML standard(s) should be used for multilevel corpus annotation? Various more or less speciﬁc standards and best practices are reviewed: TEI P5, XCES, work within ISO TC 37 / SC 4, TIGER-XML and PAULA. The conclusion of the paper is that the approach with the best claim to following text encoding standards consists in 1) using TEI-conformant schemata that are 2) designed in a way compatible with other standards and data models. Keywords: corpus encoding, TEI, linguistic annotation, XML, ISO TC 37 / SC 4, TIGER-XML, PAULA.

1

Introduction

The need for text encoding standards for language resources (LRs) is widely acknowledged: within the International Standards Organization (ISO) Technical Committee 37 / Subcommittee 4 (TC 37 / SC 4), work in this area has been going on since the early 2000s, and working groups devoted to this issue have been set up in two current pan-European projects, CLARIN (http://www.clarin. eu/) and FLaReNet (http://www.flarenet.eu/). It is obvious that standards are necessary for the interoperability of tools and for the facilitation of data exchange between projects, but they are also needed within projects, especially where multiple partners and multiple levels of linguistic data are involved. One such project is the international project KYOTO (Knowledge Yielding Ontologies for Transition-based Organization; http://www.kyoto-project. org/), involving 11 institutions from Europe and Asia. Another is the much smaller National Corpus of Polish project (Pol. Narodowy Korpus Języka Polskiego; NKJP; http://nkjp.pl/; [22,23]) involving 4 Polish institutions. What these two very diﬀerent projects have in common is the strong emphasis on the conformance with current XML standards in LR encoding. It is interesting that this common objective gives rise to very diﬀerent practices in these projects. The aim of this paper is to present the way the National Corpus of Polish (henceforth, NKJP) attempts to follow standards and best practices in encoding multiple layers of linguistic annotation. The comparison of XML encoding Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 400–411, 2011. c Springer-Verlag Berlin Heidelberg 2011

Which XML Standards for Multilevel Corpus Annotation?

401

schemata and underlying data models is a tedious and time-consuming task, and we hope that the following discussion will help other developers of LRs choose the standard best suited for their needs.

2

Requirements

NKJP is a project carried out in 2008–2010, aiming at the creation of a 1billion-word automatically annotated corpus of Polish, with a 1-million-word subcorpus annotated manually. The following levels of linguistic annotation are distinguished in the project: 1) segmentation into sentences, 2) segmentation into ﬁne-grained word-level tokens, 3) morphosyntactic analysis, 4) coarse-grained syntactic words (e.g., analytical forms, constructions involving bound words, etc.), 5) named entities, 6) syntactic groups, 7) word senses (for a limited number of ambiguous lexemes). Any standards adopted for these levels should allow for stand-oﬀ annotation, as is now common practice and as is virtually indispensable in the case of many levels of annotation, possibly involving conﬂicting hierarchies. Two additional, non-linguistic levels of annotation required for each document are text structure (e.g., division into chapters, sections and paragraphs, appropriate marking of front matter, etc.) and metadata. The standard adopted for these levels should be suﬃciently ﬂexible to allow for representing diverse types of texts, including books, articles, blogs and transcripts of spoken data.

3

Standards and Best Practices

The three text-encoding standards and best practices listed in a recent CLARIN short guide ([6])1 are: standards developed within ISO TC 37 / SC 4, the Text Encoding Initiative (TEI) guidelines and the XML version of the Corpus Encoding Standard (XCES). The following three subsections describe the current status of these standards, with two additional common practices brieﬂy characterised in the subsequent subsections. 3.1

ISO TC 37 / SC 4

There are six stages of development of any ISO standard: 1) initial proposal of a new work item, 2) preparation of a Working Draft (WD), 3) production and acceptance of the Committee Draft (CD), 4) production and acceptance of the Draft International Standard (DIS), to be distributed to ISO member bodies for commenting and voting, 5) approval of the Final Draft International Standard (FDIS), which has to pass the ﬁnal vote, and 6) the publication of the International Standard (IS). Acronyms of various standards potentially applicable in NKJP, as well as their current status (as given at http://www.iso.org/) and the latest publication freely available from http://www.tc37sc4.org/, are listed in Table 1. 1

See also [3].

402

A. Przepiórkowski and P. Bański Table 1. Relevant ISO standards standard FSR FSD WordSeg1 MAF SynAF LAF

stage version available IS ISO/DIS 24610-1 2005-10-20 DIS ISO/CD 24610-2 2007-05-03 DIS ISO/CD 24614-1 2008-06-24 DIS ISO/CD 24611 2005-10-15 DIS ISO/CD 24615 2009-01-30 DIS ISO/WD 2461[2] 2008-05-12

The ﬁrst two standards are concerned with feature structure representation (FSR) and declaration (FSD). WordSeg1 deﬁnes basic concepts and very general principles of word segmentation in diverse languages. The Morphosyntactic (MAF) and Syntactic (SynAF) Annotation Frameworks are speciﬁcations of the representation of wordform and syntactic (both constituency and dependency) information, respectively. Finally, the Linguistic Annotation Framework (LAF) deﬁnes a general abstract pivot format to which all levels of linguistic information may be mapped. Currently, only FSR is an actual published standard (ISO 24610-1). 3.2

TEI

The Text Encoding Initiative was established in 1987 to develop, maintain, and promulgate hardware- and software-independent methods for encoding humanities data in electronic form (http://www.tei-c.org/). It is a de facto, constantly maintained XML standard for encoding and documenting primary data, with an active community, detailed guidelines ([4]) and supporting tools. Its recommendations for the encoding of linguistic information are limited, but it includes the ISO FSR standard for representing feature structures, which can be used to encode various kinds of information. 3.3

XCES

The Corpus Encoding Standard ([12,10]), a corpus-centred oﬀshoot of TEI, was developed within the Expert Advisory Group on Language Engineering Standards (EAGLES) project, and subsequently translated from SGML to XML ([11]). The resulting XCES speciﬁes the encoding for primary data, for morphosyntactic annotation, and for alignment of parallel corpora. It also provides general feature structure mechanisms for the representation of other levels of information. Although there are various resources and projects following XCES, including the IPI PAN Corpus of Polish (http://korpus.pl/; [19]), the standard apparently has not been modiﬁed since 2003; http://www.xces.org/ refers to old CES documentation as supporting general encoding practices for linguistic corpora and tag usage and largely relevant to the XCES instantiation. There are two sets of XML schemata, given as XML Schema (apparently last updated in 2003) and as DTD (apparently older), specifying diﬀerent XML formats.

Which XML Standards for Multilevel Corpus Annotation?

3.4

403

TIGER-XML

TIGER-XML ([16]) is a de facto standard for XML annotation of treebanks (syntactically annotated corpora). It is well documented and exempliﬁed, it has been adopted in various projects, and it was the starting point for SynAF. In this schema, each sentence is represented as a consisting of and , where is a list of erminals (with orthographic, morphosyntactic and other information represented in attributes), and is a list of syntactic nodes. Within each node, s link to immediate constituents (s or s). Additional secondary edges ( elements within ) may be used to represent co-reference information. There is a treebank search engine working on TIGER-XML corpora, TIGERSearch ([15,14]), and converters from TIGER-XML to other formats, including the PAULA format used by ANNIS2 (http://www.sfb632. uni-potsdam.de/d1/annis/) and the Poliqarp ([13,20]) format. 3.5

PAULA

PAULA (Ger. Potsdamer AUstauschformat für Linguistische Annotation; [7]), a LAF-inspired format developed within the SFB 632 project in Potsdam and Berlin, is an example of a family of general encoding standards for the annotation of multi-modal data.2 In the PAULA data model there are objects (“markables”), various types of relations between them, and features of objects. Markables may be simple spans of text () or abstract ures bearing ations to other markables. For example, a syntactic constituent with 3 immediate daughters (one word and two syntactic constituents) may be represented as follows:3

This representation closely corresponds to the following representation in TIGER-XML, though PAULA’s is a generalisation of TIGER-XML’s and may be used for the representation of various types of relations.

Additionally, elements associate markables with feature values. 2 3

See [8] for references to other such largely graph-based encodings. This is a modiﬁcation of an example from [7].

404

4

A. Przepiórkowski and P. Bański

Discussion

Of the de facto and purported standards described above, the ﬁrst to be rejected is XCES, as 1) it has speciﬁc recommendations only for the linguistic level of morphosyntactic annotation, 2) the general feature structure mechanisms envisaged for other levels are diﬀerent from FSR, an established ISO standard, 3) XCES includes no mechanisms for discontinuity, 4) or alternatives, and 5) there is a potential for confusion regarding the version of the standard. XCES was derived from TEI version P4, but it has not been updated to TEI P5 so far. Apart from the TEI-derived XCES, TEI P5 is the only standard which includes detailed speciﬁcations for the encoding of metadata and text structure, so its deployment for these levels, as well as for text segmentation into sentences, is uncontroversial. At the layers of word-level segmentation and morphosyntactic representation, the proposed ISO standards WordSeg1 and MAF are relevant. WordSeg1 provides general principles of word segmentation, and its main rule — that word segmentation should be lexicon-driven — is followed in NKJP.4 MAF oﬀers speciﬁc recommendations for the encoding — within a single XML ﬁle — of what we consider to be three layers: ﬁne-grained segmentation, morphosyntactic analysis, and syntactic words. For this reason MAF cannot be applied verbatim in the project described here, and a more general stand-oﬀ representation must be adopted. The speciﬁc XML encodings proposed in §§5.2–5.4 may be easily mapped into MAF. For the syntactic level, either the speciﬁc TIGER-XML encoding or the more general SynAF model may be employed. In fact, TIGER-XML is a concrete instantiation of SynAF. Unfortunately, TIGER-XML assumes that both terminal and non-terminal nodes are present within the same XML element, while in NKJP they should be separated, as there are two diﬀerent and potentially conﬂicting syntax-like levels (syntactic groups and named entities) that refer to the same word level. In §5.5 we propose a stand-oﬀ encoding inspired by (and mappable to) TIGER-XML, satisfying the general SynAF model. None of the above standards provides speciﬁc mechanisms for representing word senses. In §5.6, we propose encoding analogous to that of morphosyntactic information, but implementing a mechanism of referring to particular entries within a sense dictionary, reminiscent of the @entry attribute in MAF. Wherever there are no speciﬁc schemata for particular linguistic levels, general graph and feature structure representation mechanisms could be used as proposed, e.g., in LAF, and implemented, e.g., in PAULA. We follow this general approach, and the encodings proposed in the following sections are compatible with it. However, at this stage, the proposed ISO standards are still under development, with LAF and SynAF proposing only very general data models rather than speciﬁc 4

The dictionary used in the project is a new version of Morfeusz ([26]), encoding the data of the Słownik gramatyczny języka polskiego (‘Grammatical dictionary of Polish’; [24]). Occasionally, in well-deﬁned cases, this general rule is in conﬂict with the principle of bound morpheme (“If a bound morpheme is attached to a word, then the result is a word.”).

Which XML Standards for Multilevel Corpus Annotation?

405

solutions. Being aware of past eﬀorts of developing annotation schemata which would “adhere as much as possible to the proposals for the new ISO/TC 37/SC 4 standard for linguistic resources” ([18]), but which do not adhere to them anymore as those proposed standards evolved, we decided to rely on established rather than proposed standards. Two such general standards are TEI and PAULA; although the former is not generally thought of as a graph-encoding formalism, its reference mechanisms can be used to represent graphs in a way not less straightforward than that implemented in PAULA. In the end, we chose TEI P5 as the general encoding standard in NKJP also for a number of other reasons: 1) for primary data and metadata levels there is no real alternative to TEI, 2) TEI implements the ISO FSR standard, which can be used for the representation of linguistic content, as proposed in LAF (while PAULA introduces its own feature mechanism), 3) TEI is much more ﬁrmly established as a de facto standard for text encoding, with a much larger user base. This approach is radically diﬀerent from that adopted in the KYOTO project, mentioned in §1, where the approach of maximal adherence to established and proposed ISO ISO TC 37/SC 4 standards is assumed. This approach is justiﬁed to the extent that one of the main emphases of the project is the encoding of semantic dictionaries, and it relies in this regard on the established ISO 24613 standard (Lexical Markup Framework). Nevertheless, certain tensions resulting from the attempts to follow other, less developed ISO standards are visible in [1], where section 4, ﬁrst describing MAF and SynAF on over 20 pages, ends with the following statement (p. 36): We decided to remove MAF and SYNAF from the system design. Instead of that, we added to the KAF [Kyoto Annotation Framework] format some syntactic layers, thus representing among the diﬀerent KAF levels also the morphological and syntactic levels. Basic motivation for that were that MAF is not ﬁnalized and complete, and that current documents are not consistent. Moreover SYNAF contains a lot of information that we do not need and, embedding representation of data into the original text documents, it complicates the representation and manipulation of information.

5

Standards in NKJP

For reasons discussed above, TEI P5 has been adopted as the main standard in NKJP. However, TEI is a rich toolbox, providing a variety of tools to address particular problems. Whenever there is a choice, an attempt has been made to select a solution isomorphic with other proposed, oﬃcial and de facto standards. 5.1

Metadata, Primary Data and Structure

The CLARIN short guide on metadata ([5]) makes the following recommendation: We recommend using. . . (1) IMDI and its special proﬁles including TEI

406

A. Przepiórkowski and P. Bański

elements or (2) OLAC, and later adds: Also components and proﬁles will be offered that contain IMDI, TEI and OLAC speciﬁcations to take care of the already existing metadata records. Hence, the use of TEI headers is in line with current best practices, and natural for LRs otherwise represented according to the TEI Guidelines. Apart from a TEI header for each text (header.xml), there is a general TEI corpus header, describing NKJP as a whole (NKJP_header.xml). There is also no viable alternative to TEI for the representation of primary data and text structure. Texts are acquired for NKJP from a variety of sources, including previous participating corpora, publishers, Internet, media, original recordings of spontaneous conversations. They come with diﬀerent kinds of structural information and diﬀerent front and back matters. Some are divided into paragraphs or paragraph-like blocks, others into conversation turns. TEI Guidelines provide well-deﬁned elements for all these situations. TEI P5 encoding of metadata, primary data and structural information, as employed in the National Corpus of Polish, is presented in detail in [21]. The outline of text_structure.xml, containing a single text and any structural annotation, is as follows, with and (matter) elements optional:

In the case of written texts, the element contains possibly nested elements, expressing the overall structure of the text and containing

aragraphs (or paragraph-like anonymous blocks, ). For spoken data, consists of tterances. 5.2

Segmentation

Within any ann_segmentation.xml ﬁle, the element contains a sequence of

, or elements mirroring those found in the of the corresponding text_structure.xml. The parallelism is expressed via TEI @corresp attributes on these elements; their values refer to the corresponding elements in text_structure.xml. Any other structural markup is not carried over to this or other linguistic levels. Each paragraph or utterance is further divided into entences and even further into ments which deﬁne the span of each segment, by providing

Which XML Standards for Multilevel Corpus Annotation?

407

oﬀsets to an appropriate element in text_structure.xml.5 Each such element bears the implicit attribute @type="token".

5.3 Morphosyntax The overall structure of ann_morphosyntax.xml, down to the level of (also implicitly marked as @type="token"), is identical to that of ann_segmentation.xml, with each referring — via the value of @corresp — to the corresponding segment at the segmentation level. Within , however, a feature structure — encoded in conformance with the FSR ISO standard — represents information about all morphosyntactic interpretations of a given segment, as well as about the tool used to disambiguate between them and the result of the disambiguation. For example, the logical structure of the content of a representing the noun komputer (singular, inanimate masculine, nominative or accusative) may be represented as follows:6 ⎡

morph

⎤

⎢orth komputer ⎡ ⎤⎥ ⎢ ⎥ lex ⎢ ⎥ ⎢ ⎥ base komputer ⎢ ⎥ ⎢interps ⎣ ⎥ ⎦ ⎢ ⎥ ctag subst ⎢ ⎥ ⎢ 1 sg:acc:m3 ⎥ msd sg:nom:m3 ∨ ⎢ ⎥ ⎡ ⎤ ⎢ ⎥ tool_report ⎢ ⎥ ⎢ ⎢tool Anotatornia ⎥ ⎥ ⎢ ⎢ ⎥ ⎥ ⎢disamb ⎢date 2009-07-03 00:21:17⎥ ⎥ ⎣ ⎣resp PK + AA ⎦ ⎦ choice 1

Note that the names of features base, ctag and msd are taken from XCES. The value of interps may actually be a list of feature structures, representing interpretations diﬀering in base form (base) or grammatical class (ctag). In cases where interpretations diﬀer only in morphosyntactic description (msd), they are listed locally, as alternative values of msd. Hence, it is the value of msd that is used for the disambiguation information within disamb|choice. 5.4

Syntactic Words

Word segmentation in the sense of the previous two levels, as produced by a morphological analyser used in NKJP, is very ﬁne-grained: segments never contain spaces, and sometimes orthographic (“space-to-space”) words are broken into smaller segments. For this reason an additional level is needed that will contain multi-token words, e.g., analytical tense forms of verbs. It is this 5 6

Two complexities concerning alternative segmentations and information about boundedness of segments are discussed — and solutions are proposed — in [2]. In this case manual disambiguation was performed by two annotators, anonymised here as PK and AA, with the help of a tool called Anotatornia.

408

A. Przepiórkowski and P. Bański

level that corresponds most closely to MAF. However, while MAF assumes that s and s reside in the same ﬁle (with perhaps referring to primary data in a diﬀerent ﬁle), we need a stand-oﬀ encoding referring to ann_morphosyntax.xml. Down to the entence level, ann_words.xml follows the same design as other levels, and links its elements to those in ann_morphosyntax.xml, again via @corresp. Each sentence at this level is a list of ments of @type="word" covering the whole original sentence. In the default case, a ment at this level will be co-extensive with a at the lower level, but it may also correspond to a possibly discontinuous list of such token-level ments. Two diﬀerent syntactic words may also overlap, as in Bał się zaśmiać ‘(He) feared (to) laugh’, where for two inherently reﬂexive verbs, bać się ‘fear’ and zaśmiać się ‘laugh’, one occurrence of the reﬂexive marker się suﬃces. One way to represent such syntactic words in TEI is given schematically below. The feature structure contains information about the lemma and the morphosyntactic interpretation of the word, similarly to the information at the morphosyntactic levels, but without ambiguities. Segments in ann_morphosyntax.xml (and possibly syntactic words in ann_words.xml) within the given word are referenced via the element. ...

5.5

Named Entities and Syntactic Groups

Files representing the following two levels, ann_named.xml for named entities (NEs) and ann_groups.xml for syntactic groups, also have the same overall structure down to the level, but within each sentence only the information pertinent to the current level is represented, so, in particular, some elements within ann_named.xml may be empty, if the relevant sentences do not contain any named entities. Both levels refer independently to the level of syntactic words. Within ann_groups.xml, each sentence is a sequence of ments of @type="group" structured in a way analogous to the word-level elements described above: they consist of a feature structure describing the syntactic group, as in the following simpliﬁed example. Note that the @type attribute of deﬁnes the kind of relation between the node and its immediate constituent; note also that elements have @xml:id values and, hence, may be referenced from within the description of the group. ...

The representation of NEs is analogous, with the following diﬀerences: 1) the implicit value of @type is "named" instead of "group", 2) diﬀerent information is represented within the description; this includes the type of the named entity, as well as the base form of the NE, which, obviously, does not need to be a simple concatenation of base forms of words within the NE, 3) there seems to be no need for the @type attribute within . 5.6

Word Senses

Within NKJP, a limited number of semantically ambiguous lexemes will be disambiguated.7 In a manner analogous to the morphosyntactic level, each contains a sequence of token-level ments, with @corresp references to ments in ann_segmentation.xml.8 Each contains a feature structure with a reference to the appropriate sense in an external word sense inventory, e.g.:

In a way analogous to the two levels described in the preceding subsection, only those segments are represented here which were semantically disambiguated, so some elements will be empty.

6

Conclusion

For each speciﬁc TEI P5 solution presented above there are other ways of representing the same information in a way conformant with the TEI P5 Guidelines. For example, instead of recycling the element with diﬀerent @type values, TEI elements such as (for words), and (for syntactic groups), and even , , and (for various kinds of named entities) could be used at diﬀerent levels. Instead of using links, nested structures could be represented straightforwardly via the nesting of XML elements, or — much less straightforwardly — as feature structures ([25]), etc. 7 8

See also [17] in these proceedings. This is a technical decision; in the future, the word sense level may be changed to reference syntactic words rather than segments.

410

A. Przepiórkowski and P. Bański

The encoding proposed in this paper was designed with the view of maximising compatibility with other standards, whether sanctioned by ISO or de facto in use. It is directly mappable to speciﬁc encodings such as TIGER-XML and PAULA, and it is an instantiation of sometimes rather abstract models developed within ISO TC 37 / SC 4. We conjecture that — given the stability, speciﬁcity and extensibility of TEI P5 and the relative instability and generality of some of the other proposed standards — this approach is currently the optimal way of following corpus encoding standards.

References 1. Aliprandi, C., Neri, F., Marchetti, A., Ronzano, F., Tesconi, M., Soria, C., Monachini, M., Vossen, P., Bosma, W., Agirre, E., Artola, X., de Ilarraza, A.D., Rigau, G., Soroa, A.: Database models and data formats, KYOTO Deliverable NR. 1/WP NR. 2, Version 3.1, 2009-01-31 (2009) 2. Bański, P., Przepiórkowski, A.: Stand-oﬀ TEI annotation: the case of the National Corpus of Polish. In: Proceedings of the Third Linguistic Annotation Workshop (LAW III) at ACL-IJCNLP 2009, Singapore, pp. 64–67 (2009) 3. Bel, N., Beskow, J., Boves, L., Budin, G., Calzolari, N., Choukri, K., Hinrichs, E., Krauwer, S., Lemnitzer, L., Piperidis, S., Przepiórkowski, A., Romary, L., Schiel, F., Schmidt, H., Uszkoreit, H., Wittenburg, P.: Standardisation action plan for Clarin (2009), state: Proposal to CLARIN Community (August 2009) 4. Burnard, L., Bauman, S. (eds.): TEI P5: Guidelines for Electronic Text Encoding and Interchange, Oxford (2008), http://www.tei-c.org/Guidelines/P5/ 5. Component metadata: A CLARIN shortguide (2009), http://www.clarin.eu/documents 6. Standards for text encoding: A CLARIN shortguide (2009), http://www.clarin.eu/documents 7. Dipper, S.: Stand-oﬀ representation and exploitation of multi-level linguistic annotation. In: Proceedings of Berliner XML Tage 2005 (BXML 2005), Berlin, pp. 39–50 (2005) 8. Dipper, S., Hinrichs, E., Schmidt, T., Wagner, A., Witt, A.: Sustainability of linguistic resources. In: Hinrichs, E., Ide, N., Palmer, M., Pustejovsky, J. (eds.) Proceedings of the LREC 2006 Workshop on Merging and Layering Linguistic Information, pp. 14–18. ELRA, Genoa (2006) 9. ELRA: Proceedings of the Second International Conference on Language Resources and Evaluation, LREC 2000 (2000) 10. Ide, N.: Corpus Encoding Standard: SGML guidelines for encoding linguistic corpora. In: Proceedings of the First International Conference on Language Resources and Evaluation, LREC 1998, pp. 463–470. ELRA, Granada (1998) 11. Ide, N., Bonhomme, P., Romary, L.: XCES: An XML-based standard for linguistic corpora. In: LREC [9], pp. 825–830 12. Ide, N., Priest-Dorman, G.: Corpus encoding standard (1995), http://www.cs.vassar.edu/CES/, accessed on 2009-08-22 13. Janus, D., Przepiórkowski, A.: Poliqarp: An open source corpus indexer and search engine with syntactic extensions. In: Proceedings of the ACL 2007 Demo and Poster Sessions, Prague, pp. 85–88 (2007)

Which XML Standards for Multilevel Corpus Annotation?

411

14. König, E., Lezius, W., Voormann, H.: TIGERSearch 2.1: User’s Manual. Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart (2003) 15. Lezius, W.: TIGERSearch — ein Suchwerkzeug für Baumbanken. In: Busemann, S. (ed.) Proceedings der 6. Konferenz zur Verarbeitung natürlicher Sprache (KONVENS 2002), Saarbrücken (2002) 16. Mengel, A., Lezius, W.: An XML-based encoding format for syntactically annotated corpora. In: LREC [9], pp. 121–126 17. Młodzki, R., Przepiórkowski, A.: The WSD development environment. In: Vetulani, Z. (ed.) Proceedings of the 4th Language & Technology Conference, Poznań, Poland, pp. 185–189 (2009) 18. Pianta, E., Bentivogli, L.: Annotating discontinuous structures in XML: the multiword case. In: Proceedings of the LREC 2004 Workshop on XML-based Richly Annotated Corpora, pp. 30–37. ELRA, Lisbon (2004) 19. Przepiórkowski, A.: The IPI PAN Corpus: Preliminary version. Institute of Computer Science, Polish Academy of Sciences, Warsaw (2004) 20. Przepiórkowski, A.: Powierzchniowe przetwarzanie języka polskiego. Akademicka Oﬁcyna Wydawnicza EXIT, Warsaw (2008) 21. Przepiórkowski, A., Bański, P.: XML text interchange format in the National Corpus of Polish. In: Goźdź-Roszkowski, S. (ed.) The Proceedings of Practical Applications in Language and Computers PALC 2009, Peter Lang, Frankfurt am Main (2009) 22. Przepiórkowski, A., Górski, R.L., Lewandowska-Tomaszczyk, B., Łaziński, M.: Towards the National Corpus of Polish. In: Proceedings of the Sixth International Conference on Language Resources and Evaluation, LREC 2008. ELRA, Marrakech (2008) 23. Przepiórkowski, A., Górski, R.L., Łaziński, M., Pęzik, P.: Recent developments in the National Corpus of Polish. In: Levická, J., Garabík, R. (eds.) Proceedings of the Fifth International Conference on NLP, Corpus Linguistics, Corpus Based Grammar Research, Smolenice, Slovakia, November 25-27, pp. 302–309. Tribun, Brno (2009) 24. Saloni, Z., Gruszczyński, W., Woliński, M., Wołosz, R.: Słownik gramatyczny języka polskiego. Wiedza Powszechna, Warsaw (2007) 25. Witt, A., Rehm, G., Hinrichs, E., Lehmberg, T., Stemann, J.: SusTEInability of linguistic resources through feature structures. Literary and Linguistic Computing 24(3), 363–372 (2009) 26. Woliński, M.: Morfeusz — a practical tool for the morphological analysis of Polish. In: Kłopotek, M.A., Wierzchoń, S.T., Trojanowski, K. (eds.) Intelligent Information Processing and Web Mining. Advances in Soft Computing, pp. 511–520. Springer, Berlin (2006)

Corpus Academicum Lithuanicum: Design Criteria, Methodology, Application Aurelija Usoniene, Linas Butenas, Birute Ryvityte, Jolanta Sinkuniene, Erika Jasionyte, and Algimantas Juozapavicius Vilnius University, Universiteto 3, LT-01513 Vilnius, Lithuania [email protected], {erika.jasionyte,birute.ryvityte}@flf.vu.lt, {linas.butenas,algimantas.juozapavicius}@mif.vu.lt, [email protected]

Abstract. The paper is devoted to the overview of the building process of the Corpus of Academic Lithuanian (CorALit) (http://coralit.lt), a specialized, synchronic, written corpus which has been compiled at the University of Vilnius within the framework of the national high-tech development programme 20072013. The main issues in CorALit building are corpus design, encoding of textual data and grammatical annotation. The corpus design attempts to achieve representativeness which is determined in the given case by the number of research and study fields represented and the range of academic genres and text types included. The encoding of textual data has been carried out following TEI P5 guidelines for electronic texts. The paper will also look at the basic problems regarding the annotation of linguistic information. Keywords: corpus building, encoding, specialized corpus, corpus design, representativeness.

1 Introduction Language Technology (LT) provides a range of technologies which enable computers to deal with spoken and written human language. LT is a strongly interdisciplinary field, with applications in any situation where people interact with machines and with other people. Language is the natural way for humans to communicate and the larger part of information content in our computer systems is expressed in natural languages like English or Lithuanian. We need language technology to help us create documents, translate them, read them and increasingly to extract information from them without reading them. In order to address these challenges we need to develop mathematical and logical tools which are suitable for the analysis of linguistic data, collect empirical language data, i.e. large databases of written and spoken language (known as general and special corpora) and develop language processing tools to manipulate and enhance this data and conduct basic research on the technologies needed to build innovative LT applications. Therefore corpus building projects bring together researchers from different disciplines and provide conditions for their cooperation. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 412–422, 2011. © Springer-Verlag Berlin Heidelberg 2011

Corpus Academicum Lithuanicum: Design Criteria, Methodology, Application

413

1.1 The Use of Language Corpora Language corpora and language data banks have become an integral part of our lives. They serve as a source of practical information, easily accessible resource for teaching and learning, open new areas for research and function as part of research infrastructure. The availability of general, parallel and specialized language corpora opens new perspectives in human learning which also encompasses language learning and naturally Data-driven Language Learning (DDL) or corpus-based inductive learning. It allows supplementing the traditional “three Ps” model (Presentation-PracticeProduction) with the ‘three Is” model (Illustration-Interaction-Induction) [11]. Exposing the learner to large amounts of automatically generated usage examples can make learners more aware of natural language patterns [15]. Access to a specialized academic language corpus enables learners to come closer to their discipline-specific written discourses, and the more engaged with the texts they become, the more time they are willing to spend on them, a motivational fact which Mishan [14] identifies as the process of learners authenticating the corpus data as they actually become engaged in a data-driven learning-cum-research exercise. CorALit provides large resources of discipline-specific texts which could be used by learners to study discipline-specific terms as well as conventions of academic writing in Lithuanian. Electronic resources facilitate modern linguistic research, interdisciplinary studies, lexicographical practice, terminology studies in theory and practice. Corpora are particularly useful for establishing the frequency and co-occurrence of lexical and grammatical features. There are no dictionaries (cf. the CoBuild dictionaries) or grammar books of Lithuanian that take a corpus-based approach and focus on a huge amount of authentic language data in actual use. Lexicology and semantics use corpus analysis to obtain objective empirical data for the investigation of consistent semantic patterns and semantic peculiarities. Specialized corpora can be used for elucidating and comparing how different disciplines use language in their major genres [8]. One of the basic aims of the CorALit is to provide researchers with an open access to a disciplinary variety of authentic written academic language and to encourage the study of Lithuanian academic identity: to reveal presence or absence of signs of cultural resistance or adaptive attitudes in Lithuanian academic discourse, to enhance intradisciplinary and interdisciplinary research into structural, lexical, grammatical and semantic features of academic Lithuanian. 1.2 Digitalised Resources of the Lithuanian Language Currently there are just a few corpora and language data banks of the Lithuanian language, some of them are still in the process of development. First of all, Corpus of the Contemporary Lithuanian Language (http://donelaitis.lt) which comprises over 100 million words and is designed to represent current Lithuanian is worth a special mention. The corpus is compiled at the Centre of Computational Linguistics, Vytautas Magnus University in Kaunas. The major part of the collected texts comes from the press genre (general press, popular press and special press). The rest of the corpus consists of fiction, memoirs, scientific and popular literature and various official texts. Other ongoing projects include building of the Parallel Corpus (Czech-LithuanianCzech, English-Lithuanian-English) at the Centre of Computational Linguistics and

414

A. Usoniene et al.

the Corpus of Spoken Lithuanian under compilation at the Regional Studies Department (http://www.vdu.lt/LTcourses/?pg=41&menu_id=112). Another project, started in 2003 by the State Lithuanian Language Commission, is the Lithuanian Term Bank (http://www3.lrs.lt:10001/pls/tb/tb.search?p_sid=106425, which provides definitions, usage areas and sources of various terms. All of those corpora are very useful tools for the investigation of the Lithuanian language, however, they are not sufficient for extensive investigations into Lithuanian academic discourse and language. The Corpus of the Contemporary Lithuanian Language has a section of academic texts, but it is too small to adequately represent the full range of Lithuanian scientific discourse. The Term Bank does not provide usage patterns and collocations. Therefore there was a need for a specialized academic corpus that would offer a complete disciplinary variety of Lithuanian academic language and would help to meet different purposes of present-day researchers interested in academic discourse. This was one of the primary reasons for the compilation of Corpus Academicum Lithuanicum.

2 The Building of Corpus Academicum Lithuanicum The process of internationalisation and globalisation is affecting academic discourse in all European countries and there is a growing interest in academic discourse worldwide. Due to the prevailing use of English as a lingua franca in research publishing, the number of research articles published in Lithuanian has been decreasing since 1995, especially in biomedical and physical sciences. Therefore the compilation of the corpus of academic Lithuanian is of crucial importance today as it will help researchers to reveal the presence/absence of signs of cultural resistance or adaptive attitudes in academic discourse and to better understand identity forming features. Corpus Academicum Lithuanicum (CorALit) (http://coralit.lt) is a specialized, synchronic, written corpus which is under compilation at the University of Vilnius within the framework of the national high-tech development programme 2007-2013 launched by the Government of Lithuania and funded by the Lithuanian State Science and Studies Foundation (http://www.vmsfondas.lt/index.php?lang=en). The project has brought together researchers from two faculties of Vilnius University: Faculty of Philology and Faculty of Mathematics and Informatics. Since February 2010 CorALit has been providing open access to a disciplinary variety of academic language which enables researchers (i) to find out the features of professional academic Lithuanian, (ii) to discover similarities and differences between genres produced in different disciplines, (iii) to research the frequency and range of academic lexis, and (iv) it provides a rich database for future academic writing research. At the moment the corpus contains 9.2 mil. words. The main issues in CorALit building are corpus design, encoding of textual data and grammatical annotation. 2.1 Corpus Design In order to provide authentic written academic language material for intradisciplinary and interdisciplinary contrastive research into structural, lexical, grammatical and semantic features of academic Lithuanian the data for the corpus are compiled from five science areas and 47 study fields according to the classification adopted by the Ministry of Education and Science in 1998:

Corpus Academicum Lithuanicum: Design Criteria, Methodology, Application

• • • • •

415

B – Biomedical sciences (Medicine, Biology, Botany, Agronomy, Veterinary Science, etc.) H – Humanities (Architecture, Ethnology, Philosophy, Linguistics, History, Theology, etc.) P – Physical sciences (Mathematics, Astronomy, Physics, Chemistry, Geography, Informatics, etc.) S - Social sciences (Law, Political Science, Economics, Psychology, Education, Management, etc.) T – Technological sciences (Chemical Technology, Materials Science, Mechanics, Metrology, etc.)

The results of text collection for the corpus are presented in Table 1. The greatest number of texts comes from Biomedical sciences and Humanities, while the greatest number of words comes from Humanities and Technological sciences. The number of words representing the five science areas ranges from over 1.5 million words for Social sciences to 2.2 million words for Humanities. It must be noted that the figures do not include front matter (titles, author names, affiliations, e-mail addresses etc.), back matter (references, endnotes and footnotes), more or less text-external matter (tables, charts, pictures and their captions, mathematical formulae and equations, linguistic examples in a separate paragraph, block quotes). Table 1. No. of texts and words from five science areas

Science area B H P S T Total:

Entries Total 639 452 271 346 375 2083

No. of words Total 1771779 2229111 1696816 1529196 1987779 9214681

In designing the corpus a decision had to be made concerning genres and text types which would represent the academic discourse most appropriately. Recent studies in Academic English [7], [8], [17], [18] emphasise the importance of a research article. If the main public goal of the academic community is the creation of new knowledge, then research articles published in refereed journals can be considered the main genre which the academic community uses to achieve that goal. Therefore, the decision was made to represent the science areas and study fields in Lithuanian academic discourse primarily through research articles published in refereed journals. Another important academic genre is a monograph. However, the writing of a monograph requires more time and effort and this genre is used more for consolidating new knowledge and giving it public recognition. Thus, the monograph is considered the second most important academic genre. The corpus also includes abstracts and summaries of research articles. The texts themselves are very short but they perform a very important function of informing the academic community. These two subgenres employ a specific move structure and linguistic means [1]. The academic community has to undertake public evaluation of the

416

A. Usoniene et al.

new knowledge. That function is performed by the book review genre. Book reviews are published in research journals and perform the quality control function for academic publications. The book reviews are collected for the corpus as examples of the evaluative genre. Just as any other discourse community, the academic community uses certain genres (i.e. chronicles) to inform its members about important events, provide the evaluation of the contribution of a certain publication to a field of science in forewords, and express gratitude to each other for assistance or to sponsors for funding in acknowledgements. Since these genres perform the function of public communication, their language should also be represented in the corpus. The above mentioned genres and sub-genres (research articles, monographs, abstracts, summaries, book reviews, chronicles, forewords, and acknowledgements) are supposed to foster communication between specialists working in the same science area or study field. The purpose of a textbook is to disseminate new knowledge and to ensure communication between a specialist and a lay person or a novice (a student) in the field. Such communication requires a different level of abstraction, a more careful use of terms, definitions, clarifications, repetitions, etc. Therefore, textbooks will be included in the corpus as a genre with specific linguistic features. To sum up, the genres and subgenres included in the corpus will represent the main public functions of the academic discourse community: • • • •

knowledge creation and consolidation (research article, monograph) knowledge dissemination (textbook) knowledge evaluation (book review, review article, foreword) information (abstract, summary, chronicle, acknowledgements).

Due to the limited scope of the project only the genres of public communication represented by published texts are included in the corpus. The project team has considered the inclusion of genres of internal communication in academic community, such as course descriptions, study programme descriptions, grant proposals, PhD dissertations, and summaries of PhD dissertations. Some of these texts were collected during the pilot project stage in 2007. However, many researchers claim that corpus building should follow a cyclical pattern, with carefully planned pilot corpus compiled first and then tested extensively. According to Douglas Biber, empirical research carried out on the pilot corpus should reveal which direction should be taken in the further development of the corpus, which categories of corpus design are adequate and which ones should be improved [2]. Therefore, after some testing of the pilot corpus and some discussions, the project group decided to take the elitist approach and not to include unedited texts in the corpus. Table 2 presents the results to date of the collected texts representing the genres of public communication of the academic discourse community. The above mentioned genres chosen to represent the public functions carried out by the academic discourse community and linguistic means used to perform those functions should be sufficient for the study of characteristic features of academic language. However, due to very different sizes of texts it is hardly possible to balance the number of texts and the number of words representing a certain genre. The project team had to make decisions concerning the number of texts for every genre with regard to the number of words. On the other hand, the corpus is to represent the situation in real life where abstracts are much shorter than monographs. A decision was made that the corpus would provide tools enabling researchers to select and analyse texts belonging to one specific genre.

Corpus Academicum Lithuanicum: Design Criteria, Methodology, Application

417

Table 2. No. of texts and words in selected genres

Text type review article chronicle monograph foreword book review summary article textbook

No. of docs 3 106 44 21 54 59 1 651 136

No. of words 8 841 76 921 1 106 163 10 928 72 535 69 606 4 668 505 3 143 118

Average no. of words in a doc 2 947 725.67 25 140.07 520.38 1 343.24 1 179.76 2 827.68 23 111.16

The language of the texts in the corpus is not regarded simply as the authors’ own preferred language use. Since the texts belong to the sphere of public communication in the academic discourse community, their language is seen as the result of complex interactions between the authors, the guidelines for the writing of articles and editorial policies of the various journals. The degree to which the demands and changes made by referees and editors affect the language of the authors varies considerably between the disciplines as well as between the individual journals. This holds for text structure and style as well as for grammar and lexical choices. On the other hand, editorial censorship performs the function of quality control in academic writing; therefore, the inclusion of academic texts from as many publishing houses as possible enables the corpus to represent the prevailing academic writing conventions that are perceived as a norm by the Lithuanian academic community. Besides, the project team has attempted to include both single-author and multi-author articles and, for the singleauthor ones, articles written by women as well as by men. 2.2 Representativeness One of the major concerns in corpus design is that of representativeness as the essential purpose of any corpus is to represent a certain language or language variety as accurately as possible. As noted by many researchers in corpus linguistics, the concept of representativeness is difficult to define as it depends on many external and internal features. According to Biber [2], “representativeness refers to the extent to which a sample includes the full range of variability in a population”. Since a corpus is only a sample of a certain language or its specific domain or subject area, it cannot possibly cover everything that natural language offers, yet it has to be balanced in terms of range of genres included and in terms of texts, selected for every genre. The balance is achieved not by just selecting text samples of the same size, but setting proportions based on certain criteria. To achieve representativeness, corpus compilers have to consider a number of issues that directly relate to the purpose for which the corpus is intended. Some criteria are more important for general reference corpora; some are more relevant for a special purpose corpus. Lynne Bowker and Jennifer Pearson outline such general issues for special purpose corpus design as size, number of texts, medium, subject, text type, authorship, language, publication date [4]. Della Summers in her extensive overview

418

A. Usoniene et al.

of text selection criteria outlines random selection, readability of text, demographic features, influentialness and typicality of text [9]. Linked to the corpus design is the question of copyright and text availability. The optimal design of the corpus may be carefully planned and text selection criteria may be well-considered and transparent, but it might be difficult or even impossible to obtain examples of all text categories, especially in cases of building specialized corpora. The process of text collection has already revealed that the use of English as a lingua franca in academic journals published in Lithuania is increasing, e.g. it is hardly possible to find a research article on biology or chemistry written in Lithuanian. It seems that there is no absolutely objective way on how to reliably measure the representativeness of a corpus, as it depends on a number of factors and issues, some of which are often outside the control of compilers. As McEnery and Wilson aptly put it, “corpus-building is of necessity a marriage of perfection and pragmatism” [12]. One of the possible ways to test the representativeness of a corpus on lexical level is to measure the ratio between tokens (running words of the text) and types (number of different words) [12]. The corpus is broken into equal parts and each part is then tested for the number of tokens and types. When there is no obvious lexicon growth in the newly added parts of the text, the so called lexical closure or saturation is reached and the corpus may be said to be representative. Using Wordlist function in WordSmith Tools an attempt was made in 2009 to find out the degree of lexical closure of each of the five science areas. First of all, the test revealed that there was a difference in the frequency of new words encountered in separate science areas of CorALit. The field of Humanities obviously displayed a higher proportion of new words than any of the other four scientific areas. The second interesting aspect was that even at over 1.5 million words there was still an observable lexical growth in all of the science areas, with only Social sciences field slowly becoming more stabilized with regard to new words added. Of course, these were just preliminary tests, but they can give some insights about further attempts to balance and supplement the corpus. It should be noted, however, that Lithuanian is an inflectional language and since the corpus is not lemmatised, the program calculates every form of the same word as a separate type. As a result, a consistent lexicon growth that was still observed at 7 million words might also be partly due to frequently repeated different forms of the same words, for example, various cases of nouns. When the corpus is lemmatised, the type and token ratio will be more precise, as only lemmas will be calculated as separate types. Since specialized corpora are designed to represent a particular language variety or field, they reflect it in much more detail than general corpora, which aim at having a little bit of everything in their design. What is more, a special purpose corpus makes it possible to find much more of specialized linguistic features, such as lexis, grammatical constructions, and patterns of collocation which are typical of that language or language variety represented in the corpus. Finally, specialized corpora are easier to adapt for language teaching and learning. 2.3 Encoding of Textual Data Encoding of textual data starts with the conversion of text files into machine-readable files. This process may become very labour-intensive when character recognition

Corpus Academicum Lithuanicum: Design Criteria, Methodology, Application

419

problems (visual images, font types, Lithuanian characters) are encountered. The coding of texts is in broad agreement with the TEI P5 guidelines for electronic texts: textual features are marked by opening and closing tags enclosed within angle brackets, e.g. [@pavadinimas_Pradzia] and [@pavadinimas_Pabaiga]. There are, however, some differences and there is a slight variation of the TEI model developed for CorALit. The current model involves our own tags (like the ones mentioned above) which are preferable for technical reasons: text markup is done using OpenOffice as a tool, thus all markup tags cannot be in the XML format. During the “Save as TEI” phase all XML tags are treated as ordinary text and they are automatically converted to entity references (“” will be converted to "<word>"). Our own tags (like [@...]) allow us to use OpenOffice and later translate these tags into ordinary TEI tags (like “”). Reproduction of texts in the electronic environment is a complex and labourintensive task. The preparation of texts has two distinct stages: the conversion of text files into a required format and the text data processing or text coding. Since the texts from publishing houses nearly always come in *.pdf or *.p65 (Page Maker) format, they have to be converted into *.doc, *.html, *.txt or *.xml. formats to allow editing. The stages of text conversion include: i) documenting all the metadata of a text in *.pdf format (a title, publishing house, year of publication, author‘s name, etc). The table must be completed in order: a) not to lose the text metadata (bibliographical information) as the Corpus requires only the body text. ii) automatic generating of a filename (identifier), where a science area to which the text belongs, the author‘s name, the type of the publication, the year of publication, the title and the pages are indicated. This automatically generated file name is given to *.odt format documents. iii) conversion from the text in the *.pdf format to the *.odt format. So far all the text has been marked using the Select Tools command and saved in Open document type (*.odt) format (*.odt format is needed in order to change automatically into TEI format afterwards using the OpenOffice XSL-T annex). It is important to note that when converting from *.pdf format, textual information is preserved while all graphic, visual, illustrative information may be lost. Text Sources .html .pdf .doc .txt PageMaker ...

Tool for text encoding – OpenOffice Original Text in editable format .odt

encoding

Markup is applied using an interactive tags set to ease the procedure

Encoded Text .odt

Text in TEI format .tei

Fig. 1. The process of text encoding

420

A. Usoniene et al.

iv) processing of the text in the *.odt format (markup of basic structural and textual features: editorial interventions, unusual language, quotations, etc.). Two copies in the *.odt format are made: one coincides with the original in *.pdf format, i.e. the text is preserved in the original form, while the other file is created for the tagging procedure (the file names end in […]_orig and […]_red respectively). Figure 1 illustrates the steps necessary for the text preparation and encoding process. The choice of OpenOffice was determined by the following reasons: • •

•

It is free; OpenOffice document (.odt) files use the OpenDocument format (ODF) which is an XML-based file format. The XML format provides easy and clear document processing using: o built-in or optionally added OpenOffice tools (as “teioop5.jar” package from http://www.tei-c.org.uk/Software/teioo/); o external software (JAVA, C#, PHP, etc.); It has several built-in possibilities for programming interfaces (OpenOffice.org Basic, Python, JavaScript, etc.) o OpenOffice.org Basic programming language was used to create a convenient tool for further text encoding.

2.4 Automatic Encoding Corpus of Academic Lithuanian contains 9.2 mil. words. To speed up the manual encoding process of texts a dynamic neural network building framework (DNNBF) has been developed. It uses an algorithm to identify structural parts of a text such as titles, content, footnotes, etc. The purpose of this framework is to find the smallest neural network that can efficiently accomplish a given task. Currently all corpus texts are encoded manually. They are used as testing data for tuning and testing DNNBF. An encoded text must be converted to a vector format in order to be understandable to the neural network. The vector contains information about the number of words, symbols, capital letters, ending of the line and non Latin letters in a text segment. The result of 84.79% correctly identified segments is quite good even though the network is taught with a very small data set (Table 3). The more data is given for teaching, the bigger the amount of correctly identified content parts is, but the amount of correctly identified titles decreases. The decrease is determined by the increasing variety of title formats. In future this aspect can be improved by taking only the largest cluster of the similarly formatted titles. Table 3. DNNBF correctness Segments For teaching For testing 10 10000 100 10000 1000 10000

Total 84.79 % 97.91 % 97.91 %

Correctly identified Content parts 83.08 % 96.79 % 97.91 %

Titles 81.8 % 53.6 % 0%

Corpus Academicum Lithuanicum: Design Criteria, Methodology, Application

421

2.5 Perspectives Future work on the corpus will include the annotation of linguistic information at different levels: morphological, syntactic and semantic. However, the project group will face quite a few problems before being able to accomplish this task as there are not many annotation tools designed for the Lithuanian language which has free word order and is highly inflected, i.e. new words are easily formed by inflectional affixation. One of the first attempts at corpus annotation was the morphological analizer, lemmatizer and tagger Lemuoklis [22] which offered a combination of properties with respect to 13 categories: part of speech, aspect, reflexiveness, voice, mood, tense, group, degree, definiteness, gender, number, case and person. Lemuoklis, as it was originally designed, was mostly based on existing grammars and dictionaries and did not tackle morphological ambiguity. The accuracy of the results of automatic extraction of linguistic units can be enhanced if implicit information is made explicit. The project group is considering the recognition of tokens and sentences, a lexical analysis which assigns lemmas and parts of speech and the disambiguation of the part of speech tags. Acknowledgments. The project team would like to express their gratitude to Hilary Nesi (Coventry University, UK) and Rūta Marcinkevičienė (Research Council of Lithuania) for their kind assistance and advice in the process of creating Corpus Academicum Lithuanicum (CorALit).

References 1. Bhatia, V.K.: Analysing Genre: Language Use in Professional Settings. Longman, London (1993) 2. Biber, D.: Representativeness in Corpus Design. J. Literary and Linguistic Computing 8(4), 243–257 (1993) 3. Biber, D., Conrad, S., Reppen, R.: Corpus Linguistics: Investigating Language Structure and Use. Cambridge University Press, Cambridge (1998) 4. Bowker, L., Pearson, J.: Working with Specialized Language: A practical guide to using corpora. Routledge, New York (2002) 5. Čermak, F.: Today‘s Corpus Linguistics: Some Open Questions. J. International Journal of Corpus Linguistics 7(2), 265–282 (2002) 6. Facchinetti, R.: Drawbacks and Pitfalls of Machine – Readable Texts for Linguistic Research. J. International Journal of Corpus Linguistics 3(2), 211–228 (1998) 7. Flowerdew, J. (ed.): Academic Discourse. Longman, London (2002) 8. Hyland, K.: Disciplinary Discourses: Social Interactions in Academic Writing. Longman, London (2000) 9. Kennedy, G.: An Introduction to Corpus Linguistics. Addison Wesley Longman Limited, Amsterdam (1998) 10. Kralik, J., Šulc, M.: The Representativeness of Czech corpora. J. International Journal of Corpus Linguistics 10(3), 357–366 (2005) 11. McCarthy, M.: Spoken Language and Applied Linguistics. Cambridge University Press, Cambridge (1998)

422

A. Usoniene et al.

12. McEnery, T., Wilson, A.: Corpus Linguistics. an Introduction. Edinburgh University Press, Edinburgh (2001) 13. McEnery, T., Xiao, R., Tono, Y.: Corpus-Based Language Studies. Routledge, New York (2006) 14. Mishan, F.: Authenticating Corpora for Language Learning: a Problem and Its Resolution. J. ELT Journal 58(3), 219–227 (2004) 15. Schmied, J.: Translation Corpora in Contrastive Research, Translation and Language Learning. J. Trad.Term. 10, 83–115 (2004) 16. Summers, D.: Computer Lexicography: the Importance of Representativeness in Relation to Frequency. In: Thomas, J., Short, M. (eds.) Using Corpora for Language Research. Studies in the Honour of Geoffrey Leech. Longman, Harlow (1996) 17. Swales, J.M.: Genre Analysis: English in Academic and Research Settings. Cambridge University Press, Cambridge (1990) 18. Swales, J.M.: Research Genres: Explorations and Applications. Cambridge University Press, Cambridge (2004) 19. Teubert, W., Čermakova, A.: Corpus Linguistics. A Short Introduction. Continuum (2007) 20. Tognini Bonelli, E.: Corpus Linguistics at Work. John Benjamins Publishing Company, Amsterdam (2001) 21. Williams, G.: In Search of Representativity in Specialized Corpora. J. International Journal of Corpus Linguistics 7(1), 43–64 (2002) 22. Zinkevicius, V.: Lemuoklis – morfologinei analizei. J. Darbai ir Dienos 24, 245–273 (2000) 23. Corpus Academicum Lithuanicum (CorALit), http://coralit.lt 24. Corpus of the Contemporary Lithuanian Language, http://donelaitis.lt 25. Corpus of Spoken Lithuanian, http://www.vdu.lt/LTcourses/?pg=41&menu_id=112 26. Lithuanian Term Bank, http://www3.lrs.lt:10001/pls/tb/tb.search?p_sid=106425

The EM-Based Wordnet Synsets Annotation of NP/PP Heads Elżbieta Hajnicz Institute of Computer Science, Polish Academy of Sciences ul. Ordona 21, 01-237 Warsaw, Poland [email protected]

Abstract. This paper contains an application of the EM selection algorithm to semantic annotation of NP/PP heads by means of wordnet synsets. Firstly presented are the preparation of a corpus to be semantically annotated and the wordnet on which the annotation is based. Next, the process of semantic annotation is discussed. Finally, its results are evaluated and compared with the well known solution proposed by Resnik. Keywords: corpus linguistics, word sense disambiguation, wordnet, Polish language.

1

Introduction

The main goal of our work is to enrich a valence dictionary of Polish verbs by adding semantic information. This information may be represented by means of wordnet synsets. The plain syntactic valence dictionary is a collection of predicates (here: verbs) provided with a set of verb frames. Verb frames consist of syntactic slots that represent phrases occurring in the corresponding position in a sentence. Thus, our goal is to provide syntactic slots (here: NPs/PPs) with a list of appropriate semantic classes for the corresponding nouns. In order to automatically acquire semantic information for a syntactic valence dictionary, we need a large treebank where all NP/PP semantic heads are semantically annotated. In [1] we presented an application of the EM selection algorithm to selecting the most probable semantic categories for each NP/PP head in a clause. In this paper a generalisation of the procedure for the entire wordnet hypernymy hierarchy is presented. Our problem intersects with the Word Sense Disambiguation (WSD) task [2]. However, contrary to the typical WSD task, we are interested in the most general sense of a noun (i.e., including senses of its hypernyms) that is adequate for a particular context (a clause).

2

Data Resources

Our main resource was the IPI PAN Corpus of Polish written texts [3], referred to as Kipi. From this corpus, we selected a small subcorpus, referred to as SemKipi, Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 423–434, 2011. c Springer-Verlag Berlin Heidelberg 2011

424

E. Hajnicz

containing 195 042 sentences. Selected sentences contain at least one verb from a preselected set of verbs. Details of the SemKipi creation can be found in [1]. Sentences from SemKipi were parsed with the Świgra parser [4, 5] based on the metamorphosis grammar GFJP [6]. The parser was provided with the valence dictionary prepared especially for our task, based on Świdziński’s ([7]) valence dictionary augmented with automatically created valence dictionary by Dębowski [8]. The dictionary entries of verbs preselected for the experiment were carefully elaborated. Next, parsing trees of particular clauses were identiﬁed in parsing trees of each sentence and reduced to their ﬂat forms representing only arguments of a verb (i.e., the subject and complements included in corresponding valence frames). As a result, we obtained reduced parses of a clause composed of a verb and a set of slots. Świgra tends to produce large parse forests. The number of reduced parses of a sentence is much smaller than the number of entire parse trees, the more so as we have considered only actual arguments of verbs (without adjuncts). They were disambiguated by means of the EM selection algorithm proposed by Dębowski [8] for the task of creating a syntactic valence dictionary. The whole process is presented in [1]. In order to prepare an initial sense annotation for NPs/PPs semantic heads (which would be later automatically disambiguated), we used the Polish WordNet [9, 10, 11], called Słowosieć (English acronym PlWN). PlWN is a network of lexical-semantic relations, an electronic thesaurus with a structure modelled on the Princeton WordNet [12] and the EuroWordNet project [13]. PlWN describes the meaning of a lexical unit (LU) of one or more words by placing this unit in a network which represents such relations as synonymy, hypernymy, meronymy, etc. For the construction of the semantic valence dictionary only the part of PlWN describing relations between nouns was needed. We have focused on the synonymy represented by synsets and on hypernymy represented by a directed acyclic graph. Formally, let H be a hipernymy relation, C be the set of all its classes (synsets). Then Cc↑ = {c} ∪ {c : H (c, c )}, Cc↓ = {c} ∪ {c : H (c , c)} are sets of hipernyms and hiponyms of c, respectively. Moreover, let cw = {c: sw ∈ c} be the set of all synsets containing senses (lexical units) sw of a word w. In the present experiment the version of PlWN containing 15870 LUs representing 11270 nouns and collected in 11780 synsets was used. Synsets were linked by 12550 hypernymy relation arcs. Each LU and each synset has its unique identiﬁer (a natural number). Before the entire net was constructed, the set of LUs was divided according to the predeﬁned set of 25 general semantic categories. The process of the manual creation of the net was performed to the large extent w.r.t. this division. Thus, all synsets contain LUs of the same category. To a large extent this concerns hypernymy as well (most exceptions concern synsets with several hypernyms). In Table 1 the list of LUs representing synsets positioned on the top of the hypernymy hierarchy together with the corresponding semantic categories.

The EM-Based Wordnet Synsets Annotation of NP/PP Heads

425

Table 1. The set of tops in the hypernymy hierarchy correlated with the predeﬁned set of semantic categories in the Polish WordNet category top synsets category act

action activity animal animal artifact concept medium net thing attribute detail feature body body corpse gene meat organ tissue cognition cognition destiny exercise idea intelligence psyche

top synsets

communication code communication etnolect information sound event event feeling feeling food drink food group fauna ﬂora group set location container line part place space motive duty goal position solution object object person person

category

top synsets

phenomenon nature phenomenon plant plant plant part possession business economics possession process process quantity degree quantity value relation connection domain institution relation team shape shape state state substance substance sbst state sbst type time course period phase

There exists another Polish wordnet [14]. Since we do not use the internal structure of wordnet data, we can easily adapt it to our algorithms. The only problem would be caused by manually prepared data used for evaluation.

3

Semantic Annotation

In [1] the process of semantic annotation of verb arguments by means of semantic categories was presented. The EM selection algorithm [8] was adapted for the WSD task. In this paper the application of this method to the entire PlWN structure is discussed. We start with assigning a list of synsets that contain an NP/PP head lemma. However, in contrast to the usual WSD task, we are interested in the most general sense of a noun that is adequate in a particular context. Thus, the list is extended with all hypernyms of its elements. An example of a clause together with a valence schema selected for it, a corresponding reduced parse, and lists of senses assigned for the arguments is presented in (1).

426

E. Hajnicz

The clause mężczyzna widzi w kobiecie anioła contains two NPs mężczyzna (a man) and anioł (an angel ) and one PP w kobiecie (in a women) predicated by the verb widzieć (to see). The syntactic frame chosen for the clause is presented (in brackets) together with the corresponding reduced parse. Syntactic information about arguments is augmented with a list of corresponding synsets’ identiﬁers. Translations of LUs belonging to those synsets can be found in Table 2. The clause is not ambiguous: a single lexical unit (framed in (1)) is assigned to each noun. However, due to the hypernymy hierarchy, lists of synsets are multi-element even in this simple case. The appropriate part of the PlWN is presented in Figure 1. (1) % ’Wspomniała pani, że mężczyzna widzi w kobiecie anioła.’ (You have mentioned that a man sees an angel in a woman.) 4-9 widzieć aff:fin:sg: :ter [4-5:np:mężczyzna:sg:nom:m1:ter:: 5995 6047 6776, 6-8:prepnp:w:kobieta:loc:: 6047 6129 6776, 8-9:np:anioł:sg:acc:m12:ter:: 66 67 5908 6045 6047 6771 6778]

Table 2. Lists of lexical units belonging to synsets having identiﬁers presented in (1) syns. id. list of LUs of a synset 66 67 5908 5995 6045

angel, good person angel, good spirit supernatural being man being (creature)

syns. id. list of LUs of a synset 6047 6129 6771 6776 6778

person woman person positively judged person w.r.t. sex person w.r.t. his/her features 6047

6045

H HH

6778 5908 6771 67

6776

HH

5995

6129

66 Fig. 1. The part of PlWN hierarchy for example (1)

Next, we split the reduced parse into syntactic-semantic valence frames. Thus, we obtain tuples in which every NP/PP has only one synset assigned. All pronouns obtain an artiﬁcial sense pron, represented by artiﬁcial synset 0 with no hyponyms and hypernyms. The disambiguation process consists in selecting (using the EM algorithm) the most probable frames. So, the reduced parse of the sentence (1) after splitting transforms into the 3 × 3 × 7 = 63 frames, some of them listed in (2).

The EM-Based Wordnet Synsets Annotation of NP/PP Heads

427

(2) % ’Wspomniała pani, że mężczyzna widzi w kobiecie anioła.’ acc: 66, nom: 5995, w loc: 6047 acc: 66, nom: 5995, w loc: 6129 acc: 66, nom: 5995, w loc: 6776 acc: 66, nom: 6047, w loc: 6047 acc: 66, nom: 6047, w loc: 6129 acc: 66, nom: 6047, w loc: 6776 acc: 66, nom: 6776, w loc: 6047 acc: 66, nom: 6776, w loc: 6129 acc: 66, nom: 6776, w loc: 6776 acc: 67, nom: 5995, w loc: 6047 acc: 67, nom: 5995, w loc: 6129 acc: 5908, nom: 5995, w loc: 6047 acc: 5908, nom: 5995, w loc: 6776 acc: 6045, nom: 5995, w loc: 6047 acc: 6045, nom: 6047, w loc: 6047 acc: 6047, nom: 5995, w loc: 6129 acc: 6047, nom: 6047, w loc: 6047 + acc: 6771, nom: 5995, w loc: 6047 acc: 6771, nom: 6776, w loc: 6129 acc: 6778, nom: 5995, w loc: 6047 acc: 6778, nom: 6776, w loc: 6776 In this way we obtain a set of observations O = {s, vs , gs , Fs }, where s ∈ S is a clause, vs ∈ V is the verb predicating s and gs = r1 , . . . , rK is its syntactic schema. Finally, Fs ⊆ F is the set of frames attached to s, where f ∈ Fs and f = c1 , . . . , cK , with c ∈ C being synsets. The EM algorithms were performed for each pair v, g separately, hence we can consider Ov,g ≈ {s, Fs } with vs = v, gs = g. 3.1

The EM Selection Algorithm

In [1] three versions of the algorithm were proposed and compared. First of them, called EM-whole, treats each frame as a whole, and it is a direct application of Dębowski’s [8]. He considers the following statistical task. Let Z1 , . . . , ZM , with Zi : Ω −→ J be a sequence of discrete random variables and let Y1 , . . . , YM be a random sample of sets, where every Yi : Ω −→ 2J \ ∅ satisﬁes Zi ∈ Yi . The objective is to guess the conditionally most likely values of Zi . The EM algorithm for this task comes down to iterating: (n)

pj,i =

⎧ ⎪ ⎨ ⎪ ⎩

(n)

0

pj

j ∈Yi

(n)

pj

for j ∈ Yi , else,

(3)

428

E. Hajnicz (n+1)

pj

=

M 1 (n) p , M i=1 j,i

(4)

(n)

with p1j = 1. In this iteration, pj,i is the conditional probability of j ∈ J as an element of Yi . We can apply this for WSD task letting Yi = Fs , j = f and M = |Ov,r |. EM-indep is based on the assumption that senses of arguments occur in a clause independently. It was elaborated to manage the problem of sparseness K of data. So, let Z i = Zi,1 , . . . , Zi,K satisfy P (Z i ) = P (Zi,k ) and let j = k=1

j1 , . . . , jK ∈ J = J K . Namely, we intend to guess the most likely values of Z i on the basis of distribution (j) instead of distribution (j). The EM algorithm for this task comes down to iterating: ⎧ K ⎪ ⎪ (n) ⎪ pjk ⎪ ⎪ ⎪ ⎪ ⎨ k=1 for j ∈ Yi , (n) K pj,i = (5) (n) ⎪ pj ⎪ ⎪ k ⎪ ⎪ j ∈Yi k=1 ⎪ ⎪ ⎩ 0 else, (n+1)

pj

=

M K 1 (n) pj,i 1{j} (jk ), KM i=1 j

(6)

k=1

with p1j = 1, where 1A is a characteristic function of a set A. To apply this for the WSD we set Y i = Fs , j = f and jk = ck . Again M = |Ov,r |. Assuming that semantic interpretations of arguments in verb frames are independent is not justiﬁed. Therefore, an intermediate algorithm called EM-incr was elaborated. It incrementally applies EM-whole for all subframes of a particular number of arguments (starting from 2 of them), ﬁnds most probable ones for each clause, and extends them with one argument not considered yet, producing a set of subframes for next EM-whole. In example (2), the frame selected by the EM-indep algorithm is marked by + symbol. Observe that this is the most general sense person in the case of every slot. EM selection algorithms assign probabilities to frames from Fs . Let Fsmax ⊆ Fs be the most likely ones. All hypernyms of each synset always co-occur with it. Thus, the frequency of synsets increases according to hypernymy relation. Hence, if there exist f, f ∈ Fsmax such that ck ∈ Cc↑k for each k, then there exists no f ∈ Fv,g with ck ∈ Cc↓ (except ck itself). Therefore, less general frames ˇ c1 , . . . , cˇn ∈ k Fsmax are ﬁnally selected from the results of the entire EM algorithm. For the “traditional” WSD, we can select F˜smax = {f˜ = ˜ cw1 , . . . , c˜wn : c˜wk ∈ cwk ∧˜ cwk ∈ Cc↓k } (actually, the ones with maximal their own probability calculated by an EM algorithm). The results of such a modiﬁcation of the EM algorithms will be called “word” methods in contrast with “hypernymy” methods.

The EM-Based Wordnet Synsets Annotation of NP/PP Heads

3.2

429

Related Works

The method of using the relationship between a predicate and its argument in order to disambiguate the sense of the latter was proposed by Resnik [15, 16]. He observes that selection preferences can be expressed in a diﬀerence between the prior probability p(c, r) of occurrence of words of a particular class c at slot r with the posterior probability p(c|v, r) of its co-occurrence with a particular verb v. He deﬁnes a selectional association A: p(c|v, r) p(c, r) A(v, c, r) = . p(c |v, r) p(c |v, r) log p(c , r) p(c|v, r) log

(7)

c

McCarthy [17] and Ribas [18] criticise the way Resnik estimates probabilities indispensable for calculating the selectional association. We use the method suggested in [19]: f(w) f(c) f(c) = , pˆ(c) = , |c | w f(c ) ↓ w∈c c ∈Cc

c

f(w, r) f(c, r) = , |cw | ↓ c ∈Cc w∈c

f(v, c, r) =

f(v, w, r) , |cw | ↓

c ∈Cc

f(c,r) pˆ(c, r) = , f(c , r ) c r f(v, c, r) f(v , r ) pˆ(c|v, r) =

w∈c

f(v, r)

v r v

c

f(v , c , r )

,

r

where f are frequencies in O. The idea is to ﬁnd a synset c ∈ cw with one of its hypernyms having the maximal selectional association with v, r. Thus, we compute AH (v, c, r) = max A(p, c , r), c ∈Cc↑

Cw = arg max AH (v, c, r). c∈cw

(8) (9)

There could exist several cw ∈ Cw . First, the maximal value of A could be obtained for diﬀerent c . Second, several cw ∈ Cw could have a common hypernym, in particular the one with the maximal A. In such cases we suggest to chose ones having maximal A by their own: max Cw = arg max A(p, c, r). c∈Cw

(10)

Resnik considered only one argument at once, whereas in the EM selection based method the whole predicate-argument structure of a clause is disambiguated.

430

E. Hajnicz

In general, max

K

f (xk ) =

k=1 max

K

max f (xk ) does not hold for any function f .

k=1 max cmax w1 , . . . , cwK

max However, since f = ∈ Fs for cmax ∈ Cw , we have wk k K K max A(v, ck , rk ) = max A(v, ck , rk ). f ∈Fs

k=1

k=1

ck ∈Cwk

In order to compare Resnik’s approach with the EM-based approach, we need to include hypernyms of lexical units to the set of potential results: Chmax = arg w

max

c∈cw

c ∈Cc↑

A(p, c , r).

(11)

Other methods based on wordnet hierarchy together with corpus based frequencies were reported in [18, 20].

4

The Experiment

4.1

Manually Annotated Data for an Evaluation of the Algorithm

In order to evaluate the algorithms, a small subcorpus of SemKipi was syntactically and semantically annotated by a group of linguists. 240 sentences for each of 32 preselected verbs were selected randomly from SemKipi.1 The linguists performed three diﬀerent tasks: 1. a correction of morphosyntactic tagging (tagger errors), 2. a division of sentences into phrases, i.e., pointing out their boundaries and syntactic and semantic heads, 3. an assignment of a single PlWN semantic category to each noun in a sentence.2 Sentences for manual annotation were selected before SemKipi was parsed. Unfortunately, only 43% of manually annotated sentences were accepted by Świgra (the problem of coverage of Świgra on sentences from SemKipi is discussed in [21]). In order to enlarge the test set and to minimise the inﬂuence of errors resulting from the preprocessing phase, we extended the set of automatically preprocessed sentences with manually annotated ones. If a sentence belongs to both sets, the manually annotated version was chosen. Manually annotated semantic categories of nouns were certainly deleted. The results of the manual annotation were transformed to the format of Świgra post-processing (presented in (1)). 4.2

Eﬃciency of the Algorithm

In order to disambiguate semantic concepts, the algorithms have to reduce the number of concepts assigned to a noun. We call this feature the eﬃciency of an algorithm. 1 2

More precisely, only single-verb sentences were chosen in this sampling. During manual annotation, the entire PlWN net was not available.

The EM-Based Wordnet Synsets Annotation of NP/PP Heads

431

Table 3. Eﬃciency of the algorithms synsets

selected

transformed

h y p e r

semcats

source EM-whole EM-incr EM-indep Resnik

1.715 1.036 1.040 1.002 ——

1.840 1.042 1.046 1.002 ——

6.472 1.417 1.343 1.037 1.147

6.543 1.478 1.393 1.042 1.167

—— 1.067 1.058 1.005 ——

—— 1.076 1.066 1.005 ——

1.838 1.044 1.038 1.003 1.109

1.962 1.051 1.045 1.003 1.125

w o r d s

algorithm

source EM-whole EM-incr EM-indep Resnik

—— —— —— —— ——

—— —— —— —— ——

2.183 1.249 1.242 1.196 1.127

2.365 1.284 1.277 1.224 1.144

—— 1.182 1.178 1.072 1.089

—— 1.207 1.203 1.082 1.101

1.779 1.048 1.043 1.006 1.019

1.893 1.054 1.049 1.007 1.022

In Table 3, we present the mean of the number of semantic concepts assigned to occurrences of nouns by all algorithms, including source data. The column semcats contains data for the algorithms run on semantic categories, whereas the column synsets contains data for the algorithms run on entire wordnet synsets. The results of selection of less general synsets are presented in column selected, and the results of transformation of selected synsets to their semantic categories are shown in column transformed. For the algorithms, which results were limited to synsets containing lexical units of actual NP/PP heads, the column selected means selecting the most probable synsets. Observe that the mean of semantic categories obtained after the transformation of source data is a bit larger than the mean of categories directly assigned. The reason is that a synset and its hypernym could be diﬀerently categorised. Each column is divided into two, the ﬁrst concerning all senses and the second calculated without pronouns, which always have a single sense, namely pron. This leads to the decrease of the mean. The experiment presented in [1] was performed with pronouns distributed among all 25 semantic categories, hence the mean calculated for all nouns is larger than calculated without pronouns. The median is always 1, even for the source data for categories. This does not concern the source data for synsets, for which the median is 5 (2 without hypernyms). Eﬃciency of all the algorithms is high, especially when they are performed on synsets (and while considering hypernyms). The best one is EM-indep, about one order of magnitude better than the other two EM algorithms.3 Observe that EM-whole is more eﬃcient for semantic categories whereas EM-incr is more eﬃcient for synsets. Note also that selection of less general synsets decreases the mean about 1 order of magnitude. Resnik’s method is less eﬃcient than EMindep. Note that the ﬁrst one is most eﬃcient while working on direct synsets of words (it is the only algorithm for which eﬃciency is better for words than for hypernyms), but after the transformation to semantic categories this relationship changes, which means that EM-indep results are more homogeneous. 3

Since the mean is always less than 1.5, the comparison is made only for its fraction part, the more so as the best possible eﬃciency is 1.

432

E. Hajnicz Table 4. Results of evaluation of the algorithms

w o r d s

h y p e r n y ms

algorithm data set c-corr n-corr

4.3

acc

prec

rec

F

EM-whole semcat synset selected EM-incr semcat synset selected EM-indep semcat synset selected Resnik synset

61.36 58.42 58.63 61.74 58.30 58.46 60.40 57.49 57.52 55.59

76.53 74.05 74.37 76.57 74.37 74.64 77.63 75.55 75.59 71.84

70.86 69.60 70.17 71.06 69.91 70.42 71.98 71.12 71.23 66.68

75.44 72.56 73.22 75.56 73.09 73.69 77.63 75.44 75.59 70.82

79.70 78.35 78.32 79.04 77.95 77.92 78.01 76.04 76.03 77.77

77.51 75.34 75.69 77.68 75.44 75.75 77.82 75.74 75.81 74.13

EM-whole synset selected EM-incr synset selected EM-indep synset selected Resnik synset selected

59.87 59.95 59.74 59.83 59.12 59.22 57.96 57.98

74.98 75.01 75.28 75.37 76.33 76.54 75.47 75.51

71.02 71.10 71.30 71.36 72.35 72.48 70.30 70.32

73.82 73.93 74.30 74.39 76.35 76.56 75.20 75.24

79.22 79.18 78.85 78.82 77.31 77.23 77.01 76.99

76.42 76.47 76.51 76.54 76.82 76.89 76.10 76.11

Evaluation of the Algorithm

For the sake of evaluation a small subcorpus HandKipi of SemKipi containing 5634 simple (single verb) sentences manually annotated with verb arguments boundaries and syntactic and semantic heads and semantic categories of nouns (cf. [1]). As a consequence of limited manual annotation in HandKipi, we cannot evaluate actual results of tuples of synsets selection, since we do not have data to compare with. Instead, we reduce synset annotation to corresponding semantic categories. Thus, we can only appraise whether we gained or lost some knowledge. In Table 4, the results of evaluation of all the algorithms are presented. Again, semcat means running algorithm on semantic categories, synsets means running the algorithm on wordnet synsets, whereas selected means evaluating results after less general synsets selection. c-corr means correctness calculated for whole clauses (i.e., all slots should have properly assigned senses) and n-corr means usual correctness calculated for single nouns. The results of evaluation are very similar for all algorithms. EM-indep gives the best results, except clause correctness and recall, which is coherent with Table 3. EM-incr is a bit better than EM-whole. All the algorithms show the best results, when they are performed on semantic categories (where the task is easier); selecting less general synsets helps a bit. This is not a really important result; the reason is that the linguists used only the semantic categories of actual nouns in annotated sentences, so semantic categories of their hypernyms worsen the evaluation results. Resnik’s method is evidently the worst one. Performed on direct word senses, EM-indep gives again the best results, and Resnik’s method gives results very similar to the ones given by EM-based methods.

The EM-Based Wordnet Synsets Annotation of NP/PP Heads

5

433

Conclusions

In the paper we presented an adaptation of EM-based disambiguation applied to annotation of NPs/PPs with wordnet senses. The results were evaluated after transformation to semantic categories of selected synsets. The results obtained for synsets were about 2 percentage points worse. This is probably the cost of more precise information. The best results were obtained by the EM-indep algorithm. However, it selects top synsets in 97.9% cases, whereas EM-whole— in 84.8% cases and EM-incr in 87.4% cases. Thus, evaluation on sentences manually annotated with entire synsets can give diﬀerent and more reliable results. On the other hand, Resnik’s method selects top synsets only in 29.9% cases, which means that it is not the best method for semantic valence dictionary creation. For the experiment presented in [1] the correctness (calculated only for semantic categories) was substantially better. However, the experiment was performed with pronouns distributed among all semantic categories. Moreover, the set of categories assigned to each noun has changed during the entire net construction, which could inﬂuence the manual annotation process as well. Thus, these results are incomparable. Acknowledgements. This paper is a scientiﬁc work supported within the Ministry of Science and Education project No. N N516 0165 33.

References [1] Hajnicz, E.: Semantic annotation of verb arguments in shallow parsed Polish sentences by means of EM selection algorithm. In: Marciniak, M., Mykowiecka, A. (eds.) Aspects of Natural Language Processing. LNCS, vol. 5070, pp. 211–240. Springer, Heidelberg (2009) [2] Agirre, E., Edmonds, P. (eds.): Word Sense Disambiguation. Algorithms and Applications. Text, Speech and Language Technology, vol. 33. Springer, Dordrecht (2006) [3] Przepiórkowski, A.: The IPI PAN corpus. Preliminary version. Institute of Computer Science, Polish Academy of Sciences, Warsaw (2004) [4] Woliński, M.: Komputerowa weryﬁkacja gramatyki Świdzińskiego. PhD thesis, Institute of Computer Science, Polish Academy of Sciences, Warsaw (2004) [5] Woliński, M.: An eﬃcient implementation of a large grammar of Polish. In: Vetulani, Z. (ed.) Proceedings of the 2nd Language & Technology Conference, Poznań, Poland, pp. 343–347 (2005) [6] Świdziński, M.: Gramatyka formalna języka polskiego. Rozprawy Uniwersytetu Warszawskiego. Wydawnictwa Uniwersytetu Warszawskiego, Warsaw (1992) [7] Świdziński, M.: Syntactic Dictionary of Polish Verbs. Uniwersytet Warszawski / Universiteit van Amsterdam (1994) [8] Dębowski, Ł.: Valence extraction using the EM selection and co-occurrence matrices. Language Resources & Evaluation 43, 301–327 (2009) [9] Piasecki, M., Szpakowicz, S., Broda, B.: A Wordnet from the Ground Up. Oﬁcyna Wydawnicza Politechniki Wrocławskiej, Wrocław (2009)

434

E. Hajnicz

[10] Derwojedowa, M., Piasecki, M., Szpakowicz, S., Zawisławska, M., Broda, B.: Words, concepts and relations in the construction of Polish WordNet. In: Tanacs, A., Csendes, D., Vincze, V., Fellbaum, C., Vossen, P. (eds.) Proceedings of the Global WordNet Conference, Seged, Hungary (2008) [11] Derwojedowa, M., Szpakowicz, S., Zawisławska, M., Piasecki, M.: Lexical units as the centrepiece of a wordnet. In: Kłopotek, M.A., Przepiórkowski, A., Wierzchoń, S.T. (eds.) Proceedings of the Intelligent Information Systems XVI (IIS 2008). Challenging Problems in Science: Computer Science. Academic Publishing House Exit, Zakopane (2008) [12] Fellbaum, C. (ed.): WordNet — An Electronic Lexical Database. MIT Press, Cambridge (1998) [13] Vossen, P. (ed.): EuroWordNet: a multilingual database with lexical semantic network. Kluwer Academic Publishers, Dordrecht (1998) [14] Vetulani, Z., Walkowska, J., Obrębski, T., Konieczka, P., Rzepecki, P., Marciniak, J.: PolNet — Polish WordNet project algorithm. In: Vetulani, Z. (ed.) Proceedings of the 3rd Language & Technology Conference, Poznań, Poland, pp. 172–176 (2007) [15] Resnik, P.: Selection and Information: A Class-Based Approach to Lexical Relationships. PhD thesis, University of Pennsylvania, Philadelphia, PA (1993) [16] Resnik, P.: Selectional preference and sense disambiguation. In: Proceedings of the ACL Workshop on Tagging Text with Lexical Semantics, Why, What and How?, Washington, DC, pp. 52–57 (1997) [17] McCarthy, D.: Lexical Acquisition at the Syntax-Semantics Interface: Diathesis Alternations, Subcategorization Frames and Selectional Preferences. PhD thesis, University of Sussex (2001) [18] Ribas, F.: On Acquiring Appropriate Selectional Restrictions from Corpora Using a Semantic Taxonomy. PhD thesis, University of Catalonia (1995) [19] Li, H., Abe, N.: Generalizing case frames using a thesaurus and the MDL principle. Computational Linguistics 24(2), 217–244 (1998) [20] Carroll, J., McCarthy, D.: Word sense disambiguation using automatically acquired verbal preferences. Computers and the Humanities. Senseval Special Issue 32(1-2), 109–114 (2000) [21] Hajnicz, E., Woliński, M.: How valence information inﬂuences parsing Polish with Świgra. In: Kłopotek, M.A., Przepiórkowski, A., Wierzchoń, S.T., Trojanowski, K. (eds.) Recent Advances in Intelligent Information Systems. Challenging Problems in Science: Computer Science, pp. 193–206. Academic Publishing House Exit, Warsaw (2009)

Unsupervised Word Sense Disambiguation with Lexical Chains and Graph-Based Context Formalization Radu Ion and Dan Ştefănescu Research Institute for Artificial Intelligence of the Romanian Academy [email protected], [email protected]

Abstract. This paper presents an unsupervised word sense disambiguation (WSD) algorithm that makes use of lexical chains concept [6] to quantify the degree of semantic relatedness between two words. Essentially, the WSD algorithm will try to maximize this semantic measure over a graph of content words in a given sentence in order to perform the disambiguation. Keywords: lexical chains, graph-based WSD algorithm, unsupervised WSD.

1 Introduction The Word Sense Disambiguation (WSD) problem has a long tradition in Natural Language Processing and refers to an automatic means of selecting the appropriate sense of a word in its context of appearance chosen from a sense inventory that is available to the algorithm. The most recent and internationally recognized competition of WSD algorithms, SemEval-2010 [1] (the successor of SemEval-2007 [8]), proved once more that this is a very difficult problem and that the best systems are not able to exceed an F-measure of 56% on the English all words task on a limited domain – the task that requires that all content words of the test text in a given domain to be disambiguated using Princeton WordNet (PWN [2], http://wordnet. princeton.edu/) as the sense inventory. Traditionally, the WSD problem has been tackled using one of the two major approaches: supervised WSD in which a classifier for each sense of each word is trained from a sense annotated corpus or unsupervised WSD in which disambiguation is performed using any other means than supervised WSD. SemEval-2010 reconfirmed the fact that unsupervised WSD (also encompassing knowledge-based WSD) is no match for the supervised WSD methods (even with “weak supervision”) and we postulate that, in the long run, there will never be enough training data for the supervised approach to WSD (be it domain dependent). For instance, if we take PWN version 2.0 into consideration, there are 115424 synsets, each one corresponding to a single sense of a literal within the synset. A supervised WSD method that uses PWN as the sense inventory, should train on a corpus that has, let’s say at least 10 examples of usage for each literal in each synset. If a PWN synset has almost 2 literals in one synset on average, this gives us an unrealistic size of the training data of 2 * 10 * 115424 = 2308480 of sense annotated occurrences! Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 435–443, 2011. © Springer-Verlag Berlin Heidelberg 2011

436

R. Ion and D. Ştefănescu

The alternative to the supervised WSD is, of course, the unsupervised approach which takes into consideration any linguistic information source it can find along with any text linguistic preprocessing it can afford in order to perform sense disambiguation. An already established linguistic information source is the lexical ontological structure of PWN which has been widely used to infer semantic judgments between senses of words occurring in some context. This context has been defined by making use of different text preprocessing steps such as POS tagging, lemmatization or syntactic analysis (parsing) either by constituents or by (word) dependency. Here we are interested in graph-based context formalization for WSD. There is a range of WSD methods [7, 10] that use the notion of sense graph as an approximation of the context in which the words of a sentence appear. The vertices of this graph are sense labels extracted from PWN and the edges of the graph are added either by relation walks in the PWN graph [7] or by semantic similarities computed over PWN [10]. On the other hand, we are interested in the lexical chains concept as a measure of semantic similarity. We follow the model of the lexical chain described by Moldovan and Novischi [6] but we generalize it by allowing paths of depth greater than 1 in PWN and by not imposing certain patterns of PWN relations that can form such a lexical chain. Along the lines of constructing lexical chains, the WSD problem gained center stage as there is a mutual dependency between the two: the lexical chains construction procedure would benefit from the sense disambiguation of the words that can participate in such a chain and, the WSD algorithm would certainly welcome semantically related sets of words as input which would be easier to disambiguate. Recognizing this mutual dependence and the fact that producing word sense disambiguated lexical chains is a computationally complex problem, Silber and McCoy [9] and Galley and McKeown [3] set off to improve running time when constructing lexical chains with nouns. The WSD methods discussed in these papers involve the construction of a PWN sense graph for all nouns in the text to be disambiguated using only synonymy, hypernymy and the ‘sibling’ relation: two nodes with the same hypernym. In what follows we will present a WSD algorithm that makes use of the same graph-based context formalization. But, whereas the aforementioned WSD methods assembled graphs of PWN senses, we will construct a graph of content words of the sentence. The connectivity of this graph will be given by the existence of at least one lexical chain between two words of a sentence. Our WSD algorithm is required to find that interpretation (the combination of senses, a sense per each word of the sentence) that will maximize the semantic similarity as defined by the lexical chains.

2 Lexical Chains The term “lexical chain” refers to a set of words which, in a given context (sentence, paragraph, section and so on), are semantically related to each other and are all bound to a specific topic. For instance, words like “tennis”, “ball”, “net”, “racket”, “court” all may form a lexical chain if it happens that a paragraph in a text contains all of them. Moldovan and Novischi [6] used an extended version of the PWN (XWN, http://xwn.hlt.utdallas.edu/) to derive lexical chains between the meanings of two words by finding relation paths in the WordNet hierarchy. They have acknowledged

Unsupervised WSD with Lexical Chains and Graph-Based Context Formalization

437

the importance of the glosses in dramatically increasing synset connectivity leading to syntagmatic associations. Thus, because XWN has word sense disambiguated glosses, a new relation named ‘gloss’ can be formed between a synset and the corresponding gloss literal along with its PWN sense label. In Moldovan and Novischi’s view, each source synset has a micro-context formed by the synsets of the content words of the gloss of the source synset and their glosses and the synsets which are related with the source synset and their glosses. Within this graph of PWN synsets, there are certain patterns of paths which are considered when constructing a lexical chain. The central idea to remember is that a lexical chain is not simply a set of topically related words but becomes a path of synsets1 in the WordNet hierarchy. This is the derived definition that we have used to implement our version of lexical chains as meaning paths through PWN. Thus, we consider the micro-context (or semantic frontier in our terminology) of a given source synset all the synsets that are related to it through one of the PWN semantic relations. We do not consider the gloss relation introduced above for two main reasons: 1. we want to construct a WordNet-independent lexical chains algorithm that will work in the same way on PWN and, for instance, Romanian WordNet which does not have word sense disambiguated glosses; 2. we observed that by using glosses on PWN (not XWN) in which we had to consider every sense of a given gloss literal and by not imposing a depth limit on WordNet relations walk (including the ‘gloss relation’), the semantically relevant synset connectivity deteriorates rapidly. The lexical chain procedure is a function of two PWN synsets, LXC(s1, s2), that returns a semantic relation path that one can follow to reach s2 from s1. On the path from s2 to s1 there are k synsets (k ≥ 0) and between 2 adjacent synsets there is a PWN semantic relation. Each lexical chain can be assigned a certain score that we interpret as a measure of the semantic similarity (SS) between s1 and s2. Ideally, SS(s1, s2) = 1 iff s1= s2, and SS(s1, s2) = 0 iff s1and s2 “have nothing to do one with the other” semantically. Moldovan and Novischi [6; Table 2] used an empirically derived set of weights for each PWN semantic relation in order to score a lexical chain. Thus, for instance, the hypernymy relation (score 0.8) is more important than the pertain relation (score 0.5). Given these semantic relations scores, if a global lexical chain score (a function of the individual semantic relations scores) is approaching 0, this would be an indication of s1and s2 “having nothing to do one with the other” semantically. We think that, while one can have meaningful experimental insights of how relevant is one PWN relation over another, there should be a principled way of assigning weights to PWN semantic relations such that SS(s1, s2) will have a 0 value when s1and s2 are indeed semantically apart one from the other. To this end, we used the widely recognized PWN sense-annotated SemCor corpus [5] with mappings to the version 2.0 of the PWN. For each sentence S of the corpus and for each pair of sense annotated words from S, sa and sb, we computed all the possible lexical chains LXC(sa, sb) and collected a frequency table of PWN semantic relation patterns (Table 1 lists the

1

Throughout this paper we will use the notions of “synset”, “concept”, “meaning”, “literal with a sense (label)” as being synonymous in the context of PWN.

438

R. Ion and D. Ştefănescu

Table 1. First 10 most frequent PWN semantic relations patterns from automatically computed lexical chains between sense-annotated words of SemCor

Lexical Chain Pattern

F

hyper,hyper eng_deriv,eng_deriv hyper,hyper,hyper,hyper,hyper,hyper eng_deriv,hyper,hyper,hyper,eng_deriv,eng_deriv hyper,hyper,hyper,hyper hyper,hyper,hyper,hyper,eng_deriv,eng_deriv eng_deriv,eng_deriv,hyper,hyper,eng_deriv,eng_deriv hyper,hyper,hyper,hyper,hyper,eng_deriv similar_to,similar_to eng_deriv,hyper,hyper,eng_deriv

6435 6171 4407 3690 3753 3437 3281 3028 3001 2929

first 10 most frequent PWN semantic relations patterns – or Structural Semantic Interconnections according to Roberto Navigli – appearing in lexical chains). From the above table we see that only 3 PWN relations, “hypernym” (hyper), “English derivative” (eng_deriv) and “similar to” make up over 40K of lexical chains. Now, not all the lexical chains are correct in the sense that when one would follow the indicated path would conclude that sa, sb are indeed semantically related but it’s reasonable to assume that shorter and frequent lexical chains would be correct in the sense defined above. Thus, we proceeded to the 100 step re-estimation of the P(r) (where h,r is a bigram of PWN relations extracted from a pattern) with the equation

P(r) = ∑ P(r | h) × P(h), P(r | h) = h

f (h, r) f (h)

where P(r) is initially uniformly distributed and f(h, r) and f(h) are scaled by the frequency of the pattern they appear in. For instance, f(hyper) in the first line of the Table 1 would be 2 * 6435 and f(hyper, hyper) would be 6435. By retaining all patterns with frequency bigger than 10 and with at most 4 relations (our experimental condition for lexical chain “correctness”), we arrived at the probabilities in Table 2. Basically, the lexical chaining algorithm expands the semantic frontier of each end of the lexical chain and searches for an intersection of frontiers (a frontier is a set of synsets that are reachable by the way of PWN semantic relations from a source synset). When such an intersection is found (more than one is found actually), the algorithm stops and retrieves each chain by backtracking from the intersection point to the source and the target synset. The score of the lexical chain is

SS (a, b) =

∑ P (r )

r∈LXC ( a , b )

LXC (a, b)

2

We have to observe that SS(a, b) will never reach the value of 1 but it will guarantee at least that if a lexical chain is semantically implausible (on the hypothesis that our aforementioned experimental condition holds), it will return a value close to 0.

Unsupervised WSD with Lexical Chains and Graph-Based Context Formalization

439

Table 2. P(r) for relations in PWN. The default value for a relation that is not in this table is a small constant.

PWN Relation hypernym eng_derivative holo_part category_domain similar_to near_antonym also_see verb_group holo_member be_in_state derived subevent causes region_domain usage_domain holo_portion

P(r) 0.57399 0.27714 0.04350 0.03149 0.01529 0.01351 0.01180 0.00772 0.00748 0.00566 0.00430 0.00275 0.00256 0.00105 0.00092 0.00075

3 The WSD Algorithm Our WSD algorithm requires that the input sentence S is POS-tagged and lemmatized. We do this with TTL, a Perl web service offering these services [11]. Then, following the main trend of these algorithms, we construct a graph G = (V, E) where the vertices are the content words wi2 of the sentence. Each vertex can take one of the available sense labels from PWN for wi, s1i … ski. Initially, the E set of edges is empty. We add an edge to E from wi to wj iff there is at least one lexical chain from some sai to some sbj 3, these being legal sense labels for wi and wj. Please note that in this phase, we have to compute all the lexical chains between each pair of sense labels of each pair of content words from S. This is a very computationally-expensive operation and since in the case of lexical chains scoring, we have considered at most 4 PWN relations to make up a “correct” lexical chain, we impose the same limit here. Furthermore, we have equipped the lexical chain function LXC with a third parameter which is a time constraint. If the computation does not end within that time constraint, the function returns the empty set. This means that the lexical chain search procedure becomes incomplete (it does not supply all solutions all the time) and thus, a future WSD performance penalty is introduced from this early stage. 2

Actually, we use lemmas of the content words of the sentence since lemmas are found as literals of the PWN synsets. 3 If there is more than one lexical chain, we retain only the highest scored one.

440

R. Ion and D. Ştefănescu

After the graph G is constructed, the WSD algorithm has as its goal to find an assignment of senses4 A = {sxi | sxi ∈ Sns(wi)} such that

∑ SS (s , s i x

j y

), s xi ∈ Sns( wi ), s yj ∈ Sns( w j )

wi , w j ∈V

is maximum, where Sns(wi) is the set of PWN senses of wi, s1i … ski. In the sum above it should be pointed out that edges 〈wi, wj〉 belong to a Maximum Spanning Tree (MST) of G in a given interpretation. Since there are an exponentially large number of interpretations for which to compute MSTs, we employed a greedy searching method. This is a beam search in which we first select the edge corresponding to the largest SS score between two sense labels. Then we continue by greedily adding edges to the MST until all vertices are in the tree (and we have an interpretation) or we have nothing left to add. In the latter case we backtrack and choose the second best edge according to the SS score and continue. At every step we keep K choices (K=10) so we have something to backup to if the algorithm finished and not all the vertices are in the MST. At this point, an example will help to clarify things. Given the sentence S, “The teacher’s student stands before the blackboard.”, we first apply the POS-tagging and lemmatization pre-processing steps and obtain the set of content words {teacher/Noun, student/Noun, stand/Verb, blackboard/Noun}. From PWN version 2.0, we can see that |Sns(teacher/N)| = 2, |Sns(student/N)| = 2, |Sns(stand/V)| = 12, |Sns(blackboard/N)| = 1. So, our graph has 4 vertices. As for the edges, there are lexical chains between the following senses of these literals: teacher(1) and student(1), teacher(1) and stand(3), student(1) and blackboard(1). So the graph edge set E is composed of {〈teacher, student〉, 〈teacher, stand〉, 〈student, blackboard〉}. Table 3 contains the maximal scores of the lexical chains between all senses involved with the exception of stand/V for which we consider only the first 3 senses (for simplification). The interpretation of the given sentence is {teacher(1)/Noun, student(1,2)/Noun, stand(3)/Verb, blackboard(1)/Noun} or, with the Table 3 notations, {t(1)/N, stu(1,2)/N, sta(3)/V, b(1)/N}. We can observe that the first edge to be selected is 〈student, blackboard〉 with the configuration 〈student(2), blackboard(1)〉 with the score of 0.537. The next edge to be added is 〈student, teacher〉 with the configuration 〈student(2), teacher(1)〉, and so on and so forth.

4

Which we will call “interpretation”.

Unsupervised WSD with Lexical Chains and Graph-Based Context Formalization

441

Table 3. The maximum scores of lexical chains (at most 4 relations) between the given senses. ‘x’ marks the “not applicability” and ‘--‘ marks the inexistence of lexical chains between those meanings.

t(1) t(2) stu(1) stu(2) sta(1) sta(2) sta(3) b(1)

t(1)

t(2)

stu(1)

stu(2)

sta(1)

sta(2)

sta(3)

b(1)

1 x x x x x x x

x 1 x x x x x x

0.34 0.224 1 x x x x x

0.474 0.3 x 1 x x x x

0.11 ---1 x x x

0.215 ---x 1 x x

0.442 ---x x 1 x

--0.4 0.573 ---1

4 Evaluations To measure up the performance of this WSD algorithm and to be able to compare it directly to other unsupervised WSD algorithms, we run it on the test data from the English All Words task of SemEval-2007 [8, 7]. The test set comprised of 435 instances of nouns, verbs and adjectives that were manually disambiguated with PWN version 2.0 sense labels. We wanted to see how different parameters affect the performance of the disambiguation. Of these, the most interesting is the length of a lexical chain in PWN relations. Thus, we have tested the WSD disambiguation accuracy by imposing 1, 2, 3 or 4 PWN relations in a lexical chain. What we obtained is listed in Table 4 below. Table 4. WSD performance on English All Words task of SemEval-2007. The sense inventory is given by PWN 2.0. Relation number 1 relation 2 relations 3 relations 4 relations

P 0.50833 0.49367 0.46300 0.43981

R 0.14022 0.35862 0.44597 0.43678

F-measure 0.21980 0.41544 0.45432 0.43829

The runs from Table 4 were completed without the Most Frequent Sense backup (if a targeted occurrence didn’t receive a sense label, the MFS would assign it the most frequent sense from the PWN). Navigli and Lapata [7] call the unsupervised system plus the MFS heuristic backup, a semi-supervised system. In our case, the MFS backup heuristic for the best performance in Table 4 (the case with at most 3 relations in a lexical chain) provided the figures P = 0.54479, R = 0.51724 and F = 0.53066 thus being a little better than our previous system SynWSD [4]. What we can deduce by studying Table 4 is that our system is currently the best one in the unsupervised setting without the MFS back off using lexical chains with at most 3 PWN relations (the best SemEval-2007 unsupervised system without the MFS back off has an F-measure of 0.431 – see [7; Table 8]). Also, expectedly enough, we

442

R. Ion and D. Ştefănescu

see that while the precision of the system decreases with the length increase of the lexical chains, the recall increases quite spectacularly. We may infer that while very short lexical chains are, most probably “semantically plausible”, they cover few of the cases of sense co-occurrence. When the size of the lexical chain increases, more cases are accounted for, but at the price of more noise introduced.

5 Conclusions We have developed a graph-based unsupervised WSD algorithm that proved to be among the best in its class (unsupervised, that is) of WSD algorithms. The performance of the algorithm may be optimized along more parameters that we have described here. For instance, we can modify the condition that governs P(r) thus obtaining different probabilities for each of the PWN relations (given the fact that the best results are obtained with a lexical chain length of most 3 PWN relations and not 4 as we had set our limit to). Another parameter is the beam search cut in the possible interpretations of the sentence graph. We have used K = 10 but other values would certainly affect the performance measures.

References 1. Agirre, E., de Lacalle, O.L., Fellbaum, C., Hsieh, S.K., Tesconi, M., Monachini, M., Vossen, P., Segers, R.: SemEval-2010 Task 17: All-wordsWord Sense Disambiguation on a Specific Domain. In: Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, Uppsala, Sweden, July 15-16, pp. 75–80. Association for Computational Linguistics (2010) 2. Fellbaum, C. (ed.): WordNet: an Electronic Lexical Database. MIT Press, Cambridge (1998) 3. Galley, M., McKeown, K.: Improving word sense disambiguation in lexical chaining. In: Proceedings of 18th International Joint Conference on Artificial Intelligence (IJCAI 2003), Acapulco, Mexico, August 9-15 (2003) 4. Ion, R., Tufiş, D.: RACAI: Meaning Affinity Models. In: Agirre, E., Màrquez, L., Wicentowski, R. (eds.) Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval 2007), Prague, Czech Republic, pp. 282–287. Association for Computational Linguistics (June 2007) 5. Miller, G.A., Leacock, C., Tengi, R., Bunker, R.T.: A semantic concordance. In: Proceedings of the 3rd DARPA Workshop on Human Language Technology, Plainsboro, New Jersey, pp. 303–308 (1993) 6. Moldovan, D., Novischi, A.: Lexical chains for question answering. In: Proceedings of the 19th International Conference on Computational Linguistics, Taipei, Taiwan, August 24-September 01, pp. 1–7 (2002) 7. Navigli, R., Lapata, M.: An Experimental Study of Graph Connectivity for Unsupervised Word Sense Disambiguation. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 32(4), 678–692 (2010) 8. Pradhan, S.S., Loper, E., Dligach, D., Palmer, M.: SemEval-2007 Task 17: English Lexical Sample, SRL and All Words. In: Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval 2007), Prague, pp. 87–92. Association for Computational Linguistics (June 2007)

Unsupervised WSD with Lexical Chains and Graph-Based Context Formalization

443

9. Silber, H.G., McCoy, K.F.: Efficiently computed lexical chains as an intermediate representation for automatic text summarization. Computational Linguistics 28(4), 487–496 (2002) 10. Sinha, R., Mihalcea, R.: Unsupervised Graph-based Word Sense Disambiguation Using Measures of Word Semantic Similarity. In: Proceedings of the IEEE International Conference on Semantic Computing (ICSC 2007), Irvine, CA (September 2007) 11. Tufiş, D., Ion, R., Ceauşu, A., Ştefănescu, D.: RACAI’s Linguistic Web Services. In: Proceedings of the 6th Language Resources and Evaluation Conference – LREC 2008, Marrakech, Morocco, ELRA – European Language Resources Association (May 2008) ISBN 2-9517408-4-0

An Access Layer to PolNet – Polish WordNet Marek Kubis Adam Mickiewicz University, Faculty of Mathematics and Computer Science, Department of Computer Linguistics and Artificial Intelligence, ul. Umultowska 87, 61-614 Poznań, Poland [email protected]

Abstract. The paper describes an access layer developed in order to provide access to PolNet (a lexical database developed for the Polish language). The access layer was developed on top of a domain-specific language designed to query WordNet-like lexical databases (WQuery). The paper presents the overall architecture of the access layer and shows typical queries passed by an AI system with NL competence (POLINT-112-SMS) to WQuery. The paper discusses the reasons for integrating an ontology into an NLP system through a domainspecific query language. Keywords: wordnet, ontology, lexical database, query language, domain specific language.

1 Introduction POLINT-112-SMS is a natural language processing AI system focused on supporting information management in crisis situations [1]. POLINT-112-SMS has to access PolNet (a WordNet-like lexical database developed for the Polish language [2]) to guide such procedures as assignment of possible meanings to words and construction of the internal representation of a whole sentence. In order to integrate POLINT-112-SMS with PolNet, a software layer has been created that refers to the wordnet stored outside of the system and provides an API used by the system modules to access the database. The layer has been developed on top of WQuery – a system designed to query WordNet-like lexical databases using domain specific artificial language. WQuery is an open source tool. It operates on platforms that provide Java Runtime Environment and is able to work with any wordnet that is stored in an XML file that corresponds to the Global WordNet Grid DTD [3]. The rest of the paper is organized as follows: In the first part the overall architecture of the access layer is described. The second part explains the basic syntactic constructs of the WQuery language used in queries generated by POLINT-112-SMS. The third part presents typical queries passed to WQuery by POLINT-112-SMS. The last part confronts the adopted approach with other possible solutions.

2 Access Layer Architecture The overall architecture of PolNet Access Layer is shown in Fig. 1. The layer is accessed by POLINT-112-SMS modules, such as NLPM (Natural Language Processing Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 444–455, 2011. © Springer-Verlag Berlin Heidelberg 2011

An Access Layer to PolNet – Polish WordNet

445

Module), SAM (Situation Analysis Module) and DMM (Dialogue Maintenance Module). POLINT-112-SMS modules access PolNet through API (Application Programming Interface), which consists of procedures responsible for the execution of queries formulated in the WQuery language. An API procedure checks if a query result may be retrieved from PolNet Cache. If the query result is not available or if PolNet Cache has been switched off, then the procedure passes the query directly to WQuery Client, which delegates it further to WQuery Server. PolNet Cache has been introduced into the architecture to reduce query processing time and to make PolNet available to POLINT-112-SMS even if WQuery Server is down. Query processing takes place on WQuery Server, which is independent of POLINT-112-SMS and may be deployed on a separate physical machine. The server contains WQuery Interpreter, which is responsible for query execution and Wordnets Datastore, which contains one or more (possibly different) PolNet instances. PolNet is loaded into the datastore from an XML file.

Fig. 1. PolNet Access Layer architecture

3 WQuery Language The complete description of the WQuery language is outside the scope of this paper (for details see [4]). The description below concentrates on constructs that are used in POLINT-112-SMS and skips the other ones.

446

M. Kubis

3.1 Data Types Basic Data Types. There are six basic data types in WQuery. Two of them – the word sense data type and the synset data type – represent concepts around which WordNet-like lexical databases are organized [5]. The other basic data types – strings, integers, floats and booleans – have been introduced in order to support general purpose computations. The string data type represents character data. The integer and float data types represent numbers. The boolean data type represents logical values – true and false. Word senses are presented in this paper as triples that consist of a word form, a sense number and a part of speech symbol (n for nouns, v for verbs, etc.) separated by colons as in the following example przedmiot:1:n

(an object)

Synsets are presented as lists of word senses surrounded with “{” and “}”. { obiekt:1:n przedmiot:1:n rzecz:1:n } Relations. Let X mean a set of values of the basic data type T (predecessor set), Let Y mean a set of values of the basic data type U (successor set). A relation is a subset of the Cartesian product of X and Y. WQuery uses relations to transform members of predecessor sets into members of successor sets. For example, hyponyms is a relation that transforms synsets into their hyponyms and gloss is a relation that transforms synsets into strings that represent their descriptions. Tuples. Tuples are finite, ordered collections. Each element of a tuple must belong to a basic data type. Tuples in WQuery are flat. It is impossible to construct a tuple that contains another tuple as its element. 3.2 Basic Syntax Generators. Generators are expressions that represent1 multisets of objects which share the same (basic) data type. Generators are the simplest valid WQuery queries. Boolean, integer and floating point constants generate multisets that contain exactly one element defined by the expressed constant. wquery> true true wquery> 123 123 wquery> 1.2e2 120.0 1

Generate in WQuery terminology.

An Access Layer to PolNet – Polish WordNet

447

The content of the multiset generated from a string depends on the surrounding characters: • • •

If the string has been surrounded with single quotes, then the generated multiset contains that string if and only if it is a valid word form in the given wordnet; otherwise, the generated multiset is empty. If the string has been surrounded with back quotes, then the generated multiset contains that string. If the string has not been surrounded with any quotation marks, then it is treated as if it has been surrounded with single quotes.

wquery> 'przedmiot' przedmiot wquery> `a1b2c3d4` a1b2c3d4 wquery> 'a1b2c3d4' (no result) wquery> przedmiot przedmiot A multiset that contains at most one word sense may be generated from a string, a sense number and a part of speech symbol joined together with colons. If the constructed triple does not point to a correct word sense in the wordnet, then the generated multiset will be empty. wquery> przedmiot:1:n przedmiot:1:n wquery> a1b2c3d4:1:n (no result) A multiset that contains synsets may be generated from string and word sense generators by surrounding them with “{” and “}” signs. The multiset will contain every synset that includes at least one word form (or word sense) represented by the generator found between “{” and “}”. wquery> {przedmiot} { obiekt:1:n przedmiot:1:n rzecz:1:n } { obiekt:2:n przedmiot:4:n cel:4:n przedmiot:4:n } { przedmiot:3:n temat:1:n } { przedmiot:2:n przedmiot nauczania:1:n } { przedmiot:5:n } wquery> {przedmiot:1:n} { obiekt:1:n przedmiot:1:n rzecz:1:n } Curly brackets “{}” without a generator inside represent all synsets found in the given wordnet.

448

M. Kubis

Paths. A path consists of a generator followed by zero or more transformations.2 Each transformation begins with a dot followed by the name of the relation that should be applied to the expression stated before the dot. The result of transforming an expression that generates the multiset M using the relation R is a multiset { b | a belongs to M and (a, b) belongs to R }. For example, to find all hyponyms of synsets that contain the second noun sense of the word form spoiwo (a binder) one may write wquery> {spoiwo:2:n}.hyponyms (gypsum) { gips:1:n } { klej:1:n } (glue) { wapno:1:n } (lime) and to find all words that belong to the hyponyms mentioned above one may transform the former expression using the words relation wquery> {spoiwo:2:n}.hyponyms.words gips klej wapno If the result of a transformation has the same data type as the source, then it is possible to compute transitive closure of the transforming relation by placing the “!” operator after the relation name. For example, to find all transitive hyponyms of synsets that contain the second noun sense of the word form spoiwo one may write wquery> {spoiwo:2:n}.hyponyms! { gips:1:n } { klej:1:n } { lepik:1:n } { wapno:1:n } Filters. A filter is an expression that may be placed after any step of a path in order to remove some elements from the multiset generated by that step. A filter consists of a condition surrounded with “[” and “]”. A condition is an expression that involves such operators as the equality check “=”, the inequality check “!=”, the multiset membership check “in”, etc. Conditions may be combined together using logical operators “and”, “or” and “not”. Each element generated by the filtered step is passed separately to the filter. The element being analyzed in the current pass may be referenced in the condition using “#” operator. For example, to find all synsets that contain the word form przedmiot, except the one that contains the first noun sense of the word form przedmiot, the following expression may be written 2

The generator and the following transformations are called (path) steps.

An Access Layer to PolNet – Polish WordNet

449

wquery> {przedmiot}[# != {przedmiot:1:n}] { obiekt:2:n przedmiot:4:n cel:4:n przedmiot:4:n } { przedmiot:3:n temat:1:n } { przedmiot:2:n przedmiot nauczania:1:n } { przedmiot:5:n } A reference and the following dot may be omitted if they are followed by at least one transformation, so the expression wquery> {przedmiot}[rzecz in #.words] may be reformulated as wquery> {przedmiot}[rzecz in words] Filters may also be used as generators. A filter placed as the first step of a path generates a multiset that consists of exactly one boolean value (a result of the condition inside the filter). wquery> [rzecz in {przedmiot}.words] true Selectors. A selector is an expression that makes it possible to retrieve data from the multiset generated by the chosen step of a path. The selector consists of “” signs that surround the chosen step. For example, to find hyponyms of hyponyms of synsets that contain the word form izba in its second noun sense (a chamber of parliament) together with hypernyms of their hypernyms, one may write wquery> .hyponyms. { izba:2:n } { sejm:1:n } { senat:1:n } Path Expressions. Paths may be combined together using union, intersect, except and “,” operators. They are responsible for creating unions, intersections, differences and Cartesian products of multisets generated by paths passed to them as arguments. wquery> {spoiwo:2:n}.hyponyms union {przedmiot} { obiekt:1:n przedmiot:1:n rzecz:1:n } { obiekt:2:n przedmiot:4:n cel:4:n } { gips:1:n } { cement:1:n } { przedmiot:3:n temat:1:n } { przedmiot:2:n przedmiot nauczania:1:n } { przedmiot:5:n } { klej:1:n } { wapno:1:n }

450

M. Kubis

wquery> {spoiwo:2:n}.hyponyms intersect {gips} { gips:1:n } wquery> {spoiwo:2:n}.hyponyms except {gips} { klej:1:n } { wapno:1:n } Imperative Expressions. WQuery possesses several expressions that support the imperative programming paradigm. •

An emission is an expression of the form emit path_expr which passes tuples generated by the path expression path_expr to the output.

•

A conditional3 is an expression of the form if path_expr then imp_expr_a else imp_expr_b which executes the imperative expression imp_expr_a if the path expression path_expr is true and executes imp_expr_b otherwise. The subexpression else imp_expr_b is optional. The path expression path_expr is assumed to be false if and only if it generates an empty multiset or the generated multiset includes exactly one tuple that consists of boolean values and at least one of those values is false. The second case makes it possible to use filter generators as conditionals in a convenient way, as shown in the example at the end of this section.

•

A block is an expression of the form do imp_expr_1 imp_expr_2 … imp_expr_n end which executes sequentially imperative imp_expr_2, …, imp_expr_n.

•

expressions

imp_expr_1,

An assignment is an expression of the form var_decls = path_expr where the path expression path_expr has to generate exactly one tuple. The assignment binds variable names from the comma separated list var_decls to consecutive elements of the tuple generated by the path expression path_expr.

3

Conditionals shall not be confused with expressions called conditions, which are placed inside filters.

An Access Layer to PolNet – Polish WordNet

•

451

An iterator is an expression of the form from var_decls in path_expr imp_expr which for every tuple generated by the path expression path_expr executes the imperative expression imp_expr with variable names from the comma separated list var_decls bound to consecutive elements of the tuple being processed in the current step.

The following expression iterates through all senses of synsets that contain the word form przedmiot and, depending on the sense number, returns the word form or the part of speech symbol of the word sense processed in the current step. wquery> from $a in {przedmiot}.senses wquery> do wquery> if [$a.sensenum emit $a.word wquery> else wquery> emit $a.pos wquery> end n przedmiot nauczania przedmiot temat n n n obiekt rzecz przedmiot obiekt

4 Typical Queries Appearing in POLINT-112-SMS The following subsections present three tasks performed by modules of POLINT-112SMS that require access to PolNet to be fulfilled. The first and second task play an important role in the process of analyzing and interpreting sentences in the NLP module. The third one is executed by the PolNet Access Layer itself. Besides the tasks described below, all other procedures of POLINT-112-SMS that require access to PolNet obtain it through the layer presented in this paper. 4.1 Obtaining Word Meanings Queries that retrieve all word senses of a particular word form are executed for all lemmas of words of every sentence passed to the POLINT-112-SMS NLP module in order to link words to their possible meanings stored in PolNet. For example, to find

452

M. Kubis

the possible meanings of words in the sentence “Kibic ma czapkę.” (“A team supporter has a cap.”) the following three queries shall be executed wquery> kibic.senses wquery> mieć.senses wquery> czapka.senses A word sense that belongs to a synset which is marked as not lexicalized should not be accepted as a possible meaning of a word passed to the NLP module. In order to exclude such senses, the NLP module extends the queries presented above with filters that check whether the senses belong to synsets that are transformable using the relation nl (not lexicalized) to the boolean value false wquery> kibic.senses[synset.nl = false] wquery> mieć.senses[synset.nl = false] wquery> czapka.senses[synset.nl = false] 4.2 Creating and Composing Frames POLINT-112-SMS internal knowledge representation is based on the idea of frames [6]. The construction of frames is guided by queries that check synset collection membership. Firstly, queries of this kind are used to choose frames appropriate to represent the meanings of particular words. For example, the system represents articles by the frame article and the NLP module assumes that every article is a transitive hyponym of a synset that contains the word form przedmiot in its first noun sense, so if the sentence “Kibic ma czapkę.” is analyzed, then for every sense w:n:p of the word form czapkę (a cap) found by the last query in Section 4.1, the NLP module executes a query wquery> [{w:n:p} in {przedmiot:1:n}.hyponyms!] to decide whether the word form czapkę may be mapped to the frame article. In this stage the majority of possible meanings found by the queries in Section 4.1. is eliminated because no corresponding frames exist. Secondly, queries that check synset collection membership are used to determine if one frame may be nested as a slot of the other one. The following query checks whether a synset that contains the sense w:n:p is equal to or is a transitive hyponym of a synset that contains the word form nakrycie głowy in its first noun sense (headgear). This query is executed for every sense w:n:p of the word form czapkę to check if the frame that represents czapkę may be put in the slot head of the frame appearance. wquery> [{w:n:p} in wquery> ({'nakrycie głowy':1:n} wquery> union wquery> {'nakrycie głowy':1:n}.hyponyms!)]

An Access Layer to PolNet – Polish WordNet

453

This stage eliminates meanings for which frame nesting has not led to a representation of the whole sentence as a single frame. All single frame based representations of the whole sentence created during this stage are passed to the Dialogue Maintenance Module as alternative meanings of the sentence. 4.3 Refreshing PolNet Cache PolNet Cache stores links between synsets and their transitive hyponyms. All pairs that consist of a synset followed by its transitive hyponym are obtained by the query wquery> . The cache also stores links between synsets and their senses represented as triples that consist of a word, sense number and synset. The triples are generated by the query below wquery> from $s, $w in . wquery> emit $w.word, $w.sensenum, $s

5 Discussion The simplest method of integrating a wordnet with an NLP system is to represent the wordnet directly by using data structures of the system (for example, to store the wordnet as a list of terms in a system that is implemented in Prolog). This approach was rejected because it introduces high coupling between the system and the wordnet. Every time a new version of PolNet was integrated the code of the system would have to be updated, thus resulting in a new version of the system. The second method is to store the wordnet outside the system and to provide a set of basic access procedures, such as “get a synset by its word form and sense number” or “get hyponyms of a synset”. This approach was rejected because the queries responsible for creating and nesting frames (like those shown in Section 4.2.) combine calls to such procedures together with the other ones (like multiset union, transitive closure computation and multiset filtering). These combinations vary between frames and if they were hard-coded into the system it would be hard to understand and modify them. The third method is to integrate a wordnet through a tool that provides a versatile query language. However, query languages (besides WQuery) do not operate on wordnet related data structures, such as synsets, word senses and relations, but rather enforce the adoption of their own data model (e.g. tables and columns in SQL, XML elements and attributes in XQuery, RDF triples in SPARQL). This makes queries more verbose and harder to understand. For example, to formulate an SQL query analogous to those presented at the bottom of Section 4.1 one would have to write select distinct w.id from wsenses w where w.word = 'czapka' and ( (select nl.value from wsenses s inner join nl

454

M. Kubis

on s.synset_id = nl.synset_id where w.id = s.id) = false) order by w.id where wsenses(id, word, sensenum, synset_id) is a table that stores word sense identifiers, words, sense numbers and synset identifiers, and nl(synset_id, value) is a table that stores synset identifiers together with boolean values that indicate whether a synset is lexicalized.

6 Conclusion This paper describes how to integrate an AI system with NL competence (POLINT112-SMS) with a WordNet-like lexical database (PolNet). The approach is based on a domain-specific language WQuery used to query the wordnet. Usage of WQuery as a part of the layer developed to access PolNet has two main advantages. Firstly, it decouples the system from the wordnet, thus permitting to switch between different versions of PolNet without the need to create a new version of the system. Secondly, it provides straightforward API that allows to create concise queries that involve such operations as computing transitive closures and filtering multisets. Although the paper describes integration between POLINT-112-SMS and PolNet, the approach based on WQuery is so generic that it can be easily adopted in other systems that have to integrate a wordnet. The only requirement is to provide a copy of the wordnet that is compatible with the Global WordNet Grid XML DTD.

Acknowledgements This work has been partially supported by the Polish Ministry of Science and Higher Education, grant R00 028 02 (within the Polish Platform for Homeland Security).

References 1. Vetulani, Z., Marciniak, J., Konieczka, P., Walkowska, J.: An SMS-based System Architecture (Logical Model) to Support Management of Information Exchange in Emergency Stuations. POLINT-112-SMS. In: Shi, Z., Mercier-Laurent, E., Leake, D. (eds.) Intelligent Information Processing IV. IFIP International Federation for Information Processing, Subject collection: Computer Science, vol. 288, pp. 240–253. Springer, Boston (2009) 2. Vetulani, Z., Walkowska, J., Obrębski, T., Marciniak, J., Konieczka, P., Rzepecki, P.: An Algorithm for Building Lexical Semantic Network and Its Application to PolNet - Polish WordNet Project. In: Vetulani, Z., Uszkoreit, H. (eds.) LTC 2007. LNCS, vol. 5603, pp. 369–381. Springer, Heidelberg (2009) 3. Global WordNet Grid XML DTD, http://www.globalwordnet.org/gwa/grid/bwn2.dtd (access date: September 23, 2010)

An Access Layer to PolNet – Polish WordNet

455

4. Kubis, M.: WQuery User Guide, http://wquery.org/user-guide.pdf (access date: September 23, 2010) 5. Fellbaum, C.: WordNet: An Electronic Lexical Database (Language, Speech, and Communication). MIT Press, Cambridge (1998) 6. Minsky, M.: A framework for representing knowledge, MIT-AI Laboratory Memo 306 (1974), http://web.media.mit.edu/~minsky/papers/Frames/frames.html (access date: October 10, 2009)

OTTO: A Tool for Diplomatic Transcription of Historical Texts Stefanie Dipper and Martin Schnurrenberger Linguistics Department Ruhr University Bochum, Germany [email protected], [email protected]

Abstract. In this paper, we present OTTO, a web-based transcription tool which is designed for diplomatic transcription of historical language data. The tool supports fast and accurate typing, by use of user-deﬁned special characters, and, simultaneously, providing a view on the manuscript that is as close to the original as possible. It also allows for the annotation of rich, user-deﬁned header information. Users can log in and operate OTTO from anywhere through a standard web browser. Keywords: Transcription transcription.

1

tool;

historical

corpora;

diplomatic

Introduction1

Since the ﬁrst days of corpus-based linguistic investigations, historical language data has been in the focus of research. Starting with the Dominican cardinal Hugh of St Cher, who in 1230 compiled the ﬁrst concordance of the Bible (more precisely, of the Latin translation Vulgate), up to Johann Jakob Griesbach, who published the ﬁrst Greek Gospel synopsis in 1776. Concordances served (and still serve) as the basis for comparing the meaning and usage of speciﬁc words in diﬀerent texts, such as the books of the Bible. Synopses are often used to reconstruct lost original sources, or to construct a stemma, i.e. the relationships and dependencies between diﬀerent text witnesses (diﬀerent text versions of the same underlying content). With the advent of electronic corpora in the 1960s and 1970s, focus shifted to modern languages, with recent data, because machine-readable texts were more easily available for modern languages than historical ones. A notable exception is the Helsinki Corpus of English Texts, a corpus of diachronic English data, compiled at the University of Helsinki between 1984 and 1991 [13,10]. Early manuscripts (or prints) exhibit a large amount of peculiarities (special letters, punctuation marks, abbreviations, etc.), which are not easily encoded by, e.g., the ASCII encoding standard. Hence, an important issue with historical 1

This paper is a revised and extended version of [6]. The research reported in this paper was supported by Deutsche Forschungsgemeinschaft (DFG), Grant DI 1558/11. All URLs provided in this paper have been accessed 2010, Sep 15.

Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 456–467, 2011. Springer-Verlag Berlin Heidelberg 2011

OTTO: A Tool for Diplomatic Transcription of Historical Texts

457

corpora is the level of transcription, i.e. “how much of the information in the original document is included (or otherwise noted) by the transcriber in his or her transcription” [7]. Diplomatic transcription aims at reproducing a large range of features of the original manuscript or print, such as large initials or variant letter forms (e.g. short vs. long s: vs. ). Another matter is the amount of variation in language: prior to the emergence of a standard national language with orthographic regulations, texts were written in dialects, rendering dialectal vocabulary and pronunciation in a more or less accurate way. Important texts, such as The Song of the Nibelungs, have often been handed down in a large variety of witnesses, which, to a greater or lesser extent, diﬀer from each other with regard to content and language (dialect). In the 19th century, Karl Lachmann, one of the formative scientist in stemmatics, created a kind of “ideal”, artiﬁcial language for texts written in diﬀerent dialects from Middle High German (MHG). This language “normalizes” and levels out regional diﬀerences and thus facilitates comparison and understanding of MHG texts. Of course, on the other hand, it impedes in-depth linguistic research because the languages of these texts are, in a certain sense, corrupted. Unfortunately, the normalized language has been widely used in editions for MHG texts. Hence, electronic corpora that are based on such editions are useful only to a certain extent. As a consequence, a new project has been launched at the Universities of Bochum and Bonn, entitled “Reference Corpus Middle High German (1050–1350)”, which aims at creating a reference corpus of MHG texts that (i) does not make use of normalized text editions but (copies of) original manuscripts only, and (ii) applies diplomatic transcription. The project group has more than 20 years of experience with transcribing and annotating historical texts. Until recently, however, they used ordinary text processing tools for transcribing. In this paper, we present the tool OTTO (“Online Transcription TOol”), which is designed for diplomatic transcription of historical texts. It provides interfaces for text viewing and editing and entering of header information. Output formats are XML or plain text. The paper is organized as follows. In Sec. 2, we list requirements speciﬁc to historical language data that transcription tools have to meet. Sec. 3 presents related work. Sec. 4 introduces OTTO, followed by concluding remarks in Sec. 5.

2 2.1

Requirements of Transcription Tools Characteristics of Historical Texts

Diplomatic transcription aims at rendering a manuscript as original as possible, so that virtually no interpretation is involved in the transcription process. However, certain decisions still have to be made. Some of these decisions can be made once and for all, and apply to the transcription of the entire corpus. These are conventions, speciﬁed in the form of guidelines that have to be followed. Other decisions are up to the transcriber and have to be made as the case arises in the course of the transcription process, e.g. if the language’s alphabet contains

458

S. Dipper and M. Schnurrenberger

characters and symbols that can be mistaken and the transcriber has to decide which character is at hand. Transcription guidelines specify how to transcribe letters that do not have a modern equivalent. They also specify which letter forms represent variants of one and the same character, and which letters are to be transcribed as diﬀerent characters. Relevant cases include “normal” and tailed z ( vs. < >) and short vs. long s ( vs. ). In both cases, one of the variants has been abolished in the course of time : in modern European alphabets, only and are still used. This means that there is no straightforward one-to-one mapping between medieval and modern alphabets. Abolished letters have to be encoded in a special way. Frequently, (remnant or emerging forms of) diphthongs are rendered in a way that the second vowel is superscribed over the ﬁrst vowel. In most modern (European) alphabets, only diacritics such as accents or umlaut can be used as superscripts. For some examples of special letters, see Fig. 1. The last word in the ﬁrst line of the fragment is ſ — long s and an i without a dot (the dot over the Ó i developed in the 14th century). The ﬁfth word in the last line is C , i.e. the ﬁrst name Conrad — superscribed o. The word to the left of the initial D º contains a combination of y and a superscribed dot: ſ .

Fig. 1. Fragment of the Erfurter Judeneid (‘Erfurt Jewish oath’), around 1200 (Erfurt, Stadtarchiv, 0-0/A XLVII Nr. 1). For a transcription of the complete text, see Fig. 2.

To save space and time, medieval writers used a lot of abbreviations. For instance, nasal or is often encoded by a superscribed horizontal bar: < > (“Nasalstrich”), as in , which stands for von ‘from’. A frequentlyoccurring word form is for the conjunction und/unde ‘and’. A superscribed hook: < >, (“er-K¨ urzung”), abbreviates er (or re, r, and rarely ri, ir ), as in mart , which represents marter ‘martyrdom’. Another kind of special characters are initials, which can range over the height of two, three or even ﬁve lines, and need also to be encoded in some way. Further, medieval texts often contain words or passages that have been added later, e.g., to clarify the meaning of a text segment, or to correct (real or assumed) errors. Such additions or corrections can be made either on top of the line that is concerned, or else at the margin of the page. A special case is provided by interlinear translations or glosses, where, e.g., a German word-for-word translation is superscribed over the Latin original text. Finally, the layout of the texts (lines, columns, front and back page) should be encoded in the transcription. First, this information provides the usual access

OTTO: A Tool for Diplomatic Transcription of Historical Texts

459

to the texts; text positions are usually speciﬁed by these coordinates. Second, information about line breaks is essential for lyrics, and could be useful for determining word boundaries in prose. Let us brieﬂy summarize the main encoding issues with historical texts: – Encoding of letters, symbols, and combinations thereof that do not exist in modern alphabets – Encoding of abbreviations – Encoding of later additions – Encoding of layout information – For bilingual glosses: encoding of alignment (word-for-word correspondences) 2.2

Meta-information: Header and Comments

A lot of research on historical texts focuses on the text proper and its content, rather than its language. For instance, researchers are interested in the history of a text (“who wrote this text and where?”), its relationship to other texts (“did the writer know about or copy another text?”), its provenance (“who were the owners of this text?”), or its role in the cultural context (“why did the author write about this subject, and why in this way?”). To answer such questions, information about past and current depositories of a manuscript, peculiarities of the material that the text is written on, etc. are collected. In addition, any indicator of the author (or writer) of the text is noted down. Here, the text’s language becomes relevant as a means to gather information about the author. Linguistic features can be used to determine the text’s date of origin and the author’s social and regional aﬃliation. This kind of meta-information, which pertains to the entire text, is encoded in the header. Typical header information further includes observations of all kinds of peculiarities of the text under consideration, such as special writing conventions (“the writer uses a peculiar ‘ﬀ’ ligature”) or uncertainties within the transcription (“the exact placement of the circumﬂex accent is often unclear; in the transcription it is always placed on the ﬁrst letter”). Similar meta-information can be encoded in the form of comments, if it only concerns speciﬁc parts within the text rather than the text as a whole. Comments are used, e.g., for passages that are not well readable, that are destroyed, or otherwise questionable. Transcriber use them to mark uncertainties, to mark remarkable properties of letter or word forms, or to mark later additions/corrections. This information could be used for later (semi)automatic creation of a critical apparatus. To summarize the encoding issues related to meta-information: – Encoding of information about the text, its author and/or writer (header) – Marking of text peculiarities (header and comments) 2.3

Requirements of Transcription Tools

The characteristics of (research on) historical texts that we identiﬁed in the previous sections put speciﬁc requirements on transcription tools.

460

S. Dipper and M. Schnurrenberger

Diplomatic transcription. Above all, use of Unicode is indispensable, to be able to encode and represent the numerous special symbols and characters in a reliable and sustainable way. Of course, not all characters that occur in historical texts are already covered by the current version of Unicode. This is especially true of character combinations, which are only supported partially (the main reason being that Unicode’s Combining Diacritical Marks focus on superscribed diacritics rather than characters in general). Therefore, Unicode’s Private Use Area has to be used as well. Similarly, there are characters without glyphs deﬁned and designed for them. Hence, an ideal transcription tool should support the user in creating new glyphs whenever needed. Since there are many more characters in historical texts than keys on a keyboard, the transcription tool must provide some means to enter all characters and combinations. In principle, there are two ways to do this: the transcriber can use a virtual keyboard, which can support various character sets simultaneously and is operated by the mouse. Or else, special characters, such as “ ”, “@”, “(“, “#”, etc., are used as substitutes for historical characters; these characters are commonly used in combination with ordinary characters, to yield a larger number of characters that can be represented. Of course, with this solution transcribers have to learn and memorize the substitutes. Given the fact that each text can exhibit its own letter forms and writing conventions, it must be possible to customize the tool and adapt it to individual texts. Meta-information. The tool must provide suitable means for encoding header information. To promote use of standardized values (and to minimize the risk of typos), the header should provide drop-down menus or radio buttons wherever possible. For other features, the tool must provide free-text input. Again, these settings are highly dependent on the text that is transcribed and on the project’s goal, and, hence, the tool should be customizable in these respects. Work ﬂow. Projects that deal with the creation of historical corpora often distinguish two processes: (i) transcribing the manuscript, (ii) collating the manuscript, i.e., comparing the original text and its transcription in full detail. Often two people are involved: One person reads out the manuscript letter for letter, and also reports on any superscript, whitespace, etc. The other person simultaneously tracks the transcription, letter for letter. This way, high-quality diplomatic transcription can be achieved. This kind of workﬂow implies for the tool that there be an input mode that supports straightforward entering of new text, from scratch. In addition, there should be a collation mode, which allows the user to view and navigate within the text in a comfortable way, and to easily jump to arbitrary text positions where transcription errors have to be corrected. Finally, we add a further requirement. In our project, multiple parties distributed over diﬀerent sites are involved. To minimize time and eﬀort required for tool installation and data maintenance, the tool is preferably hosted on a

OTTO: A Tool for Diplomatic Transcription of Historical Texts

461

server and operated via a web browser. This way, there is no need of multiple installations at diﬀerent sites, and data on the server does not need to be synchronized but is always up to date.

3

Related Work

Many projects that create corpora of historical languages derive their electronic text basis from printed editions since this saves a lot of work. To them, collating is a prominent step (if they collate at all—not all projects have enough funding to collate or have access to the original manuscript). To our knowledge, there is currently no tool available which supports collating a transcription with its manuscript. There are some tools that support collating multiple electronic texts with each other, such as transcriptions of diﬀerent text witnesses, or diﬀerent printed editions from one and the same source. These tools help the user by aligning text passages from the individual texts that correspond to each other, just like in a synopsis. Such tools are, e.g., Juxta [12, App. 3], TUSTEP [15], or the UNIX command diﬀ. Another technique of collating involves visual merging of copies of the texts that are to be compared (e.g., by overlays). This method presupposes that the texts are suﬃently similar, at the visual level. Hence, there is no tool that would work with hand-written texts using old scripts, which often require expert readers for deciphering. Similarly, for transcribing historical texts from scratch, there are no speciﬁc tools, to our knowledge. A task which is somewhat similar is (phonetic) transcription of speech data. There is a range of linguistic tools for this task, which all focus on the alignment of audio and transcription data, such as Praat [2], EXMARaLDA [14], or ELAN [9]. In fact, canonical usage of the term “transcription” applies to convertion from sound to characters. By contrast, “transliteration” means transforming one script into another script. We nevertheless stick to the term “transcription” since transcription can be viewed as a mapping from analog to digital data, whereas transliteration usually involves digital-to-digital mapping. Manuscripts obviously represent analog data in this sense. Another option would be to use common text-processing tools, such as MS Word or LaTeX. In MS Word, special characters are usually inserted by means of virtual keyboards but character substitutes can be deﬁned via macros. Substitutes are converted to the actual characters immediately after typing. However, macros often pose problems when Word is upgraded. LaTeX supports character substitutes, without upgrade problems. However, substitutes require additional post-processing by interpreters and viewers to display the intended glyphs, i.e., it does not oﬀer instant preview (unless a wysiwyg-editor such as LyX is used). Immediate preview seems advantageous in that it provides immediate feedback to the user. We argue below that diplomatic transcriptions would proﬁt considerably from a combination of both methods, i.e. parallel display of the character substitute that the user types combined with instant preview of the actual character.

462

4

S. Dipper and M. Schnurrenberger

OTTO

OTTO is an online transcription tool which is used through a standard web browser. OTTO is designed for high-quality diplomatic transcription of historical language data and supports distributed, collaborative working of multiple parties. It is written in PHP and uses MySQL as the underlying database. In the following, the currently-implemented features of OTTO are described in brief.

Fig. 2. Screenshot of OTTO, displaying the editor interface with the text fragment of Fig. 1 in lines 11–13. Lines 1–12 have already been transcribed, line 13 is just being edited. Each line is preceded by the bibliographic key (‘ErfJE’), and the folio and line numbers, which are automatically generated.

Menu ‘Documents’ The Documents menu provides facilities for the import and export of documents, opening and closing of transcriptions, and viewing and printing them. Import and export. Besides creating a transcription ﬁle and starting on an empty sheet directly in OTTO itself, there often are other sources for transcription ﬁles, such as electronic editions, which still need to be collated. For importing these transcription ﬁles, OTTO provides the Import option. It lists all available import sources, which the project group can deﬁne to ﬁt their individual needs. Once a transcription ﬁle has been imported to OTTO, all further editing takes place within OTTO.

OTTO: A Tool for Diplomatic Transcription of Historical Texts

463

Manuscript scans can be imported together with the transcriptions. Putting scan and transcription side by side facilitates the task of collating. Transcriptions can be exported to a plain text format or XML. In the near future, we will provide a TEI-compliant export format [3,4]. Open and close. The Open option lets members of the transcription team see which transcription ﬁles have already been transcribed within OTTO (or imported to OTTO) and are available for further editing. Since there is only one of any transcription ﬁle, having two transcribers work on the same ﬁle at the same time can lead to overwriting problems. OTTO faces this issue by keeping a ﬁle lock log. The moment a transcriber opens a transcription ﬁle it is locked and the other members of the team will see that this ﬁle is in use by another transcriber. Also the name of the transcriber is displayed so members can negotiate turns. In addition to opening an existing ﬁle, a new, empty transcription ﬁle can be created by one click. The transcriber is then asked to ﬁrst enter the folio and line number of the ﬁrst line that she is going to transcribe. This information is used to automatically create line counts for further lines. View and print. The View option shows the transcription ﬁle in its original layout. It displays the diplomatic transcription in form of pages, page sides and columns. This format is well suited for collating and can be used to print out a paper version. Menu ‘Edit’ The Edit menu contains the core functionality of OTTO, for entering meta information, transcribing a text, and specifying substitute characters. Header. The header of transcription ﬁles contains data about the ﬁle itself, its original corpus, its original corpus’ origin, its transcription process, etc. Which information will be recorded in the header depends on the individual project’s goals and resources. Hence, OTTO lets transcription teams deﬁne a customized but ﬁxed header, which can for example contain preformatted values, thus reducing typing mistakes. Using ﬁxed headers will make it more easy to exploit the information for further processing of the transcription ﬁles, or for use in a corpus search tool. Editor. The Text Editor (see Fig. 2) is OTTO’s core feature. The look and feel is highly customizable (see Customize further down below). It provides an editing window which resides at the current editing position. The editing position, when just having opened a ﬁle, is usually at the end of the ﬁle, so the transcriber can continue working right away. Usually she will enter a new line into the input ﬁeld denoted as ‘Transcription’ (left frame). While she is doing this, the input ﬁeld denoted as ‘Unicode’ (right frame) does a live (hence ‘Online’) transformation of her entered line into its actual diplomatic transcription form, using a set of rules

464

S. Dipper and M. Schnurrenberger

(see paragraph ‘Rules’ below). By keeping an eye on this online transformation, the transcriber gets feedback on whether her input was correct or not. In Fig. 2, the dollar sign ( ) serves as a substitute for long s (ſ), see the ﬁrst Ó word of the text, De ); and the combination u\o stands for , see Cu\onrat in the Transcription ﬁeld at the bottom. When the line or several lines have been transcribed, the new entry can be saved. This will navigate the editing window down. Buttons ‘New Page’, ‘New side’ and ‘New column’ will add marks to the current line, which are used for the automatically generated line counts (denoted in Fig. 2 as ‘ErfJE,1,01’ for example). Above and below the editing window, all currently transcribed lines are displayed with their line count, the entered line and the diplomatic line generated by applying the transformation rules. The line counts also function as links for moving the editing window to a line of one’s choice, in the act of proof reading or collating, for example. Rules. The transcription group may deﬁne rules for transforming the entered lines into the diplomatic lines. These rules can be set up to be valid for all transcription ﬁles or just for the current ﬁle. Transcription rules have the form of “search-and-replace” patterns. The ﬁrst entity speciﬁes the character “to be searched” (e.g. , the character substitute), the second entity speciﬁes the diplomatic Unicode character that “replaces” the actual character. Transcription rules are deﬁned by the user, who can consult a database such as the ENRICH Gaiji Bank [11] to look up Unicode code points and standardized mappings for them, or deﬁne new ones. OTTO uses UTF-8encoded Unicode and the Junicode font [1]. Junicode supports many of MUFI’s medieval characters (Medieval Unicode Font Initiative [8]), partly deﬁned in Unicode’s Private Use Area. Table 1 shows the rules used in the sample text in Fig. 2 (plus one sample rule involving a MUFI character). Column 1 displays the character that the transcriber types, column 2 shows the target character in Junicode font. Columns 3 and 4 supply the code points and names as deﬁned by Unicode. For example, line 1 speciﬁes ‘ ’ as a substitute for long s. Line 4 speciﬁes the apostrophe as a substitute of the er hook, as deﬁned by MUFI. In our project, abbreviations such as the horizontal bar or the er hook are not solved since we aim at diplomatic transcription. Other projects might want to deﬁne rules that replace abbreviations by the respective full forms. OTTO allows for the use of comments, which can be inserted at any point of the text. Since the current version of OTTO does not provide special means to take record of passages that have been added, deleted, or modiﬁed otherwise, the comment functionality is exploited for this purpose in our project. Menu ‘Project’ The Project menu provides support for distributed, collaborative working of multiple parties on collections of documents. Projects that deal with the

OTTO: A Tool for Diplomatic Transcription of Historical Texts

465

Table 1. Sample substitute rules Encoding Character Unicode Code Point Unicode or MUFI name ſ

U+017F

LATIN SMALL LETTER LONG S LATIN SMALL LETTER U + COMBINING LATIN SMALL LETTER O LATIN SMALL LETTER Y WITH DOT ABOVE MUFI name: COMBINING ABBREVIATION MARK SUPERSCRIPT ER

2

u\o

Ù

U+0075 U+0366

3

y\.

Ýª

U+1E8F

4

’

ê

1

U+F152

creation of historical corpora often involve a cascade of successive processing steps that a transcription has to undergo (e.g. double keying, resolving divergences, collating). To cope with the numerous processing steps, transcription projects often involve a lot of people, who work on diﬀerent manuscripts (or diﬀerent pages of the same manuscript), in diﬀerent processing states. OTTO supports such transcription projects in several aspects: First, it allows for remote access to the database, via standard web browsers. Second, documents that are currently edited by some user are locked, i.e., cannot be edited or modiﬁed otherwise by another user. Third, OTTO provides facilities to support and promote communication among project members. Finally, graphical progress bars show the progress for each transcription, measuring the ratio of the subtasks already completed to all subtasks. Menu ‘Settings’ The Settings menu lets each user customize the look and feel of OTTO. For example, displaying font sizes can be set to ﬁt the needs of every individual. The transcriber can also customize the number of lines she would like to edit at once. The arrangement of the Transcription and Unicode windows can also be modiﬁed: the Unicode window can be placed on top of the Transcription window rather than side by side. We conclude this section with some considerations that led us to the design of OTTO as described above. Any text-processing system that deals with special characters, which are not part of common keyboards, has to supply the user with some means as to input these characters. A frequently-chosen option is to provide a virtual keyboard. Virtual keyboards are “wysiwyg” in that their keys are labeled by the special characters, which can then be selected by the user by mouse clicks. As an alternative, (combinations of) keys provided by standard keyboards can serve as substitutes of special characters. In such systems, a sequence such as “”a” would be automatically replaced, e.g., by the character “¨a”. As is well known, virtual keyboards are often preferred by casual users, beginners, or non-experts, since they are straightforward to operate and do not require any extra knowledge.

466

S. Dipper and M. Schnurrenberger

However, the drawback is that “typing” with a computer mouse is rather slow and tedious and, hence, not a long-term solution. By contrast, regular and advanced users usually prefer a system that provides character substitutes, because once the user knows the substitutes, typing them becomes very natural and quick. Transcription projects often involve both beginners and advanced users: having people (e.g. student assistants) join and leave the team is rather often the case, because transcribing is a very labor- and time-intensive task. OTTO faces these facts by combining the two methods. The user types and simultaneously gets feedback about whether the input is correct or not. This lessens the uncertainty of new team members and helps avoiding typing mistakes, thus increasing the quality of transcription. Line-by-line processing, as provided by OTTO, is modeled after the line-based way of transcribing diplomatically. The lines of text that are currently not part of the editing window are write-protected. This reduces the risk of accidentally modifying parts of the transcription.

5

Conclusion and Future Work

We have presented OTTO, a transcription tool designed for diplomatic transcription of historical texts. Its main feature is to support fast high-quality transcriptions, by use of user-deﬁned special characters, and, simultaneously, providing a view on the manuscript that is as close to the original as possible. Future steps include an XML export that is compliant to the TEI standards, with respect to the encoding of properties of the proper text [3] as well as header information [4]. To further support collating, we plan to experiment with the manuscript scans. Putting the transcriptions as transparent overlays on top of the scans, could considerably facilitate collating, especially if a project cannot aﬀord employing two people for this task. Finally, we currently work on integrating a part-of-speech tagger [5] into OTTO. OTTO will provide an interface to run an external tagger. Its output is fed back into OTTO and can be corrected manually. Tags that are assigned a probability below a certain threshold will be presented by a drop-down menu, which also lists less probable tags for selection by the user. OTTO will be made freely available for non-commercial research purposes.

References 1. Baker, P.: Junicode, a Unicode/OpenType font for medievalists. Font Software, http://junicode.sourceforge.net 2. Boersma, P.: Praat, a system for doing phonetics by computer. Glot International 5(9/10), 341–345 (2001), software http://www.fon.hum.uva.nl/praat 3. Burnard, L., Bauman, S.: Representation of primary sources. In: P5: Guidelines for Electronic Text Encoding and Interchange, ch. 11. TEI Consortium (2007), http://www.tei-c.org/release/doc/tei-p5-doc/html/PH.html

OTTO: A Tool for Diplomatic Transcription of Historical Texts

467

4. Burnard, L., Bauman, S.: The TEI header. In: P5: Guidelines for Electronic Text Encoding and Interchange, ch. 2. TEI Consortium (2007), http://www.tei-c.org/release/doc/tei-p5-doc/en/html/HD.html 5. Dipper, S.: POS-tagging of historical language data: First experiments. In: Semantic Approaches in Natural Language Processing. In: Proceedings of the 10th Conference on Natural Language Processing (KONVENS 2010), pp. 117–121 (2010) 6. Dipper, S., Schnurrenberger, M.: OTTO: A tool for diplomatic transcription of historical texts. In: Proceedings of 4th Language & Technology Conference, Poznan, Poland (2009) 7. Driscoll, M.J.: Levels of transcription. In: Burnard, L., O’Keeﬀe, K.O., Unsworth, J. (eds.) Electronic Textual Editing, pp. 254–261. Modern Language Association of America, New York (2006), http://www.tei-c.org/About/Archive_new/ETE/Preview/driscoll.xml 8. Haugen, O.E. (ed.): MUFI character recommendation. Medieval Unicode Font Initiative, Bergen (2009), version 3.0, http://www.mufi.info 9. Hellwig, B., Uytvanck, D.V., Hulsbosch, M.: ELAN — linguistic annotator. Manual, Version 3.9.0, Max Planck Institute for Psycholinguistics, Nijmegen (2010), software http://www.lat-mpi.eu/tools/elan 10. Kyt¨ o, M. (ed.): Manual to the Diachronic Part of The Helsinki Corpus of English Texts: Coding Conventions and Lists of Source Texts, 3rd edn. University of Helsinki, Finland (1996) 11. Manuscriptorium project: The ENRICH project and non-standard characters, character database, http://beta.manuscriptorium.com/ (menu item ‘gaiji bank) 12. Nowviskie, B., McGann, J.: NINES: a federated model for integrating digital scholarship. White paper by NINES (Networked Infrastructure for NineteenthCentury Electronic Scholarship) (2005), software http://www.juxtasoftware.org 13. Rissanen, M., Kyt¨ o, M., et al.: The Helsinki Corpus of English Texts. Department of English, University of Helsinki. Compiled by Matti Rissanen (Project leader), Merja Kyt¨ o (Project secretary); Leena Kahlas-Tarkka, Matti Kilpi¨ o (Old English); Saara Nevanlinna, Irma Taavitsainen (Middle English); Terttu Nevalainen, Helena Raumolin-Brunberg (Early Modern English) (1991) 14. Schmidt, T.: Creating and working with spoken language corpora in EXMARaLDA. In: LULCL II: Lesser Used Languages & Computer Linguistics II, pp. 151– 164 (2009), software http://www.exmaralda.org 15. Zentrum f¨ ur Datenverarbeitung, Universit¨ at T¨ ubingen: TUSTEP: T¨ ubinger System von Textverarbeitungs-Programmen. Handbuch und Referenz. Manual, Version 2010, T¨ ubingen University (2000), http://www.tustep.uni-tuebingen.de

Automatic Author Attribution for Short Text Documents Monika Nawrot The University of Science and Technology AGH, 30 Mickiewicza Av., 30-059 Krakow, Poland [email protected]

Abstract. This paper presents the results of studies on author attribution for short text documents. It briefly presents features used in authorship identification, outlines the proposed hybrid algorithm, and describes the results of experiments which explore conditions needed to reliably recognize author identity. Finally, it summarizes obtained results, presents main problems faced while authorship attribution for short text documents and proposes improvements which when applied might lead to the better authorship description and in consequence recognition. Keywords: authorship analysis, author attribution, statistical stylistic, support vector machine.

1 Introduction The problem of authorship attribution based on text documents has been present since the beginning of literature. Establishing the author identity for an anonymous text, attributing a pen name to an actual author, confirming text authorship believed to be written by a given author, and detecting single author text parts in collective works are the main problems when considering the author identity in the broad context of text samples. However long the questions have existed, only now are we able to provide reliable and accurate answers to them. Due to technology development there is an opportunity to carry out complex and computationally challenging research which is the source of knowledge of factors pointing at a certain authorship.

2 Authorship Analysis Classification Author identity establishment is one of subjects of interest of authorship analysis which consists of the following aspects: authorship attribution, authorship characterization and plagiarism detection. The authorship attribution is a process of an anonymous text authorship recognition based on text samples written by a group of already known authors. It relies on a set of features deduced from text documents, and attempts to establish whether texts of an unknown authorship are significantly similar to any of the known text samples. The aim of an authorship characterization is to define the sociolinguistic group of the author. The sociolinguistic group is usually based on age, status, sex, background Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 468–477, 2011. © Springer-Verlag Berlin Heidelberg 2011

Automatic Author Attribution for Short Text Documents

469

or native language. Additionally, attributing psychological traits to authors of text documents is researched. Although there are not many works on the subject so far, the results may be useful in authorship characterization and in authorship attribution as well. The third aspect of authorship analysis is plagiarism detection. As a part of a wider problem, plagiarism detection in the context of text documents investigates similarity of two pieces of text. The problem of finding similarities between text documents differs from previously defined authorship attribution and authorship characterization. That is why the methods used in algorithms constructed for such purposes face different problems and use different measures, whereas the methods used in two latter matters are somewhat similar.

3 Features There are many text indicators which may be taken into consideration while establishing author identity. They can be divided into two categories, according to the approach to language, into structural and semantic features. Structural features have been taken into consideration from the very beginning of the authorship attribution as a scientific study. At the end of 18th century Edmund Malone1 presented a study of metre and rhythm in Shakespeare plays. Since the second half of the 19th century, more and more studies have been conducted on the subject of author description based on structural features retrieved from text samples. Wide range of researchers exploring the usability of such features proved that there are many factors which may be good indicators of author identity. The most important groups of them are described in the following paragraphs. The document-based features are features which describe document as a whole. Among others, they consist of the way the author divides text into paragraphs, the notation of dialogue sections and the sentence complexity. In fact, a sentence is the smallest text entity considered in the extraction of these kinds of features. The usability of the described indicators was proven for the first time by H. T. Eddy2. In this work, the average length of sentences and the distribution of sentences lengths were proven successful in authorship attribution. Afterwards W. B. Smith3 extended Eddy's idea and showed that the sentence length measured in characters (not in words as in Eddy) is also a good authorship indicator. Further investigations done independently by G. U. Yule4,Williams5, Morton6 and Ellegard7 confirmed the stability of pointed document-based features among the works of a single author. Simultaneously, the problem of insufficient discrimination properties was rised. The conclusion was that

1

A Dissertation on the Three Parts of Kind Henry VI, 1787. The Characteristic Curves of Composition, 1887. 3 Curves of Pauline and Pseudo-Pauline Style I/II, 1888. 4 On Sentence-length as a Statistical Characteristic of Style in Prose, with Application to Two Cases of Disputed Authorship, 1939. 5 A Note on the Statistical Analysis of Sentence-length as a Criterion of Literary Style, 1940. 6 The Integrity of the Pauline Epistles, The Authorship of the Pauline Epistles, 1965. 7 A Statistical Method for Determining Authorship, 1962. 2

470

M. Nawrot

the document-based features may only be used to limit the number of suspected authors of anonymous texts but cannot be the only factors indicating the authorship. As in document-based features where sentence was the smallest entity considered while feature extraction, in word-based features, a word is the base of processing. A. de Morgan is known to be an originator of using words as a base for authorship attribution algorithms. In 1851 he proposed an average word length as authorcharacterizing feature. The concept was used by Mendenhall who proved that the distribution of word length is stable among works written by a single author. The results of his studies were publicized in The Characteristic Curves of Composition (1887) and A Mechanical Solution to a Literary Problem (1901). Although other researchers confirmed Mendenhall's results (e.g. C. S. Brinegar in Mark Twain and the Quintus Curtius Snodgrass Letters: A Statistical Test of Authorship, 1963), the authorship identity properties of word length distribution came in for a lot of criticism. C. B. Williams8 in and M. W. A. Smith9 pointed out that this measure is highly correlated with the subject matter of the text. The same objection applies to the vocabulary richness – the measure combining the total number of words with a total number of distinct words in a text sample. However this measure is a frequent factor used in authorship analysis. Successful results can be found in G. U. Yule's The Statistical Study of Literary Vocabulary (1944) and R.H. Tweedie''s and F. J. Baayen's Outside The Cave of Shadows: Using Syntactic Annotation to Enhance Authorship Attribution (1996). Another universally recognized measure is the word frequency distribution. Many studies have proven that words with highest frequencies have outstanding properties for author characterization. The confirming results were publicized by Ellegard10. The number of words considered is usually limited to function words. As function words have little lexical meaning, they are believed to be subject independent. Function words frequency distribution as an authorship indicator was first used by W. B. Smith11. Interesting results can also be found in works by Mosteller and Wallace12, Anthony Kenny13 and R. M. Dabagh14. Character-based features are also present in numerous studies of authorship attribution. Character frequency distribution was proven to be useful in author identity assignment by G. U. Yule15. Later research has shown that character features have strong characterization properties. There have been studies investigating which groups of signs are the most significant in author description – alphabetic letters, digits, white space, punctuations – and they are the most common ones. Next to word- and character-based features are the character and word chains. In this method the coexistence of these entities is investigated. The first to propose this 8

Style and Vocabulary, 1970. Recent Experience and New Developments of Methods for the Determination of Authorship, 1983. 10 A Statistical Method for Determining Authorship: 1769-1772, 1962. 11 Curves of Pauline and Pseudo-Pauline Style I/II, 1888. 12 Inference and Disputed Authorship: The Federalist 1964, 1984. 13 The Aristotelian Ethics: A Study of the Relationship between the Eudemian and Nicomachean Ethics of Aristotle, 1978. 14 Authorship Attribution and Statistical Text Analysis, 2007. 15 The Statistical Study of Literary Vocabulary, 1946. 9

Automatic Author Attribution for Short Text Documents

471

measure in authorship attribution was R. W. Bennett16 who proved that two-letter chains can by useful in author identification. The following studies were presented in Using Markov Chains for Identification of Writers (2001) by D. V . Khmelev. F. J. Tweedie and in N-gram-based author profiles for authorship attribution (2003) by V. Keselj, F. Peng, N. Cercone, C. Thomas. Errors, yet another text feature, are probably the most questionable factors indicating authorship. Making errors is questionable in that it can be typical of a writer or it may have an incidental character. However, studies were conducted in which errors proved to be useful in authorship recognition e.g. in A. O. Osbourne's “Questioned Documents” (1910, 1929) and in M. Koppel's, J. Schler's “Exploiting Stylistic Idiosyncrasies for Authorship Attribution” (2003). All of the methods described above take into consideration only the structural properties of the text. The semantic features take advantage of the text content. Based on the vocabulary used both topic assignment as well as sociolinguistic profile creation is possible. There are more and more studies which show that it is possible to define the age, sex, status, background, native language and even psychological traits like neurosis or extroversion. Some interesting conclusions can be found in Social differentiation in use of English vocabulary: some analyses of the conversational component of the British national corpus1 (1997) written by P. Rayson, G. Leech, M. Hodgens, Effects of Age and Gender on Blogging (2005) by J. Schler, M. Koppel, S. Argamon, J. Pennebaker and in many others.

4 Algorithm The previous section gives a brief description of features which are recognised for theirs usability in authorship attribution and authorship characterization. Based on the studies conducted a set of features of different kinds was chosen and used to construct a vector in multidimensional feature space in order to select and count the similarity between author of anonymous text documents and authors of texts with a known identity. In the following paragraphs, the hybrid algorithm used in conducted experiments is described to provide information about the possibility of author attribution for short text documents. E-mail messages were used as samples of short texts. Such messages are constructed by an author usually in the same manner and their structure is not common for texts of different authors, and its’ format can be used for author description. The algorithm created embodies the idea of electronic message segmentation proposed in Learning to Extract Signature and Reply Lines from Email (2004) by V. R. Carvalo, W. W. Cohen. The text of each e-mail is divided in the following parts: greeting, citation, text, farewell, signature and advertisement. Then, a single message is assigned with a set of categories. The set can be empty, it can contain a single category or a group of categories. There are 16 subject fields that are distinguished among e-mails. Each of the categories is described by a predefined list of key words where each word is associated with a weight counted on the base of a whole corpus using function: 16

Scientific and Engineering Problem-Solving with the Computer, 1976.

472

M. Nawrot

weight

log

N df

(1)

where df – number of documents containing a given term, N – number of all documents in a corpus. For each message and for each normalized term a vector value is counted using equation:

w

tf weight M

(2)

where tf – number of term occurrences in a document, M – total number of words in a document. This is the equivalent to the indexing strategy called term frequency-inverse document frequency. Finally, for each category a cosine value is counted. Based on the values, categories are assigned to the given message. Iteratively, the category with a maximum value higher than the linearly incremented border value is assigned to the text document. Finally, the sociolinguistic features are associated which are later used to select potential authors from the group of all known authors. Three features are assigned – sex, age and the status of writer. Because the Polish is an inflectional language, sex can be easily recognized while checking the forms of verbs. If a form of a verb is in appropriate position in conjugation vector then female or male tag is assigned to the profile. It stays unknown otherwise. If sex detection is not consisted among all forms, the most frequent label is assigned. Age can be defined by minimum and/or maximum value of a range. It can also stay unknown. The border values are extracted on the basis of predefined rules. Rules input values depend on content words of the message, emoticons used, status of a writer and the signature form. Age labels are: below 25, below 35, above 25, above 30, above 40 and above 50. The following statuses are distinguished in the algorithm: professor, doctor, master of science, scientist, student, engineer. Each message can be associated with more then one status as well as none can be present in author's profile. Assigning is done in a similar way as age detection – using rules which are based on content words and signatures. After single messages processing, the author profiling is done. In this step, all text documents written by an author are taken into consideration while extracting features. The final feature vector is created which is used in the selection and classification. Each author is described by a single vector. The following sociolinguistic features P are extracted: author's age, author's sex, author's status. Where age is the shortest, none contradicting range, sex is equivalent to the most frequent sex label among all messages and status is the highest extracted status of an author. Semantic feature S is a list of pairs of category names and the number of messages in the given subject written by an author divided by the number of all messages in the given topic.

Automatic Author Attribution for Short Text Documents

473

Features based on e-mail structure E are: frequency list of the positions of cited text , frequency list of greetings , frequency list of farewells , number of messages with signature divided by the total number of messages , number of messages with advertisement divided by the total number of messages , and binary indicator if diacritic signs are used. In feature the position of text is one of the following values: only author's text, only cited text, author's text before cited text, cited text before author's text, alternate author's text and cited text (author's text first), alternate cited text and author's text (cited text first). Classical features are features based on characters, words and document structure. Character features C are: frequency of capital letters , frequency of alphabetical letters , frequency of digits, frequency of white characters, frequency of white characters, frequency of space signs , frequency of space signs in relation to the number of all white characters, frequency of tab signs in relation to the number of all white characters, and list of frequencies of punctuations. Word-based features W used are the following: average word length in characters, vocabulary richness, frequency of function words, frequency of short words (length < c l u s t e r i d = " E1c_1_2 " l e f t = " 1 " r i g h t = " 2 " t o k e n = " s o y o n s " l e x = " E1F2 | s o y o n s " / > < edge i d = " E1e008 " s o u r c e = " E1n023 " t a r g e t = " E1n005 " t y p e = " s u b s t " l a b e l = " xcomp " > Fig. 2. Fragment of DEP XML

5.2 Quotation Extraction As mentioned earlier, one of the main features of journalistic quotations is their mixed characteristic. It is indeed very common to find, in journalistic content, a mixed type of quotations, with both typographic quotes and for instance a finite subordinate clause containing the RS. It is also frequent that not the whole RS is between quotes, but only one or several chunks of it, as in Example 2. Besides, those chunks do not necessarily correspond to constituents. Thus, the simple matching of text surrounded by quotes does not provide a complete retrieval of RS parts. This is why S APIENS expands verbatims by reconstructing a coherent RS based on parsing results, hence putting forward information not directly retrievable by the user, while still distinguishing between different types of speech reporting. Linguistic studies about RS offer different ways for us to define those configurations, and thus quotations, in the current context of journalistic corpora. They usually distinguish between various forms of RS, mainly direct and indirect, based on the presence or absence of typographic signs and particular syntactic structures, among other criteria. Without discussing here detailed aspects of RS theories, we can mention the most frequent configurations used for reporting speech, especially in journalistic content. In those constructions, the RS part is considered to be the grammatical object of the verb introducing it. As such, it can be realized as a noun phrase or a finite subordinate clause. It is also considered as the object, although at a weaker degree, when the RS verb appears in an incident clause, after or in the middle of the quoted text. For a more precise and complete study of quotations in news wires and of formal features of quotation syntactic structures, we refer to [9] and [2]. When the dependencies produced by FRMG contains such configurations, the quotation extraction module looks up the list of 230 quotation verbs at our disposal and whose making is described in [9]. The module then appoints the sentence as an RS segment if the verb head of the main clause appears in it; the verb object, whether it is an NP or a subordinate clause, is selected as the quotation segment, while the subject of the verb is selected as the author. A few patterns were defined using a Perl-based query-langage over dependencies inspired by XPath, an exemple being given below to retrieve edges linking an active-voice citation verb (as source) with a sentential object (is_xcomp):

530

É. de La Clergerie et al.

dpat h is_xcomp { $ c i t a t i o n _ v e r b s −>{$_−>s o u r c e −>lemma } } { $_−>a p p l y ( d p a t h s o u r c e i s _ a c t i v e ) }

Adverbial and prepositional clauses are finally examined in search of possible other quotation satellites, such as the date of utterance or the audience in front of which it was uttered. If previously detected verbatims are included in the current sentence, the RS segment is extended from the beginning of the clause containing the first quote to the end of the clause containing the last one. Thus the quotation can include inserted nonverbatim chunks, and stretch out on several clauses. Besides, the presence of verbatims in a sentence forces the appointment of the sentence as an RS segment even when no specific configuration or quotation verb could be found by the parser. In such a case the same extension is applied in order to put the verbatim back in a syntactical interpretable context. RS are also frequently introduced without a specific verb, mainly in prepositional attribution phrases; in this case they are found before or after prepositional phrases such as “selon X” or “pour X” (“according to X”, “for X”). The parser also looks for these configurations. The clause or clauses introduced by the prepositional phrase is or are selected as the quotation, and the NP following the preposition is selected as its author. We have performed a limited evaluation of our work, mostly to guide our future efforts. We manually sampled 40 quotations from 40 different news items and evaluated both the span of the quotation and the correctness of the author. 32 quotations are found and in 19 cases, both the span and the author of the quotation are correct. Most other quotations lack an author (12 cases) or receive an incorrect one (7 cases, incl. 2 because of an erroneous anaphora resolution); 4 receive an incorrect span. More interestingly, 28 quotations exhibit patterns that would prevent a parsing-free processing chain from detecting the entire quotation span sucessfully (in most cases because not all the quotation lies between quotes, as explained above and illustrated by Example 2). Thanks to the use of the parser, S APIENS correctly identifies the span of 21 of those 28 cases.

6 Web Interface for Visualization The S APIENS Web interface provides a visualization of AFP news items with a focus on detected quotations and related named entities. The access is organized by quotations authors: one can choose among entities, showed in a cloud, to whom one or more quotations have been assigned. An access via a search menu will soon be available, in order to enable search of quotations made by a particular person, as well as a keyword search for the retrieval of quotations related to a particular topic. On this latter possibility, we can argue that the association between a topic and a quotation will be larger and thus richer than the one offered for instance by InQuotes: InQuotes makes out this link only if the keyword itself is present within the quotation, whereas in S APIENS’ case the keyword is part of the news item metadata among other related terms which can thus be linked to the quotation in an indirect although relevant fashion. This possibility of linkage is also due to the thematic homogeneity of a news item.

Extracting and Visualizing Quotations from News Wires

531

Fig. 3. Example of an enriched news item as visualized in S APIENS

Once an entity has been chosen, the user is directed to a clickable list of news items including quotations from the selected author. It gives access to the enriched view of each of these news items, i.e. with a set of highlighted text elements: quotation(s), with distinction between verbatims and non verbatim parts, the named entity selected as quotation author - if the author is referred to with a pronoun, a tooltip indicates to which entity the anaphora has been resolved; all entities detected in the news item can be highlighted, and a link is provided for each of them, to Google Maps for locations and to the corresponding Wikipedia page for the other types of entities.

7 Conclusions and Perspectives In this paper, we introduced S APIENS, a platform for quotations extraction that relies on a deep linguistic processing chain. In particular, we have described different modules for named entities extraction, verbatims extractions, deep parsing, anaphora resolution and quotation extractions, as well as a visualization interface. We showed how we applied this chain on a corpus of news wires from the Agence France-Presse (AFP) news agency. All components of S APIENS, including the processing chain and the resources it relies on, are free software. The information made available by S APIENS are richer and more accurate than other systems such as Google InQuotes, in part thanks to the use of a deep parser within the chain.

532

É. de La Clergerie et al.

In the future, S APIENS should evolve into an operational tool used by AFP journalists. More precisely, the S APIENS webservice will be queried by the news items editor used by journalists as soon as a news item is written, in order to automatically provide a list of descriptors to be used as metadata; these disambiguated descriptors may include named entities, topics13 , quotations associated with their author, or others; the journalist will then validate or correct these descriptors, which are meant to be used for sub-wires generation and for indexing purposes.

References 1. Danlos, L.: ILIMP: Outil pour repérer les occurrences du pronom impersonnel il. In: Proceedings of TALN 2005, Dourdan, France (2005) 2. Danlos, L., Sagot, B., Stern, R.: Analyse discursive des incises de citation. In: Actes du Deuxième Colloque Mondial de Linguistique Française, p. (à paraî tre), La Nouvelle-Orléans, Louisiane, USA (2010) 3. Lappin, S., Leass, H.J.: An algorithm for pronominal anaphora resolution. Computational Linguistics 20(4), 535–561 (1994) 4. Mitkov, R.: Robust pronoun resolution with limited knowledge. In: Proceedings of COLINGACL, pp. 869–875 (1998) 5. Pouliquen, B., Steinberger, R., Best, C.: Automatic detection of quotations in multilingual news (european commission - joint research centre). In: Proceedings of RANLP 2007 (2007) 6. Sagot, B., Boullier, P.: From raw corpus to word lattices: robust pre-parsing processing with S XPipe. Archives of Control Sciences, special issue on Language and Technology 15(4), 653–662 (2005) 7. Sagot, B., Clément, L., Villemonte de La Clergerie, E., Boullier, P.: The Lefff 2 syntactic lexicon for French: architecture, acquisition, use. In: Proc. of LREC 2006 (2006), http://atoll.inria.fr/~sagot/pub/LREC06b.pdf 8. Sagot, B., Boullier, P.: S XPipe 2: architecture pour le traitement présyntaxique de corpus bruts. Traitement Automatique des Langues (T.A.L.) 49(2), 155–188 (2008) 9. Sagot, B., Danlos, L., Stern, R.: A lexicon of french quotation verbs for automatic quotation extraction. In: Proceedings of LREC 2010, La Valette, Malte (2010) 10. Thomasset, F., Villemonte de la Clergerie, E.: Comment obtenir plus des méta-grammaires. In: Proceedings of TALN 2005, ATALA, Dourdan, France (June 2005), ftp://ftp.inria.fr/INRIA/Projects/Atoll/Eric.Clergerie/mg05.pdf 11. Villemonte de la Clergerie, E.: Convertir des dérivations TAG en dépendances. In: Proc. of TALN 2010 (July 2010)

13

Chosen within the IPTC ontology used by AFP (http://www.iptc.org/).

Using Wikipedia to Improve Precision of Contextual Advertising Alexander Pak Universit´e de Paris-Sud, Laboratoire LIMSI-CNRS, Bˆ atiment 508, F-91405 Orsay Cedex, France [email protected]

Abstract. Contextual advertising is an important part of the Web economy today. Proﬁt is linked to the interest that users ﬁnd in the ads presented to them. The problem is for contextual advertising platforms to select the most relevant ads. Simple keyword matching techniques for matching ads to page content give poor accuracy. Problems such as homonymy, polysemy, limited intersection between content and selection keywords as well as context mismatch can signiﬁcantly degrade the precision of ads selection. In this paper, we propose a method for improving the relevance of contextual ads based on “Wikipedia matching”. It is a technique that uses Wikipedia articles as “reference points” for ads selection. In our research, we worked on English language, but it is possible to port the algorithm to other languages.

1

Introduction

Internet advertising plays an important role in today’s Web. Worldwide spending on Internet advertising was total $65.2 billion in 2008, or nearly 10% of all ad spending across all media, and will grow 15-20% a year to reach $106.6 billion in 2011, or 13.6% of total ad spending, according to IDC’s Digital Marketplace Model and Forecast. A large part of Internet advertising consists of contextual advertising. In order to improve the user experience and increase the user’s attention, contextual advertising use mechanisms to select ads that are related to the user’s interest. It uses the page where ads are displayed to determine interest of the user. For example, if a user is viewing a page about “traveling in Europe”, then showing ads with “airplane tickets information” or “hotel information” would probably be a right choice. A textual advertisement usually consists of a title (on the average 2-5 words), a body (5-20 words) and a link to an advertiser’s Web page. In some systems, a publisher can also specify bid-phrases which are phrases that should match the page content. 1.1

Problem Statement

A traditional approach for selecting ads is based on keyword matching. A Web page is split into terms (such as words, phrases or n-grams) and they are matched Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 533–543, 2011. c Springer-Verlag Berlin Heidelberg 2011

534

A. Pak

against similar terms from an ad’s title, body, URL or bid-phrases. Information retrieval and natural language processing methods can be applied to make the match more eﬀective, such as the vector space model [9] or latent semantic indexing [3]. However, traditional keyword matching faces several problems that degrade its performance: – Homonyms and polysems1 cause semantic ambiguities and as a result lead to selecting irrelevant ads. An example of polysemy is a word “wood” which has a meaning of a piece of a tree as well as a geographical area with many trees. Homonyms can appear as common names, such as “plant” (manufacturing plant or living organism). They also appear as proper names, such as personal names (Condoleeza Rice or a football player Kevin Craft) or organizations names (Jaguar Cars, Chicago Bulls). The presence of such words in the page may lead to ads misplacements. For example, a placing a “grain product” advertisement on a Web page devoted to “Condoleeza Rice’s visit to Europe”. – The low intersection of keywords is caused by the limited size of the content of an ad. It is diﬃcult to establish the text similarity for keyword matching, because keywords of pages and ads have a low intersection [8]. Also, a concept can be represented as diﬀerent terms (synonyms). For example, “car” may be also referred to as “vehicle” or “automobile” and if an ad contains only one term it will not directly match related term contained in a page. – The context mismatch occurs when an ad does not match the topic of a page, while the keyword match can be exact [2]. An example of the context mismatch could be placing ads related to “tourism in China” on a Web page about “earthquake in China”. In order to solve these problems, we propose a method that we call “Wikipedia matching”. Due to the problems mentioned above, we cannot match pages and ads exactly. Therefore we introduce “reference points”, to which we can relate pages and ads. Through these reference points we establish matching between pages and ads. We chose a set of Wikipedia articles evenly distributed by diﬀerent topics to be the reference points. For each ad we ﬁnd Wikipedia articles related to that ad. The relation is established by a text similarity measure. For a given page we follow the same procedure and ﬁnd related Wikipedia articles. Using Wikipedia articles as reference points, we ﬁnd ads that share same related articles as the page, and construct a ranking function, that we call Wikipedia distance. We also show how to improve it using a dimension reduction technique. Finally, we show how to combine Wikipedia matching with traditional keyword matching to increase the relevance of selected ads. The reasons we have chosen Wikipedia among other encyclopedias and text corpora are as follows: 1

Homonyms are words with the same spelling but diﬀerent meanings, while polysems are words that bear multiple related meanings.

Using Wikipedia to Improve Precision of Contextual Advertising

535

– Wikipedia contains wide knowledge about many diﬀerent concepts, thus we can ﬁnd related articles for pages and ads – Articles in Wikipedia are regularly updated, therefore the knowledge base is always recent – Articles contain new terms that cannot be found in other linguistic corpora (i.e. no mention of “blogging” or “Google” in British National Corpus2 ) The probability of the negative eﬀect from homonymy and polysemy is low for Wikipedia matching because the relevance between an article and a page (or an ad) is high due to rich contents of articles. Because we select several articles for pages and ads, even if one or few articles are not relevant, the majority of articles determine the overall matching. For example, a Web page about football player “Kevin Craft” can be matched to some article devoted to “art and crafts”, however, there would be more articles on the sport and football thematic. The same reasoning is plausible for the context mismatch problem. The majority of articles that are related to a page will determine overall topic of that page. The lack of keywords problem is solved with Wikipedia matching. The probability of the intersection of keywords of pages and keywords of ads increases, because articles contain terms and deﬁnitions of the same concept in diﬀerent variations. 1.2

Contributions

The contributions of the paper are as follows: – Wikipedia matching algorithm: We propose an eﬃcient method for selecting relevant ads for a given page. We use Wikipedia articles as reference points to calculate a similarity score between pages and ads. Wikipedia matching can solve problems of traditional approaches to ads matching. Those problems are homonymy and polysemy, the low intersection of keywords, and the context mismatch. – Use of the Wikipedia similarity score with existing solutions: We show how our method can improve the accuracy of existing matching strategies, which are keyword matching and semantic-syntactic matching. We show that, by using our proposed similarity score, their performance is signiﬁcantly improved. – Evaluation experiments: We show that our proposed method improves existing approaches by increasing the average precision of selected ads. We also evaluate our method using TREC standard measure bpref-10 which conﬁrms the positive eﬀect of using Wikipedia matching. 1.3

Organizations

The rest of the paper is organized as follows. In Section 2, we discuss prior works on contextual advertising. In Section 3, we describe the keyword extraction 2

British National Corpus: http://www.natcorp.ox.ac.uk/

536

A. Pak

process. In Section 4, we explain our proposed Wikipedia matching method. Our experimental setup and evaluation results are presented in Section 5 Finally, we conclude our work in Section 6.

2 2.1

Related Work Keyword Matching

The study on contextual advertising is emerging with the growth of the Internet advertising market. The simplest and most straightforward method for selecting ads is to choose ads based on text similarity[5]. Classically, the cosine is used in information retrieval for gauging the similarity between text documents[9]. One of the ﬁrst results on the research about contextual advertising was presented by Ribeiro-Neto et al. in [8] where the vector space model for representing pages and ads was used. In their work, authors addressed the problem of low intersection between vocabularies of pages and ads. They have called this problem – the vocabulary impedance. In order to solve the problem Ribeiro-Neto et al. suggested to augment a page with additional keywords taken from other webpages that are similar to the considered page. They have called this approach: impedance coupling. Murdock et al. apply a noisy-channel approach in [6] representing the problem as the sparseness of the advertisement language. The authors assume that an ad can be seen as a noisy translation of a page. Using this assumption, authors select ads that provide the best translation for a given page. The authors applied relevance scores used in machine translation to determine the quality of machine translated texts, namely NIST and BLEU[10]. The main drawback of approaches based on keyword matching is due to the problems we have mentioned before. Homonymy, polysemy and context mismatch can dramatically degrade the relevance of selected ads. 2.2

Semantic Advertising

Semantic advertising applies semantic technologies to online advertising solutions. This technology semantically analyzes every web page in order to properly understand and classify the meaning of a web page, and accordingly ensures that the web page contains the most appropriate advertising. A recent research by Broder et al.[2] proposed a semantic approach to contextual advertising. The authors address problems of ambiguous keywords (homonymy and polysemy) and ambiguous page context (context mismatch). In order to overcome those problems, authors have proposed to apply automatic classiﬁcation to pages and ads. The obtained classiﬁcation information helps to ﬁlter out irrelevant ads and therefore increases the performance of ads selection. The authors use a commercial ontology, built especially for advertising purposes, to classify pages and ads. The ontology represents a hierarchical structure of advertising queries, and contains around 6000 nodes.

Using Wikipedia to Improve Precision of Contextual Advertising

537

While the proposed method showed good results, the method, however, is sensitive to the classiﬁcation precision. The obtained classiﬁcation precision in the paper is 70% for pages and 86% for ads, which gives around 60% probability of a chance for a successful semantic match.

3

Keyword Extraction

We prepared a set of web-pages (where ads will be displayed), a set of ads, and a set of Wikipedia articles to be used in our method. First downloaded a set of 100 news pages that were linked from Google News portal3 . The pages were processed with a content extraction tool[4] that extracts the main content by analyzing an HTML DOM tree, and prunes unnecessary parts such as navigation links and decoration elements. Next in our approach, we extract words from the title and the content of a page, and remove stopwords (i.e. words that bear no meaning, such as articles, prepositions etc.). Each word is then processed with a stemming algorithm which truncates suﬃxes of the word, and reduces it to a stem [7]. We experimented with the stemming algorithm and a lemmatization process. The advantage of the lemmatization is the ability to capture diﬀerent forms of a word (such as “better” and “good”) while a stemming algorithm cannot do that. However, the lemmatization algorithm is more complex and it fails for words that are not in the dictionary. Moreover, the lemmatization also requires a part of speech information which can be ambiguous. For example, word “saw” can be a noun “saw” or a verb “to see” in a past tense, and we need to know whether it is a noun or a verb for the correct lemmatization. Therefore we decided to use stemming, as it runs faster and can be applied to any word. Finally, we form n-grams out of stemmed words. We use unigrams for Wikipedia matching and keyword matching, and we use bigrams in the text classiﬁcation for semantic matching [2]. To form a dataset of ads, we queried search engines with simple queries such as “education” and “computers” that were formed out of titles of categories from the OpenDirectory project. For a given query, we receive a result page with blocks of sponsored search ads. Thus, we collected 7996 ads that form our ads dataset. Then, ads are put through the same process as pages, i.e. tokenization, stemming and stopwords ﬁltering. For Wikipedia matching, we selected 1000 featured articles4 on various topics. Because Wikipedia pages have the same HTML structure, it was easy to select only the main content avoiding unnecessary elements. The process of extracting terms from articles is similar to the one for pages and ads.

4

Wikipedia Matching

To construct the ranking function, we assume Wikipedia articles represent coordinates in a multidimensional space and use the Euclidean distance to calculate 3 4

http://news.google.com http://en.wikipedia.org/wiki/Wikipedia:Featured_articles

538

A. Pak

the overall score. The ranking is done according to the obtained Euclidean distance: ads with smaller distances to the page are ranked higher. We also apply a dimension reduction technique to improve the performance of the method. 4.1

Finding Similar Articles

To ﬁnd similar articles for the given page p and each of ad ai we use cosine similarity. The cosine similarity is equal to the cosine value of the angle between vectors of two documents [5]: sim(p, wi ) = cos ∠p, wi =

p · wi |p| · |wi |

(1)

where p is a vector representing a page p (it consists of tf-idf values of keywords of the page), wi is a vector of an article wi (it consists of tf-idf values of keywords of the article). We rank articles according to the similarity score in a decreasing order and select a set {pw} from the top-N of articles (in our experiments we use N = 100). Similarly, we calculate the cosine measure for ads. For each ad, we select the top-N of the most similar articles: {ah w}. According to our model, similar documents should be close because they are similar to the same set of articles and therefore have similar coordinate values for the corresponding dimensions. A natural way to calculate a similarity between two documents is to compute the Euclidean distance according to their coordinates. Let P W be a point representing the page p with the following coordinates: P W = (sim(p, pw1 ), sim(p, pw2 ), . . . , sim(p, pwN )) Let Ah W be a point representing an ad ah with the following coordinates: Ah W = (sim(ah , ah w1 ), sim(ah , ah w2 ), . . . , sim(ah , ah wN )) The distance between these two points is calculated as: wd(p, ah ) = (sim(p, pwj ) − sim(ah , ah wj ))2 ∀j,pwj =ah wj

We call the obtained value Wikipedia distance, and the ad with lowest value of Wikipedia distance would be considered as the most relevant for the given page. 4.2

Dimension Reduction

Because we use only the top-N of the most similar articles as coordinate values for each page and each ad to calculate the Wikipedia distance, other coordinates are regarded as equal to zero. Indeed, their value is very close to zero, because the similarity is very low or absent, thus our assumption of disregarding

Using Wikipedia to Improve Precision of Contextual Advertising

539

those coordinate does not aﬀect much the performance. Therefore if we represent all pages and ads in a matrix with rows corresponding to points and columns corresponding to the coordinates of these points in each dimension, then such matrix would have the majority of values equal to zero. To reduce the number of dimensions, while keeping the important data, we use principal component analysis (PCA). First we construct a matrix containing coordinates of all pages and ads: ⎤ ⎡ sim(p1 , w1 ) · · · sim(p1 , wR ) ⎥ ⎢ .. .. .. ⎥ ⎢ . . . ⎥ ⎢ ⎢sim(pQ , w1 ) · · · sim(pQ , wR )⎥ ⎥ ⎢ M =⎢ ⎥ ⎢ sim(a1 , w1 ) · · · sim(a1 , wR ) ⎥ ⎥ ⎢ .. . . .. .. ⎦ ⎣ . sim(aT , w1 ) · · · sim(aT , wR ) where Q is the total number of pages, T – the total number of ads, and R – the total number of articles. Next, we compute a covariance matrix S of M and ﬁnd an eigenvector for S. We take the ﬁrst eigenvector and use it to perform a dimension transformation (reduction). Finally, we obtain matrix M ∗ containing coordinates representing pages and ads in a reduced space. In our research we use 100 as the number of dimensions for the reduced space. When we obtain matrix M ∗ , we calculate the Wikipedia distance between a given page and an ad in the reduced space. The formula is the same as that for the original space (4.1), and the ad with the lowest value of Wikipedia distance is considered as the most relevant for the given page. 4.3

Combining Ranking Functions

In order to improve the precision of keyword matching, we combine it with our proposed Wikipedia distance by using weighted sum. The Wikipedia distance takes the minimum value of 0, which indicates the closest possible matching, and its value grows if there is less similarity between a page and an ad. To normalize the Wikipedia distance value, we ﬁnd the minimum and the maximum values in the whole dataset and then use them for the normalization: wd (p, ah ) = (max − wd(p, ah ))/(max − min)

(2)

The ﬁnal formula of the combination of keyword matching and the normalized Wikipedia distance in the reduced dimension space: kwd(p, ah ) = α · sim(p, ah ) + (1 − α) · wd (p, ah ) (α = 0.1)

540

A. Pak

Fig. 1. Average precision over all pages

5 5.1

Experiments Data and Methodology

We conducted experiments to evaluate our method using a dataset containing 100 pages and 7996 ads as shown in Table 1. Because the purpose of ads matching is to select the top-N of relevant ads for a given page, we evaluated the average precision for the top-1, top-3 and top-5 (usually the number of ads displayed on a Web page is not greater than 5). We also consider the sum of those precisions as an overall score. The same evaluation method was used in [8] and [2]. For each page, we collected human judgment scores that evaluate the relevance of selected ads by each of the compared methods. Selected ads on each page are marked by human judges as relevant or not. Table 1. Dataset characteristics Pages in dataset Ads in dataset Page-ad judgments Wikipedia articles

5.2

100 7996 4406 1000

Average Precision

We compared the performance of diﬀerent matching methods. We considered the following methods:

Using Wikipedia to Improve Precision of Contextual Advertising

541

Fig. 2. Sum of average precisions for the common dataset and the ambiguous dataset

– Traditional keyword matching (K) – Semantic-syntactic matching (S) [2] – Wikipedia distance after dimension reduction with keyword matching (W) The result of the average precision for all strategies is depicted in Figure 1. As we can observe from the graphs, the best matching is achieved by W combination that aggregates keyword matching and our proposed Wikipedia matching method. The next best result is obtained by S – semantic-syntactic approach, and ﬁnally the traditional keyword matching. We see that our proposed Wikipedia matching allows to improve the performance of traditional keyword approach signiﬁcantly, the obtained gain was 20%. 5.3

Results for the Ambiguous Dataset

To show that our proposed method helps to overcome the problems of keyword matching, we have selected a special dataset consisting of ambiguous pages. These are pages that contain ambiguous keywords or ambiguous context. The results for experiments run on the ambiguous dataset is presented in Figure 2. As we can see from the graph, our proposed solution performs well on both of datasets, while the performance of keyword matching degrades a lot. Thus, we conclude that by using our proposed technique, we can reduce the negative eﬀect caused by the problems of traditional keyword matching. 5.4

Performance Gain and t-Interval

We performed a statistical analysis of paired samples using the t-statistics [1] to prove that there is an evidence that our method is better than traditional

542

A. Pak

Fig. 3. 99.9% two-sided conﬁdence interval for performance gain

keyword matching and to obtain 99.9% two-sided conﬁdence interval for the performance gain. The results of obtained t-intervals are presented on Figure 3. As we can see from the graph, the most gain in precision is obtained by the W combination and it allows to achieve up to 19% of the performance gain.

6

Conclusion

Traditional methods of contextual advertising based on keyword matching are aﬀected by problems caused by homonymy and polysemy, low intersection of keywords, and context mismatch. The previously proposed approach of semanticsyntactic matching is sensitive to the document classiﬁcation precision and thus needs to be improved. In our work, we proposed a novel matching technique that uses Wikipedia articles as reference points to establish matching between ads and pages. Using Wikipedia as additional knowledge allows to overcome the problems of previous approaches. In our research, we worked on English language, but it is possible to port the algorithm to other languages. Experimental evaluations show that our proposed Wikipedia matching improves the precision of selected ads by traditional keyword matching. A statistical t-test was used to conﬁrm that our proposed method performs better than previous solutions. As a future work, we plan to test our method on other languages. We plan to conduct experiments on the diﬀerent number and quality of Wikipedia articles and their impact on overall matching performance. We also plan to exploit machine learning techniques for estimating parameter values of our algorithm.

Using Wikipedia to Improve Precision of Contextual Advertising

543

Acknowledgements This research has been conducted as a part of the master degree program at Korea Advanced Institute of Science and Technology. I would like to thank my supervisor Prof. Chin-Wan Chung for his encouragement and support.

References 1. Hayter, A.J.: Probability and Statistics for Engineers and Scientists. Duxbury, Belmont (2007) 2. Broder, A., Fontoura, M., Josifovski, V., Riedel, L.: Semantic approach to contextual advertising. In: Proc. of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands (2007) 3. Deerwester, S., Dumais, S., Furnas, G., Landauer, T., Harshman, R.: Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 407–491 (1988) 4. Jun, Z.: Comprehensive Perl Archive Network (2007), http://search.cpan.org/ ~jzhang/html-contentextractor-0.02/lib/html/contentextractor.pm 5. Manning, C.D., Sch¨ utze, H.: Foundations of statistical natural language processing. MIT Press, Cambridge (1999) 6. Murdock, V., Ciaramita, M., Plachouras, V.: A noisy-channel approach to contextual advertising. In: Proc. of the 1st International Workshop on Data Mining and Audience Intelligence for Advertising, San Jose, California, pp. 21–27 (2007) 7. Porter, M.F.: An algorithm for suﬃx stripping. Readings in Information Retrieval, 313–316 (1997) 8. Ribeiro-Neto, B., Cristo, M.: Impedance coupling in content-targeted advertising. In: Proc. of the 28th annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Salvador, Brazil, pp. 496–503 (2005) 9. Salton, G., Wong, A., Yang, C.S.: A vector space model for automatic indexing. Communications of the ACM 18(11), 613–620 (1975) 10. Zhang, Y., Vogel, S.: Measuring conﬁdence intervals for the machine translation evaluation metrics. In: Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI 2004), pp. 4–6 (2004)

Unsupervised Extraction of Keywords from News Archives Marco A. Palomino1 and Tom Wuytack2 1 University of Westminster 115 New Cavendish Street, London, W1W 6UW, United Kingdom [email protected] 2 Belga News Agency Rue Frederic Pelletier 8b, 1030 Brussels, Belgium [email protected]

Abstract. We present a comparison of four unsupervised algorithms to automatically acquire the set of keywords that best characterise a particular multimedia archive: the Belga News Archive. Such keywords provide the basis of a controlled vocabulary for indexing the pictures in this archive. Our comparison shows that the most successful algorithm is TextRank, derived from Google’s PageRank, which determines the importance of a word by the number of words with which it co-occurs, and the relative importance of those co-occurring words. Next most successful is information radius, originally used to estimate the overall semantic distance between two corpora, but here adapted to examine the contributions of individual words to that overall distance. Third most successful was the chi-square test, which determined which keywords were more typical of Belga’s Archive than a representative corpus of English language. Finally, the least successful approach was the use of raw frequency, whereby the most frequent words were the most important ones, unless they were present in a stop-word list. All four algorithms are readily portable to other domains and languages, though TextRank has the advantage that it does not require a comparison corpus. Keywords: Keyword extraction, unsupervised algorithms, TextRank, information radius, chi-square test, Europarl.

1 Introduction Keywords are linguistic atoms—typically words, pieces of words or phrases—used to characterise the subject or content of a document [1]. Previously, keywords have been employed to classify text, to build automatic indexes and to provide dense summaries. Recently, keywords have also been used to create ontological structures that support browsing through an archive [2]. As part of the EU-funded Video and Image Indexing and Retrieval in the Large-Scale (VITALAS) project [3], which aims to offer intelligent access to professional multimedia archives, we have experimented with several techniques for the automatic extraction of keywords. Our goal is to obtain keywords that typify the content of a large multimedia Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 544–555, 2011. © Springer-Verlag Berlin Heidelberg 2011

Unsupervised Extraction of Keywords from News Archives

545

archive owned by the Belga News Agency [4]. Later on, these keywords are employed as a controlled vocabulary for indexing the pictures in this archive. Belga’s archive covers Belgian and international news and current affairs—politics and economics, finance and social affairs, sports, culture and personalities. Due to its immense size—between 5,000 and 10,000 pictures are added on a daily basis—we cannot afford the choice of manually annotating Belga’s archive. Thus, we have explored the use of three unsupervised methods for the automatic extraction of keywords, namely, TextRank—a graph-based ranking algorithm developed by Mihalcea and Tarau [5]—the chi-square test—a statistical technique employed by Leech and Fallon [6] and Oakes and Farrow [7] to find words that are representative of a corpus—information radius—a measure of semantic similarity suggested by Dagan et al. [8]—and the use of raw frequency as an indication of the “relevance” of a keyword. We have evaluated these approaches by comparing their output with the feedback received from professional annotators, who determined whether the keywords retrieved by each method were truly descriptive of Belga’s archive. The remainder of this document is structured as follows: Section 2 offers an overview of related work. Section 3 describes Belga’s archive. Section 4 explains the different algorithms that we have considered for the automatic extraction of keywords. Section 5 reports on the evaluation of our experiments, and, finally, Section 6 states our conclusions.

2 Related Work Research on the automatic extraction of keywords concentrates largely on supervised learning algorithms, where a system is trained to recognise keywords in a text, based on lexical and syntactical features. This approach has been studied by Turney [9], who combined heuristic rules with a genetic algorithm into a system—GenEx—that identifies keywords in a document. A different algorithm proposed by Frank et al. [10]—the Kea key-phrase extraction algorithm—applies a naive Bayes learning scheme and improves the results observed on the same dataset used by Turney. Both Turney and Frank et al. reported on the precision of their systems: a 29% precision is achieved with GenEx for five key-phrases extracted per document [9], and a 18.3% precision is achieved with Kea for fifteen key-phrases per document [10]. Another supervised algorithm, which has successfully outperformed previously published results, was implemented by Hulth [11], who extracted keywords from a collection of 500 abstracts from the Inspec database [12]. Hulth integrated part of speech information into the learning process, and doubled the accuracy of her system by adding linguistic knowledge to the term representation. While we are aware of the advantages of machine-learning systems, we are discouraged by their inability to adapt to new languages and domains, as additional training data is required for each new collection. We have therefore chosen an unsupervised approach, which begins by normalising the collection: removing stop-words, detecting proper-names and stemming the text. This approach is similar to the one employed by OmniPaper [13], which extracts keywords from newspaper articles. Unlike OmniPaper, which selects keywords according to the number of articles that contain them, we have tested four unsupervised algorithms: a graph-based algorithm—TextRank—a statistical method—the chi-square test—an information

546

M.A. Palomino and T. Wuytack

theoretic measure—information radius—and the use of raw frequency. Further details about each of these algorithms are given in Section 4.

3 Belga News Agency Archive Belga’s content is published in several formats: text, picture, audio and video. However, this document concentrates on still images and their associated captions. A caption is a free-text field whose content is written by photographers, and offers an explanation or designation accompanying a picture posted on Belga’s website. Each group of photographers has its own conventions and styles to present information. As a consequence, certain captions may include names of photographers and their initials, acronyms of press agencies, the dates when the pictures were taken or published, and some other ancillary information. Since none of these particulars are deleted before posting, we decided to keep them in our analysis too. Belga publishes content in different languages: Dutch and French are used for the text, and English is preferred for picture captions, because it facilitates exchanges with other international agencies. To ensure that we had enough material to carry out our research, Belga granted us access to a set of 2,347,535 captions that were published on its website prior to 30 March 2009. Figure 1 displays an example of a typical picture and caption posted on Belga’s website.

Italian player Gianluca Zambrotta (R) fights for the ball with Oleh Gusev (L) of Ukraine during their Euro 2008 qualifying soccer match between Ukraine and Italy at Olympiyskiy stadium in Kiev. Fig. 1. Example of a Picture and Caption

For evaluation purposes, we had to restrict the number of captions taken into account in our study, and the number of keywords produced by each of the methods that we wanted to assess. Both the number of captions and the number of keywords had to be manageable for the professional annotators who reviewed our work. However, we made an effort to gather a representative sample of 100,000 captions that incorporates both Belga’s and Belga’s users’ interests. To obtain the sample of 100,000 captions that we utilised in our evaluation, the following procedure was undertaken: 40,000 captions from Belga’s query log files for

Unsupervised Extraction of Keywords from News Archives

547

the period between June 2007 and October 2007 were chosen randomly. This guaranteed that at least 40% of the captions in our sample were relevant to Belga users— because they had already been retrieved as a result of users’ searches. Another 30,000 captions were taken from Belga’s collection for the same period—between June 2007 and October 2007—but we made sure that none of these captions was included in the former 40,000 set, and was not in the query logs either. Finally, 30,000 different captions were sampled uniformly from the entire collection, regardless of their presence in the query logs. The final set of 100,000 captions constitutes a representative sample of the whole archive.

4 Automatic Extraction of Keywords The task of keyword extraction consists of identifying the terms that best describe a corpus. The simplest approach is the use of a frequency criterion to select “important” terms. However, this generally leads to poor results [5]. Hence, other methods have to be explored. All the methods reported here begin by normalising the collection of Belga captions: all the text that is not identified as a proper name is converted to lower case, punctuation is removed, and extremely common and semantically non-selective words are deleted—the stop-word list that we use was built for the experimental SMART information retrieval system [14]. Normalisation stems all text too, except for the proper names—the particular algorithm for stemming English words that we use is Porter’s algorithm [15]. Porter’s Algorithm is the most common algorithm for stemming English text, and one that has repeatedly shown to be “effective”. Proper names are identified by means of an heuristic suggested by Hofland and Johansson [16]. Such an heuristic seeks for proper nouns assuming that they are spelled with an initial capital in all their occurrences, as opposed to common nouns, which are spelled with a lower-case initial once or more times. This simple approach provides an advantage over other more sophisticated strategies by avoiding the need of manually annotated training data. Also, we favoured this choice over others because former research has shown that named entity recognition software developed for one domain does not typically perform well on other domains [17]. A total of 57,799 proper names were identified by Hofland and Johansson’s heuristic in the sample of 100,000 captions selected for our evaluation. Proper names were not stemmed and were not converted to lower case, but they were counted as potential keywords in our analysis. A total of 117,203 keywords were found—including proper names. After named entity recognition and normalisation take place, we apply one of the algorithms described below. 4.1 TextRank The first method that we have tested is an unsupervised graph-based ranking algorithm. Essentially, graph-based ranking algorithms are a way of deciding the importance of a vertex within a graph, based on global information recursively drawn from the entire graph.

548

M.A. Palomino and T. Wuytack

As explained by Mihalcea and Tarau [5], the basic idea behind a graph-based ranking algorithm is that of voting. When one vertex links to another one, it is casting a vote for that vertex. The higher the number of votes that are cast for a vertex, the higher the importance of the vertex. Moreover, the importance of the vertex casting the vote determines how important the vote itself is. Thus, the score associated with a vertex is based on the votes that are cast for it, as well as on the score of the vertices casting these votes [2]. Formally, let G=(V,E) be a graph with a set of vertices V and a set of edges E, where E VxV. For a given vertex vi, let In(vi) be the vertices that point to it, and let Out(vi) be the vertices that vi points to. Using a ranking model inspired by Brin and Page’s model [18], the score of vertex vi is defined as follows

⊂

⎡ ⎤ 1 S (vi ) = (1 − δ ) + δ ⎢ ∑ S (v j ) ⎥ , ⎢⎣v j ∈In ( vi ) Out (v j ) ⎥⎦ where δ is a damping factor usually set to 0.85. In the particular case of our keyword extraction application, the stems derived from Belga captions represent the vertices to be ranked. An edge between any two vertices is added to the graph whenever their corresponding stems co-occur in the same caption.

Original caption: Normalised caption: Scores:

Today started trial against Hassan DERIN in Tongeren todai start trial Hassan DERIN Tongeren trial: 0.00071 start: 0.00065 todai: 0.00036 Hassan: 0.00010 DERIN: 0.00001 Tongeren: 0.00001

Fig. 2. Example of TextRank Computation

Unsupervised Extraction of Keywords from News Archives

549

It should be observed that a single graph—undirected and unweighted—is built for the entire collection of captions. Each stem represents a single vertex, and the score associated with each vertex is set to an initial value of 1, and the ranking algorithm explained above is executed for several iterations until it converges—usually for 2030 iterations, at a threshold of 0.0001. Once a final score is obtained for each vertex, vertices are sorted in reversed order of their score, and those with the highest scores are retained for the review of the annotators. While the number of vertices retained after the ranking process may be set to any fixed value, we had to restrict our choice to 2,500, which is as much as the annotators were able to review on time for this study. Figure 2 displays an original caption, its normalised version, the part of the total graph that corresponds to the vertices associated with this caption, and the final scores for each vertex. We chose one of the smallest captions available in the collection for illustration purposes. Nevertheless, the graph displayed in Figure 2 includes not only the edges derived from the selected caption, but also those that exist between these vertices as a consequence of their co-occurrence in other captions. The graph that represents the entire corpus of 100,000 captions chosen for the evaluation is made of 117,203 vertices. 4.2 Chi-Square Test The chi-square test is a useful statistical technique to decide if observed data deviates from expected data under a particular hypothesis [19]. Such a hypothesis is whether two samples are different enough in a specific characteristic to be considered members of different populations. In this sense, the chi-square test is used to assess the null hypothesis—H0—which states that there is no significant difference between observed and expected data. Chi-square values with low probability lead to the rejection of H0, and it is assumed that a factor, other than chance, creates the deviation between expected and observed data. We have employed the chi-square test to extract keywords that are truly characteristic of Belga’s archive. The chi-square test compared Belga captions with a model of general English language. The words that deviated from the model were very specific of Belga’s archive, and thus made an appropriate set of keywords. To apply the chi-square test, we require a comparison between the corpus for which we want to find out its characteristic keywords and a second corpus, sometimes known as comparison corpus. Since we wanted to identify Belga keywords, we required a comparison corpus that did not belong to the news-agency domain. After considering our choices, we opted for Europarl [20], a parallel corpus derived from the Proceedings of the European Parliament. Europarl is available in 11 European languages, and it offers sentence aligned text for statistical machine translation. Hence, we could use it to apply the chi-square test to archives in other languages too. This is a definite advantage for us, because, later on, we plan to extract keywords from the archives owned by the Institut National de l’Audiovisuel [21], which is in French, and the Institut für Rundfunktechnik [22], which is in German. The computation of the chi-square test proceeds as follows: for every word that occurs in the caption collection, we need to find out how often it appears in both Europarl

550

M.A. Palomino and T. Wuytack

and Belga captions. For instance, the word miner, which appears 241 times in Belga captions, occurs only 125 times in Europarl. We need to know whether these figures arose by chance, or whether the word miner is genuinely more typical of Belga captions. Also, we need to know the total number of words in both corpora, so that we can work out how many words are not the word miner. All these values are arranged in a 2 by 2 table called a contingency table. Figure 3 shows the contingency table for the word miner. Number of times miner is found in Belga Number of words other than miner in Belga

Number of times miner is found in Europarl Number of words other than miner in Europarl

Fig. 3. Contingency Table for the Word miner

Since the total number of words and the number of appearances of a word in each corpus are values that we can count directly, they are called observed values. From the observed values, we can calculate the corresponding expected values, if there were no tendency for words to appear more often in either Belga captions or Europarl. We calculate the chi-square value (χ2) for a particular word whose contingency table is τ using the formula

χ2 =

∑

Each cell in τ

(ρ − ε )2

ε

,

where ρ is the observed value for each cell in τ, and ε is the expected value for each cell in τ. The expected value ε for each cell is calculated from the contingency table using the formula

ε=

Row total ⋅ Column total . Grand total

If the resulting chi-square value is more than 10.83, we can be 99.9% confident that the word really is more typical of Belga captions than of Europarl1. For example, since the chi-square value for miner is 363.31, we can be 99.9% confident that miner is genuinely more typical of Belga captions than of Europarl. As with TextRank, we retained 2,500 keywords produced by the chi-square test. Indeed, we retained the 2,500 keywords whose chi-square values were the highest, and we passed them on to the annotators for their review. In Section 5, we present the results of this review. 1

A relative standard serves as the basis for accepting or rejecting H0. In computational linguistics research, the relative standard is usually set to q=0.001, where q is the probability that the deviation of the observed values from the expected values is due to chance alone. If q is less than or equal to 0.001, then H0 is rejected, and the values are not independent from each other. The critical chi-square value associated with q=0.001 is 10.83. Hence, chi-square values higher than 10.83 are associated with a statistically low probability that H0 is true.

Unsupervised Extraction of Keywords from News Archives

551

4.3 Information Radius Kullback-Leibler (KL) divergence, also known as relative entropy is defined as follows

D( p || q) = ∑ pi ⋅ log 2 i

pi , qi

where pi is the proportion of words in corpus p which are the word i, and qi is the proportion of words in corpus q which are the word i. The contribution of each individual word to the overall KL divergence between the two corpora is represented by

pi ⋅ log2

pi , qi

and the overall KL divergence is found by adding up the contributions of all the individual words. This measure has the disadvantage that a word which is found in corpus p but not in corpus q produces an infinitely large contribution to the overall value. Also, the measure is not symmetrical: the KL divergence between p and q is not the same as that between q and p. To overcome these problems when using KL divergence as a measure of semantic similarity, Dagan et al. [8] proposed the related information radius measure, defined as follows

⎛ p+q⎞ ⎛ p+q⎞ D⎜⎜ p ⎟⎟ + D⎜⎜ q ⎟. 2 ⎠ 2 ⎟⎠ ⎝ ⎝ While information radius gives an overall measure of difference between two corpora, the contributions of the individual words in corpus p to this overall difference are each given by

pi ⋅ log2

pi . ⎛ pi + qi ⎞ ⎟ ⎜ ⎝ 2 ⎠

Thus, the individual words scoring most highly according to this measure are the most typical of corpus p as opposed to corpus q. Oakes [23] used this method to compare the detective fiction and the romantic fiction components of the FLOB corpus [24]. Findings were that words indicating males—such as man and he—past tense—was and been—and the word money were more typical of detective fiction, while words indicating females—her and she—present tense—are and be—and the word love were more typical of romantic fiction. In our study, once again we look for words which are typical of Belga captions, as opposed to Europarl, and the 2,500 keywords with the highest scores retrieved by this method were reviewed by the annotators.

552

M.A. Palomino and T. Wuytack

4.4 Raw Frequency The final method assessed in our study is the use of a frequency criterion, whereby the most frequent stems in the evaluation sample are considered characteristic of Belga captions and therefore selected as keywords. Although we did not expect this method to perform better than the others, we decided to test it as a baseline comparison for the rest of the approaches. The sample of 100,000 Belga captions used for our evaluation constitutes a corpus of 5,198,692 words. After stop-word filtering and stemming, the corpus is reduced to 117,203 unique stems. Table 1 displays the statistics for the most frequent stems found in the sample. Table 1 shows as well the original words from which the stems came from. It should be observed that we limited the number of keywords extracted by raw frequency to 2,500. Consequently, we retained only the 2,500 most frequent stems in the sample. Table 1. Most Frequent Stems in Belga’s Evaluation Sample Stem match belgian world pictur presid dai minist soccer championship show usa nation open york team year germani celebr meet final

Original Words match, matching, matches, matched belgian, belgians world, worlds picture, pictured, pictures, picturing president, presidents, presidency, presiding, presides... days, day, dai, dayed minister, ministers soccer championship, championships show, shows, showed, showing, showings, showes usa national, nation, nations, nationals, nationally... opening, open, opened, opens, openings, opener... york teams, team, teamed years, year germany celebrate, celebrates, celebration, celebrity... meeting, meet, meets, meetings final, finals, finally, finale, finalize, finalized...

Frequency 18,384 14,711 14,326 14,029 13,745 13,573 12,532 11,208 10,857 9,899 9,473 8,739 8,630 8,390 8,352 8,149 7,929 7,804 7,507 7,363

5 Evaluation Professional annotators reviewed the keywords extracted by each of the algorithms described in Section 4, and decided whether they were descriptive of our chosen sample of 100,000 selected captions. The review was performed in a binary fashion: each keyword was either accepted, which means it was truly descriptive of our 100,000 captions, or rejected, which means it was not descriptive of them.

Unsupervised Extraction of Keywords from News Archives

553

To guarantee a blind review, the keywords extracted by the different methods were gathered together in a single set, which was sorted alphabetically. The annotators were given this set, without any explanation regarding the methods we used to acquire it, or the actual scores assigned to each keyword. The annotators were familiar with Belga’s archive and their review was made solely on the basis of their knowledge of that archive. To evaluate the efficiency of the algorithms, we calculated their precision at different cut-off points, considering only the topmost keywords extracted in each case. Table 2 shows the comparison of the precision of the different methods at 500 keywords—p@500—1,000 keywords—p@1,000—1,500 keywords—p@1,500—2,000 keywords—p@2,000—and 2,500 keywords—p@2,500. The highest precision value achieved at each cut-off point is displayed in bold font. Clearly, TextRank outperforms the rest of the approaches. Table 2. Evaluation Results

p@500 p@1,000 p@1,500 p@2,000 p@2,500

TextRank 0.52 0.43 0.39 0.34 0.32

Chi-Square 0.39 0.33 0.29 0.26 0.23

Information Radius 0.45 0.38 0.33 0.30 0.277

Raw Frequency 0.27 0.25 0.25 0.20 0.20

An important aspect of TextRank is that it does not require domain or language specific annotated corpora. It does not demand a comparison corpus either, unlike the chi-square test and the information radius. Intuitively, TextRank works well because it does not rely exclusively on the local context of a keyword in a particular caption, but rather it takes into account information recursively drawn from the entire corpus. Through its iterative mechanism, TextRank goes beyond the analysis of co-occurrence, and ranks keywords based on the importance of other keywords they are connected to.

6 Conclusions We have explored the use of four unsupervised algorithms to automatically acquire a set of keywords that characterise the content of a news archive. Our work has been carried out as part of the VITALAS project, using data supplied by Belga News Agency. To confirm that the keywords that we extracted were descriptive of Belga’s archive, professional annotators reviewed our results manually. According to such a review, the TextRank algorithm outperformed the rest of the algorithms considered. TextRank derives from Google’s PageRank. However, rather than judging the importance of a Web page by the number of Web pages pointing to it and their relative importance, we judge the importance of a keyword by the number of words with which it co-occurs, and the relative importance of those words. We were particularly interested in the application of this algorithm, because it does not require deep

554

M.A. Palomino and T. Wuytack

linguistic knowledge or a comparison corpus to yield results. TextRank is readily portable to other domains and languages. Therefore, the same methodology for the automatic extraction of keywords that we have described in this paper could be adapted easily to other multimedia archives. Acknowledgments. This research was supported under the EU-funded VITALAS project—project number FP6-045389. The authors are very grateful to Belga News Agency for providing the data used to carry out their research, and to the following people for participating in the evaluation: Jonathan Dujardin, Jeroen Van Den Haute, Katrien Vanmechelen, Sini Cheang and Hecham Azarkan.

References 1. Belew, R.K.: Finding Out About: A Cognitive Perspective on Search Engine Technology and the WWW. Cambridge University Press, Cambridge (February 2001) 2. Coursey, K., Mihalcea, R., Moen, W.: Automatic Keyword Extraction for Learning Object Repositories. In: Proceedings of the Conference of the American Society for Information Science and Technology, Columbus, Ohio (October 2008) 3. VITALAS, Video and Image Indexing and Retrieval in the Large Scale (2009), http://www.vitalas.org/ 4. Belga, Belga News Agency (2009), http://www.belga.be/ 5. Mihalcea, R., Tarau, P.: TextRank: Bringing Order into Texts. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain, pp. 404– 411 (July 2004) 6. Leech, G., Fallon, R.: Computer Corpora - What do They Tell Us about Culture? International Computer Archive of Modern and Medieval English (16), 85–100 (1992) 7. Oakes, M.P., Farrow, M.: Use of the Chi-Squared Test to Examine Vocabulary Differences in English Language Corpora Representing Seven Different Countries. Literary and Linguistic Computing 22(1), 85–99 (2007) 8. Dagan, I., Lee, L., Pereira, F.: Similarity-Based Methods for Word Sense Disambiguation. In: Proceedings of the Association for Computational Linguistics, Madrid, Spain, pp. 56– 63. Morgan Kaufmann Publishers, San Francisco (July 1997) 9. Turney, P.: Learning to Extract Keyphrases from Text. Technical Report NRC-41622, National Research Council of Canada (February 1999) 10. Frank, E., Paynter, G.W., Witten, I.H., Gutwin, C., Nevill-Manning, C.G.: DomainSpecific Keyphrase Extraction. In: Proceedings of the International Joint Conference on Artificial Intelligence, Stockholm, Sweden, pp. 668–673. Morgan Kaufmann Publishers Inc., San Francisco (July 1999) 11. Hulth, A.: Improved Automatic Keyword Extraction Given More Linguistic Knowledge. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Sapporo, Japan. Association for Computational Linguistics, pp. 216–223 (July 2003) 12. Inspec: The Inspec Database, http://www.theiet.org/publishing/inspec/ 13. Martínez-Fernández, J.L., García-Serrano, A., Martínez, P., Villena, J.: Automatic Keyword Extraction for News Finder. In: Nürnberger, A., Detyniecki, M. (eds.) AMR 2003. LNCS, vol. 3094, pp. 99–119. Springer, Heidelberg (2004) 14. Buckley, C.: Implementation of the SMART Information Retrieval System. Technical Report TR85-686, Computer Science Department, Cornell University, Ithaca, New York (May 1985)

Unsupervised Extraction of Keywords from News Archives

555

15. Porter, M.: An Algorithm for Suffix Stripping. Program 14(3), 130–137 (1980) 16. Hofland, K., Johansson, S.: Word Frequencies in British and American English. Norwegian Computing Centre for the Humanities, Bergen, Norway (1982) 17. Poibeau, T., Kosseim, L.: Proper Name Extraction from Non-Journalistic Texts. In: Proceedings of Computational Linguistics in the Netherlands, Tilburg, The Netherlands, pp. 144–157 (November 2000) 18. Brin, S., Page, L.: The Anatomy of a Large-Scale Hypertextual Web Search Engine. Computer Networks and ISDN Systems 30(1-7), 107–117 (1998) 19. Greenwood, P.E., Nikulin, M.S.: A Guide to Chi-Square Testing. Wiley-InterScience, New York (1996) 20. Koehn, P.: Europarl: A Parallel Corpus for Statistical Machine Translation. In: Proceedings of the Machine Translation Summit, Thailand (September 2005) 21. INA: Institut National de l’Audiovisuel, http://www.ina.fr/ 22. IRT: Institut für Rundfunktechnik, http://www.irt.de/ 23. Oakes, M.P.: Statistical Measures for Corpus Pro¯ ling. In: Proceedings of the Open University Workshop on Corpus Profiling, London, UK (October 2008) 24. Hundt, M., Sand, A., Siemund, R.: Manual of Information to Accompany the Freiburg LOB Corpus of British English (FLOB), http://khnt.hit.uib.no/icame/manuals/flob/INDEX.HTM

Automatic Evaluation of Texts by Using Paraphrases Kazuho Hirahara1, Hidetsugu Nanba1, Toshiyuki Takezawa1, and Manabu Okumura2 1

Graduate School of Information Sciences, Hiroshima City University, 3-4-1 Ozukahigasi, Asaminamiku, Hiroshima 731-3194 Japan {hirahara,nanba,takezawa}@ls.info.hiroshima-cu.ac.jp 2 Precision and Intelligence Laboratory, Tokyo Institute of Technology, 4259 Nagatsuta, Midoriku, Yokohama 226-8503 Japan [email protected]

Abstract. The evaluation of computer-produced texts has been recognized as an important research problem for automatic text summarization and machine translation. Traditionally, computer-produced texts were evaluated automatically by ngram overlap with human-produced texts. However, these methods cannot evaluate texts correctly, if the n-grams do not overlap between computer-produced and human-produced texts, even though the two texts convey the same meaning. We explored the use of paraphrases for the refinement of traditional automatic methods for text evaluation. To confirm the effectiveness of our method, we conducted some experiments using the data from the Text Summarization Challenge 2. We found that the use of paraphrases created using a statistical machine translation technique could improve the traditional evaluation method. Keywords: text summarization, machine translation, text evaluation, synonyms.

1 Introduction The evaluation of computer-produced texts has been recognized as an important research problem for text summarization and machine translation. Traditionally, computer-produced texts were evaluated by n-gram overlap with human-produced texts [6,7,10]. However, these methods cannot evaluate texts correctly, if the n-grams do not overlap between the computer-produced and human-produced texts, even though the two texts convey the same meaning. Therefore, we explore the use of paraphrases for the refinement of traditional automatic methods for text evaluation. Several evaluation methods using paraphrases were proposed in text summarization [14] and machine translation [3,4,13], and their effectiveness was confirmed. However, these studies did not discuss what paraphrases techniques gave more accurate text evaluation. We analyzed 318 paraphrases in texts to be evaluated, and classified them into five categories. Then we examined several paraphrase methods that covered four of those categories (about 70% of the 318 paraphrases). We evaluated texts using the data of the Text Summarization Challenge 2. We found that our method could improve a traditional evaluation method. Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 556–566, 2011. © Springer-Verlag Berlin Heidelberg 2011

Automatic Evaluation of Texts by Using Paraphrases

557

The remainder of this paper is organized as follows. Section 2 describes related work. Section 3 describes the benefits of paraphrases in text evaluation. Section 4 explains our evaluation method using paraphrases. To investigate the effectiveness of our method, we conducted some experiments, and Section 5 reports on these. We present some conclusions in Section 6.

2 Related Work We describe the related studies of "automatic evaluation of texts" and "text evaluation using paraphrases" in Sections 2.1 and 2.2, respectively. 2.1 Automatic Evaluation of Texts Several measures for evaluating computer-produced texts have been proposed [6,7,10]. BLEU [10] was developed as a measure of automatic evaluation for machine translation. It compares the n-grams of the candidate with the n-grams of the reference translation, and counts the number of matches. These matches are position independent. The quality of the candidate translation depends on the number of matches. ROUGE-N [6,7] is a standard evaluation measure in automatic text summarization. The measure compares the n-grams of the two summaries, and counts the number of matches. The measure is defined by the following equation.

∑ ∑ Count ( gram ) ROUGE − N = ∑ ∑ Count ( gram ) S ∈ R gram N ∈S

S ∈ R gram N ∈S

match

N

N

where N is the length of the n-gram, gramN, and Countmatch(gramN) is the maximum number of n-grams co-occurring in a candidate summary and a set of reference summaries. Lin examined ROUGE-N with values of N from one to four, and reported that ROUGE-N had a high correlation with manual evaluation when N was one or two. In our work, we focus on evaluation of computer-produced summaries, and use ROUGE-N as a baseline method for text evaluation. 2.2 Text Evaluation Using Paraphrases Several evaluation methods using paraphrases were proposed in text summarization [14] and machine translation [3,4,13]. Zhou et al. [14] proposed a method "ParaEval" to obtain paraphrases automatically using a statistical machine translation (SMT) technique. If translations of two terms X and Y are the same term, then the terms X and Y are considered to be paraphrases. Based on this idea, they automatically obtained paraphrases from a translation model, which was created from pairs of English and Chinese sentences using the SMT technique. They then used these paraphrases for the improvement of ROUGE-N. In our work, we also use paraphrases acquired by the SMT technique as paraphrase method.

558

K. Hirahara et al.

In addition to the SMT-based paraphrases, we examined another method for automatic acquisition of paraphrases. Lin [8] and Lee [5] proposed a method for calculating the similarity between terms, called "distributional similarity". The underlying assumption of their approach is that semantically similar words are used in similar contexts. Therefore, they define the similarity between two terms as the amount of information contained in the commonality between the terms, divided by the amount of information in the contexts of the terms. In our work, we use "distributional similarity" as a method for acquiring paraphrases.

3 The Benefits of Paraphrases in Text Evaluation To investigate the benefits of paraphrases in text evaluation, we compared multiple summaries created from the same text. 3.1 Data In this investigation, we used 30 Japanese editorials1 from the Mainichi newspaper databases of 1998 and 1999. For each editorial, we asked 10 human subjects to create abstract-type summaries for a summarization ratio of 20%, which were produced to try to obtain the main idea of the editorial without worrying about sentence boundaries. We compared these summaries and obtained 318 paraphrases. 3.2 Paraphrases in Text Evaluation We classified the 318 paraphrases into the following five categories. A) Synonymous expressions with different Japanese characters The senses of two expressions are the same, but they were expressed with different Japanese characters. There were 64 (20.1%) paraphrases in this category. B) Word-level synonymous expressions Two words have the same sense, but are different, such as " " (time) and " " (moment). There were 78 (24.5%) paraphrases in this category. Synonym dictionaries are required for paraphrases in this category.

時間

時

C) Phrase-level synonymous expressions Two phrases have the same sense, but are expressed differently, such as " " (cannot be divided) and " " (linked to each other). There were 38 (11.9%) paraphrases in this category. Phrase-level synonym dictionaries are required for paraphrases in this category.

い

繋がっている

切り離せな

D) Clause-level synonymous expressions Two clauses have the same sense, but are expressed differently, such as "X Y " (If it were not for X, Y would not succeed.) and "X

ばはできなかった 1

がなけれによりY

These editorials were used in Text Summarization Challenge [2], which is an evaluation workshop of text summarization, conducted in the NTCIR workshop.

Automatic Evaluation of Texts by Using Paraphrases

559

ができた" (Y succeeded because of X). Changing voices and transitive/intransitive

alternations are also classified in this category. There were 39 (12.3%) paraphrases in this category. Clause-level synonym dictionaries or more sophisticated paraphrasing techniques are required for paraphrases in this category. E) Other paraphrases We can recognize that two expressions are the same meaning by guessing from their contexts. There were 99 (31.2%) paraphrases in this category. Taking account of paraphrases in this category is very difficult using current natural language processing technologies. Among these paraphrases, we focused on categories A, B, C, and D, and examined several paraphrase resources for these four categories.

4 An Automatic Method of Text Evaluation Using Paraphrases In this section, we describe our text evaluation method using paraphrases. In Section 4.1., we describe the procedure for our method. In Section 4.2., we explain several paraphrase methods for categories A, B, C, and D. 4.1 Procedure for Text Evaluation We evaluated texts using the following procedure, which resembles Zhou's ParaEval [14]. Step 1: Search using a greedy algorithm to find (C) phrase-level or (D) clause-level paraphrases matches. Step 2: The non-matching fragments from Step 1 are then searched using a greedy algorithm to find (A) paraphrases using different Japanese characters or (B) wordlevel paraphrases or synonym matches. Step 3: Search by literal lexical unigram matching on the remaining text. Step 4: Count the agreed words in a reference summary from Steps 1, 2, and 3, and output the Recall value for the reference summary as an evaluation score. 4.2 Paraphrase Methods We used the following four paraphrase methods for summary evaluation. z z z z

SMT (automatic): Paraphrases using the statistical machine translation (SMT) technique. DS (automatic): Paraphrases using the distributional similarity method. Word (manual): WordNet dictionary. NTT (manual): NTT Goi-Taikei dictionary.

560

K. Hirahara et al.

In the following, we explain the details of each paraphrase method. Paraphrases using the statistical machine translation technique (SMT) If translations of two expressions X and Y are the same expression, then the expressions X and Y are considered to be paraphrases. Therefore, we constructed a translation model from 150,000 pairs of English-Japanese sentences automatically extracted [12] from the Yomiuri newspaper database and Daily Yomiuri using a translation tool Giza++2. In this translation model, we deleted English-Japanese expression pairs, in which the number of words and parts of speech of each word were different. For example, we don’t consider a noun phrase and a verb phrase to be a paraphrase. From the remainder of the EnglishJapanese expression pairs, we obtained 85,858 pairs of paraphrases. Paraphrases using distributional similarity We automatically collected paraphrases using distributional similarity in the following procedure. 1.

2. 3. 4. 5. 6.

Analyze the dependency structures of all sentences in a total of 56 years of Japanese newspapers from the Mainichi, Yomiuri, and Nikkei newspaper databases using the Japanese parser CaboCha3. Extract noun-verb pairs that have dependency relations from the dependency trees obtained in Step 1. Count the frequencies of each noun-verb pair. Collect verbs and their frequencies for each noun, creating indices for each noun. Calculate the similarities between two indices of nouns using the SMART similarity measure [11]. Obtain a list of synonymous nouns4.

In Step 2, we also extracted noun-phrase-verb pairs, instead of noun-verb pairs, and obtained a list of synonymous noun phrases using the same Steps 3 to 6. As well as collecting verbs for each noun in Step 4, we similarly collected nouns for each verb, and obtained a list of synonymous verbs. WordNet dictionary WordNet [1] is a most widely used lexical resource in natural language processing. This database links nouns, verbs, adjectives, and adverbs to sets of synonyms (synsets) that are in turn linked through semantic relations that determine word definitions. We considered a set of words linked in the same synset as paraphrases and used them for text evaluation. NTT Goi-Taikei dictionary NTT Goi-Taikei is a Japanese thesaurus produced by NTT Communication Science Laboratories. In this dictionary, a list of synonymous expressions of nouns, adjectives, and verbs with different Japanese characters is included. 2

http://www.fjoch.com/GIZA++.html http://chasen.org/~taku/software/cabocha/ 4 For each noun, we extracted the top 20 similar nouns, and used them for text evaluation. 3

Automatic Evaluation of Texts by Using Paraphrases

561

The four paraphrase methods are summarized in Table 1. Table 2 shows the relations between the four paraphrase methods and four categories of paraphrases in Section 3.2. Table 1. Paraphrases for text evaluation Paraphrase method SMT DS WordNet NTT

Target POS All Noun, Noun Phrase, Verb Noun, Verb Noun, Verb, Adjective

Automatic / Manual automatic automatic manual manual

Table 2. Classification of the paraphrases and necessary correspondence Category

Paraphrase level

Number of cases

A

character

B

word

C

phrase

D

clause

E

others

20.1% (64/318) 24.5% (78/318) 11.9% (38/318) 12.3% (39/318) 31.2% (99/318)

SMT

DS

Word NTT Net

△ △ △ ◎ ○ ◎ ◎ ○ △ △ △

Required techniques

character-level paraphrases word-level paraphrases phrase-level paraphrases clause-level paraphrases paraphrases based on context analysis

5 Experiments To investigate the effectiveness of our method, we conducted several experiments. 5.1 Experimental Settings Correct data sets In addition to the 300 abstract-type summaries created from 30 editorials by 10 human subjects, we prepared another 300 extract-type summaries, which were produced by extracting important parts of the original texts. Three human subjects assigned evaluation scores manually on a one-to-four scale to each of these 600 summaries. Alternatives We conducted examinations using 15 proposed methods and a baseline method ROUGE-1, shown in Table 3.5 The proposed methods used different combinations of the four kinds of paraphrases: SMT, DS, NTT, and Word. 5

We employed ROUGE-1 as a baseline method, because ROGUE-1 obtained the best performance among a series of ROUGE family in this dataset [9].

562

K. Hirahara et al. Table 3. List of 15 proposed methods and a baseline method Combination of Paraphrases

SMT(S)

Our method

○

Baseline method

S D W N SD SW SN DW DN WN SDW SDN SWN DWN SDWN ROUGE-1

○ ○ ○ ○ ○ ○ ○

DS (D)

○ ○ ○ ○ ○ ○ ○ ○

WordNet (W)

○ ○ ○ ○ ○ ○ ○ ○

NTT (N)

○ ○ ○ ○ ○ ○ ○ ○

Experimental method We used the top extract-type summary and the top abstract-type summary in each topic as reference summaries. Then we conducted the following experiments for each topic. z z z z

EX-1: Evaluate nine extract-type summaries using the top extract-type summary as reference summaries. EX-2: Evaluate nine extract-type summaries using the top abstract-type summary as reference summaries. EX-3: Evaluate nine abstract-type summaries using the top extract-type summary as reference summaries. EX-4: Evaluate nine abstract-type summaries using the top abstract-type summary as reference summaries.

In each experiment, evaluation scores were calculated by taking the reference summary. We then ranked summaries by our methods and ROUGE-1, and compared them with a manual ranking by Spearman rank-order correlation coefficients. 5.2 Experimental Results We show the experimental results in Tables 4 and 5, which show the Spearman rankorder correlation coefficients for the 16 methods using two extract-type reference summaries and two abstract-type reference summaries, respectively.

Automatic Evaluation of Texts by Using Paraphrases

563

As can be seen from Table 4, all of our methods could evaluate abstract-type summaries more accurately than ROUGE-1. Of our 16 methods, the combination of "SDW" gave the best performance in abstract-type summaries evaluations. This combination could improve ROUGE-1 by 0.047 (15%). Table 4. Evaluation results using an extract-type reference summary

Our Method

Baseline

Combination of Extract Abstract Paraphrases (EX-1) (EX-2) S (SMT) 0.296 0.332 D (DS) 0.368 0.328 W (WordNet) 0.357 0.344 N (NTT) 0.358 0.329 0.319 0.355 SD 0.331 0.353 SW SN 0.316 0.339 DW 0.365 0.346 DN 0.331 0.370 WN 0.358 0.348 0.333 SDW 0.357 0.332 0.337 SDN 0.321 0.342 SWN DWN 0.368 0.348 SDWN 0.314 0.332 ROUGE-1 0.358 0.310

Table 5. Evaluation results using an abstract-type reference summary

Our method

Baseline

Combination of Extract Abstract Paraphrases (EX-3) (EX-4) S (SMT) 0.346 0.386 D (DS) 0.322 0.367 W (WordNet) 0.327 0.397 N (NTT) 0.323 0.400 SD 0.377 0.401 SW 0.330 0.427 SN 0.394 0.426 0.302 0.364 DW 0.368 DN 0.317 WN 0.321 0.402 SDW 0.329 0.436 SDN 0.394 0.419 SWN 0.346 0.432 DWN 0.303 0.371 SDWN 0.358 0.426 ROUGE-1 0.313 0.389

564

K. Hirahara et al.

In Table 5, the combination of SN was better than ROUGE-1 by 0.113 (36%) when evaluating extract-type summaries, while the combination of "SDW" was better by 0.047 (12%) when evaluating abstract-type summaries. 5.3 Discussion Effectiveness of paraphrases in text evaluation More than half of our methods performed worse than ROUGE-1 in the experiment Ex-1, which indicates that paraphrases were not effective for the evaluation of extracttype summaries using extract-type reference summaries. On the other hand, most of our methods improved ROUGE-1 in the experiments Ex-2, 3, and 4. We considered that they were valid experimental results, because paraphrases are generally used in abstract-type summaries. Effectiveness of SMT In the experiment Ex-4, the combinations of "DW" and "SDW" obtained 0.364 and 0.436 of Spearman rank order correlation coefficients, respectively. This indicates that the SMT-based paraphrases contributed to improve a score from 0.364 to 0.436. We can also confirm the effectiveness of SMT-based paraphrases from "W" and "SW". To confirm the effectiveness of the SMT-based paraphrases more precisely, we calculated Spearman rank-order correlation coefficients for each topic and counted the number of topics that "SDW" and "SW" were superior to "DW" and "W", respectively. The result is shown in Table 6. As can be seen from Table 6, the SMT-based paraphrases are useful to improve the combination of "DW", because the number of topics that the combination of "SDW" improved "DW" was much larger than the opposite cases. On the other hand, the combination of "SW" impaired "W" in 13 topics. In the SMT-based paraphrases, there were cases that a term X can be paraphrased into Y, but Y cannot be paraphrased into X. A pair of " " (adjudication) and " " (unsuccessful litigation) is one of such paraphrases. Recently, detecting such paraphrases has studied in the field of textual entailment recognition. In future, the SMT-based paraphrases may be improved using techniques in the field.

判決

訴

敗

Table 6. The number of topics that the combinations of SDW and SW improved DW and W Improve "SDW" vs. "DW" "W" vs. "SW"

Same 17 (0.56 ) 11 (0.37)

Impair 4 (0.13) 6 (0.20)

9 (0.30) 13 (0.43)

Effectiveness of distributional similarity Distributional similarity did not contribute to improve ROUGE-1. For example, the combinations of "SW" and "SDW" in Ex-4 obtained 0.427 and 0.436 of Spearman rank order correlation coefficients, respectively. In this case, the distributional-similaritybased paraphrases contributed to improve ROUGE-1 by only 0.009. In another case, "SDWN" impaired "SWN" from 0.432 to 0.426. The distributional similarity collected

Automatic Evaluation of Texts by Using Paraphrases

イギリス

565

フラ

more related terms rather than synonyms. A pair of " " (England) and " " (France) is one of such paraphrases. As the method expresses the senses of each noun or noun phrase with a set of verbs having dependency relations in texts, it tends to collect terms that have the same properties.

ンス

6 Conclusions We explored the use of paraphrases for the refinement of traditional automatic methods for text evaluation. We analyzed 318 paraphrases in texts to be evaluated, and classified them into five categories. Then we examined several paraphrase methods that covered four of those categories (about 70% of the 318 paraphrases). To confirm the effectiveness of our method, we conducted some experiments using the data from the Text Summarization Challenge 2. We found that the use of the combination of three kinds of paraphrases (SMT, distributional similarity, and WordNet) improved the traditional evaluation method ROUGE-1 from 0364 to 0.436.

References 1. Bond, F., Isahara, H., Uchimoto, K., Kuribayashi, T., Kanzaki, K.: Extending the Japanese WordNet. In: Proc. 15th Annual Meeting of the Association for Natural Language Processing, pp. 80–83 (2009) 2. Fukushima, T., Okumura, M., Nanba, H.: Text Summarization Challenge 2 / Text Summarization Evaluation at NTCIR Workshop 3. In: Working Notes of the 3rd NTCIR Workshop Meeting, PART V, pp. 1–7 (2002) 3. Kanayama, H.: Paraphrasing Rules for Automatic Evaluation of Translation into Japanese. In: Proc. First International Workshop on Paraphrasing, pp. 88–93 (2003) 4. Kauchak, D., Barzilay, R.: Paraphrasing for Automatic Evaluation. In: Proc. 2006 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 455–462 (2006) 5. Lee, L.: Measures of Distributional Similarity. In: Proc. 37th Annual Meeting of the Association for Computational Linguistics, pp. 25–32 (1999) 6. Lin, C.Y., Hovy, E.: Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics. In: Proc. 4th Meeting of the North American Chapter of the Association for Computational Linguistics and Human Language Technology, pp. 150–157 (2003) 7. Lin, C.Y.: ROUGE A Package for Automatic Evaluation of Summaries. In: Proc. ACL 2004 Workshop Text Summarization Branches Out, pp. 74–81 (2004) 8. Lin, D.: Automatic Retrieval and Clustering of Similar Words. In: Proc. 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics, pp. 768–774 (1998) 9. Nanba, H., Okumura, M.: An Automatic Method for Summary Evaluation Using Multiple Evaluation Results by a Manual Method. In: Proc. COLING/ACL 2006 Main Conference Poster Sessions, pp. 603–610 (2006) 10. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a Method for Automatic Evaluation of MachineTranslation. In: Proc. 40th Annual Meeting ofthe Association for Computational Linguistics, pp. 311–318 (2002)

566

K. Hirahara et al.

11. Salton, G.: The SMART Retrieval System - Experiments in Automatic Document Processing. Prentice-Hall, Inc., Upper Saddle River (1971) 12. Utiyama, M., Isahara, H.: Reliable Measures for Aligning Japanese-English News Articles and Sentences. In: Proc. 41st Annual Meeting of the Association for Computational Linguistics, pp. 72–79 (2003) 13. Yves, L., Etienne, D.: Automatic Generation of Paraphrases to be used as Translation References in Objective Evaluation Measures of Machine Translation. In: Proc. Third International Workshop on Paraphrasing (2005) 14. Zhou, L., Lin, C. Y., Munteanu, D. S. and Hovy, E.: ParaEval Using Paraphrases to Evaluate Summaries Automatically. In: Proc. 2006 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 447– 454 (2006)

Packing It All Up in Search for a Language Independent MT Quality Measure Tool – Part Two Kimmo Kettunen Kymenlaakso University of Applied Sciences Paraatikenttä 7, FIN-045 100 Kouvola, Finland [email protected]

Abstract. This study describes first usage of a particular implementation of Normalized Compression Distance (NCD) as a machine translation quality evaluation tool. NCD has been introduced and tested for clustering and classification of different types of data and found a reliable and general tool. As far as we know NCD in its Complearn implementation has not been evaluated as a MT quality tool yet, and we wish to show that it can also be used for this purpose. We show that NCD scores given for MT outputs in different languages correlate highly with scores of a state-of-the-art MT evaluation metrics, METEOR 0.6. Our experiments are based on translations between one source and three target languages with a smallish sample that has available reference translations, UN’s Universal Declaration of Human Rights. Secondly we shall also briefly describe and discuss results of a larger scale evaluation of NCD as an MT metric with WMT08 Shared Task Evaluation Data. These evaluations confirm further that NCD is a noteworthy MT metric both in itself and also enriched with basic language tools, stemming and Wordnet. Keywords: MT metrics, Normalized Compression Distance, machine translation, evaluation of machine translation systems.

1 Introduction Automatic evaluation of machine translation program output has been developed and used for about a decade. As a result of work done we have now available several MT evaluation systems or metrics, such as BLEU [1], NIST [2], METEOR [3], IQMT [4] and a few dozen of others not mentioned here. Most of the evaluation metrics are based on similar features, e.g. use of string level comparison of texts, recall and precision of translations, different penalty scores etc. Many of the programs have a quite high degree of correlation with human judgements of translations and they have been a valuable tool in making especially the statistical MT systems better. It is also well known that all the present MT evaluation programs have limitations. They may, e.g., be language dependent, i.e. they need to be tuned for specific language pairs to be able to perform or use language specific tools (stemmers, Wordnets). Also more severe concerns about MT metrics have been stated. CallisonBurch, Osborne and Koehn [5] showed in a detailed analysis that BLEU's coarse model of allowable variation in word order of translations “can mean that an Z. Vetulani (Ed.): LTC 2009, LNAI 6562, pp. 567–576, 2011. © Springer-Verlag Berlin Heidelberg 2011

568

K. Kettunen

improved BLEU score is not sufficient to reflect a genuine improvement in translation quality”. MT metrics' correlation with human judgements of translations has also been disputed [6]. We show in this paper that an alternative language independent measure for MT evaluation can be obtained from a general classification and clustering tool called Normalized Compression Distance, NCD [7, 8, 9]. In our first experiments we use a specific implementation of NCD, Complearn, which is available from http://www. complearn.org/download.html. We first show results of our proof-of-concept style empirical testing of NCD with a small corpus. After this initial evaluation, we have done a large scale evaluation of NCD as an MT metric with the WMT08 Shared Task Evaluation Data available at http://www.statmt.org/wmt08/results.html [10] as a joint project with Adaptive Informatics Research Centre of the Laboratory of Information and Computer Science at the current Aalto University (formerly Helsinki University of Technology). The first results of the evaluation were published in Väyrynen et al. [11], being followed by results of Dobrinkat et al. [12, 13]. As these results have been published elsewhere, we shall not repeat the results in full, but mainly discuss the implications of them in the discussion part of the article.

2 Research Setting We evaluated En {De, Es, Fr} translations of UN’s Universal Declaration of Human Rights (UDHR) (http://www.un.org/Overview/rights.html) of 10-12 MT programs that were either freely available or could be used with an evaluation license. The used MT programs were Promt, Google Translate Beta, Babelfish, Translate It!, LEC Translate2Go, SDL Enterprise Translation Server, Systran, InterTran, Translated, Hypertrans, MZ-Win Translator, Dictionary.com, and Translendium. MZ-Win and Translate It! translated from English to German only, all others to all the three target languages. Translations were performed in late March 2009. If the web service of the MT system had limitations in the number of words to be translated, the text was split to smaller chunks, e.g. 5-10 articles. UN’s Universal Declaration of Human Rights has 30 numbered articles and its length in English original is 1451 words and 60 sentences (7.3 Kb without spaces). Lengths of the articles vary quite a lot, some having only one sentence and others several sentences or even paragraphs. The used text is quite short and homogeneous textually, but can be considered to be long and representative enough for our preliminary experiments. Our research settings with the WMT08 data are explained in publications [11, 12, 13], and they are not repeated here, only the main results will be shown in the results section. To get a baseline of the translation quality of the MT programs we evaluated the translation results of the MT systems with a state-of-the-art machine translation evaluation metrics, METEOR 0.6 [14, 15]. METEOR is based on a BLEU like evaluation idea: output of the MT program is compared to a given reference translation, which is usually a human translation. METEOR’s most significant difference to BLEU like systems is, that it emphasizes more recall than precision of translations [15]. The evaluation metric was run with exact match, where translations are compared to reference translation as such. Basically “METEOR evaluates a translation by computing a score based on explicit word-to-word matches between the

Packing It All Up in Search for a Language Independent MT Quality Measure Tool

569

translation and a given reference translation”. When “given a pair of strings to be compared, METEOR creates a word alignment between the two strings. An alignment is a mapping between words, such that every word in each string maps to most one word in the other string. This alignment is incrementally produced by a sequence of word-mapping modules. The ‘exact’ module maps two words if they are exactly the same.” METEOR has been shown to outperform commonly used metrics BLEU and NIST in terms of correlations with human judgements of translation quality [3]. Our suggested new MT quality measurement tool, Normalized Compression Distance, is based on the work of Rudi Cilibrasi, Paul Vitányi and others [7, 8, 9]. The method is the outcome of mathematical developments, that are based on the notion of Kolmogorov complexity. Informally, for any sequence of symbols, the Kolmogorov complexity of the sequence is the length of the shortest algorithm that will exactly generate the sequence (and then stop). In other words, the more predictable the sequence, the shorter the algorithm needed is and thus the Kolmogorov complexity of the sequence is also lower [9, 16]. Kolmogorov complexity itself is uncomputable, but file compression programs can be used to approximate the Kolmogorov complexity of a given file. A more complex string (in the sense of Kolmogorov complexity) will be less compressible. From this approach grew first normalized information distance, NID [9], and as its approximation NCD using a real compressor. NCD’s basic formula for counting the distance (and thus similarity) between two files is as follows [17]:

Here C is the compressor, C(x) denotes the length of the compressed version of a string x, and C(xy) is a compressed concatenation of the pair (x,y). As a result, NCD gives a score between [0,1] for the strings (files) that are compared. Smaller numbers represent more similar files. In plain words, NCD uses the lengths of the compressed hypothesis and reference strings, computing a ratio of the compressed length of the concatenated reference and hypothesis (minus the length of the shorter compressed reference or hypothesis sequence) to the length of the longer compressed reference or hypothesis sequence. The basic idea is that if the two sequences A and B are more similar, then B will compress with a smaller 'description' when combined with A than it would when compressed separately. Since compression 'descriptions' for text are typically based on frequencies of character sequences, the compression lengths can serve as a similarity measure for the words in the hypothesis and reference(s). Parker [18] has earlier introduced an MT metric named BADGER that utilizes also NCD as one part of the metric. BADGER does not use Complearn’s NCD package, but implements NCD using the Burrows Wheeler Transformation as compressor, which enables the system to take into account more sentence context. BADGER uses also some language independent word normalization methods, such as Holographic Reduced representation, which utilizes binary vectors and relative distance counting with cosine similarity. Thus BADGER is more advanced than a bare NCD metric. Parker benchmarked BADGER against METEOR and word error rate metrics (WER). The correlation of BADGER results to those of METEOR were low and correlations

570

K. Kettunen

to WER high. The used test set was Arabic to English translations. Author considers the results preliminary and wishes to do more testing with the software.

3 Results 3.1 Original Results with Complearn Translations of MT systems were compared to one human reference translation with both METEOR 0.6 and Complearn NCD. In our case the reference translations were the French, German and Spanish translations of the Universal Declaration of Human rights from UN’s web page (http://www.ohchr.org/EN/UDHR/Pages/SearchByLang.aspx). Table 1 shows a short example of results of METEOR’s evaluations for three of the English German MT outputs in their raw form. The compared sequence with METEOR was one article, and the overall system score in the table gives a unified score for all the 30 sequences. Table 1. Example results of METEOR translation evaluation for En

Overall system score Precision Recall Fmean Penalty

De translations

Google 0.66

Babelfish 0.21

Promt 0.25

0.82 0.82 0.82 0.20

0.54 0.57 0.56 0.64

0.56 0.60 0.59 0.58

The meanings of the METEOR scores in Table 1 are as follows: 1 2

3

4

5

Overall system score gives a combined figure for the result. It is computed as follows [19]: Score = Fmean * (1- Penalty). (Unigram) Precision = unigram precision is computed as the ratio of the number of unigrams in the system translation that are mapped (to unigrams in the reference translation) to the total number of unigrams in the system translation. (Unigram) Recall = unigram recall is computed as the ratio of the number of unigrams in the system translation that are mapped (to unigrams in the reference translation) to the total number of unigrams in the reference translation. Fmean: precision and recall are combined via harmonic mean that places most of the weight on recall. The present formulation of Fmean is stated in as follows [19]: Fmean = P * R/ α * P + (1- α) * R. Penalty: This figure takes into account the extent to which the matched unigrams in the two strings are in the same word order.

Table 2 lists the scores given by METEOR 0.6 and NCD for all translations in German. It should be noted, that the scale of METEOR and NCD are reverse: bigger score in METEOR means better translation quality whereas smaller score in NCD means greater similarity and thus better translation.

Packing It All Up in Search for a Language Independent MT Quality Measure Tool Table 2. Scores for En

571

De MT translations compared with human reference translations

En De Babelfish Dictionary.com SDL Enterprise Translation Service Google Translate Beta Hypertrans InterTran LEC Translate2Go MZ-Win Promt Systran Translate It! Translendum

METEOR 0,21 0,21 0,26

NCD 0,75 0,74 0,74

0,66 0,20 0,11 0,27 0,22 0,25 0,23 0,26 0,24

0,40 0,77 0,86 0,74 0,76 0,73 0,74 0,78 0,75

Average Standard deviation Correlation co-efficient NCD vs. METEOR

0,26 0,11

0,73 0,13 -0,98

Google’s German translation was given the best score (0.66) by METEOR and translation of Intertran the worst (0.11). All the others were given a score between 0.20 and 0.27. NCD's scores for translations follow the same pattern. The last row of Table 2 shows that scores for both analyses for all translations correlate highly. The correlation seems negative, but if the scores are transformed to equal scale (this can be done by subtracting NCD score from 1, 1-NCD), the correlation is positive. Furthermore we see, that both measures indicate clearly the worst (InterTran) and best (Google) MT programs. The middle area is quite even, and there seems to be no big differences in the translation quality of other systems. Tables 3 and 4 show results of Spanish and French translation evaluations. For Spanish and French translations both METEOR and NCD were again able to distinguish the best and worst translations. Also scores for En Es and En Fr translations correlated highly: for Spanish translations the correlation was 0.995 and for French 0.99. We also have other independent data that strengthens our case. Kettunen [20, 21] shows that METEOR, NIST and BLEU scores of MT output all correlate well with mean average precisions of Cross-language information retrieval runs, thus confirming the bond between translation quality and CLIR result achieved by others, e.g. Kishida [22] and Zhu and Wang [23]. The MAP figures for the CLIR runs of Kettunen [20] were now tested with NCD scores, and a high correlation of 0.91 was found between NCD scores for translations and achieved MAP results of 56 German title and topic translations from English to German. Thus, even though our data samples are small, two

572

K. Kettunen

different evaluation settings show a high correlation between METEOR and NCD score sets both for three target language translations and one target language translation score set and MAP scores gained in CLIR evaluation of the translated queries. Table 3. Scores for En

Es MT translations compared with human reference translations

En Es Babelfish Dictionary.com SDL Enterprise Translation Service Google Translate Beta Hypertrans InterTran LEC Translate2Go MZ-Win Promt Systran Translate It! Translendum Average Standard deviation Correlation co-efficient NCD vs. METEOR Table 4. Scores for En

METEOR 0,26 0,26 0,27

NCD 0,72 0,71 0,72

0,54 0,22 0,19 0,26 N/A 0,28 0,27 N/A 0,26 0,28 0,09 -0,995

0,39 0,77 0,82 0,73 N/A 0,69 0,70 N/A 0,70 0,70 0,11

Fr MT translations compared with human reference translations

En Fr Babelfish Dictionary.com SDL Enterprise Translation Service Google Translate Beta Hypertrans InterTran LEC Translate2Go MZ-Win Promt Systran Translate It! Translendum Average Standard deviation Correlation co-efficient NCD vs. METEOR

METEOR 0,17 0,17 0,11

NCD 0,71 0,69 0,74

0,45 0,12 0,06 0,14 N/A 0,13 0,18 N/A 0,15 0,17 0,10 -0,99

0,38 0,76 0,85 0,72 N/A 0,72 0,69 N/A 0,71 0,70 0,12

Packing It All Up in Search for a Language Independent MT Quality Measure Tool

573

3.2 Results with WMT08 and WMT10 Data After the promising initial results we continued to evaluate NCD with more elaborated data that was intended for MT evaluation purposes, the WMT08 data [10]. The main benefits of this data are that it has enough sentence and language pairs and translations of MT systems are also evaluated by humans. Human evaluations of MT system output are nowadays considered very important, and if an MT metric’s evaluation of translations corresponds highly with human evaluations, this gives it more credibility than a plain MT metric score only. Our first results were published in Väyrynen et al. [11]. We tried different compressors (zlib, bzip2, gzip and ppmz) and offered a working solution to overcome the problems of the sliding window size in some of the compressors reported by Cebrian et al. [24]: when the translated sentences and their reference translations were interleaved, so that the corresponding sentences were adjacent, the problem of too large text chunks was overcome. We did not find any considerable difference between compressors when the translations were interleaved. Overall NCD performed roughly at the same level as the state-of-the art metrics, BLEU and METEOR in system level analysis and outperformed them slightly in sentence-level analysis. In Dobrinkat el al. [12, 13] we continued evaluation of NCD as an MT metric. In addition to plain MT-NCD we introduced MT-mNCD, a version that was augmented with linguistic normalization tools, Snowball stemmers and Wordnet. MT-mNCD works in an analogical fashion with M-BLEU and M-TER, which use the flexible word matching modules from METEOR for finding of relaxed word-to-word alignments. The best MT-mNCD (with PPMZ compressor, original casing) improved the best MT-NCD (PPMZ) method by 15% in the RANK category. In the CONST category the best MT-mNCD (bz2, lowercasing) improved the best MT-NCD (bz2) by 3.7%. For the total average, the best MT-mNCD (PPMZ, original casing) improved the best MT-NCD (bz2, lowercasing) by 7.2%. Correlation results for the RANK category by target language showed that MT-mNCD clearly outperformed MT-NCD for English. Correlations for other languages showed mixed results and on average, MT-mNCD gave lower correlations than MT-NCD [13]. The new development introduced to evaluation of MT-NCD was usage of multiple references, which is a common practice in MT evaluation. Some new system parameters were also tried out. These included different character casings, segment replication and block size. When segments were replicated two times, a small improvement in correlations between MT-NCD and human judgments was found, when results of all the languages were averaged [13]. We participated in the MetricsMATR10 task with both MT-NCD and MT-mNCD [12]. Results of the evaluation task with the WMT10 data have been published in Callison-Burch et al. [25]. At system-level evaluation both MT-NCD and MT-mNCD performed very well in comparisons with the human judgment for translation both into and out of English. With translation into English MT-NCD was on a shared third position with several other systems when scores were averaged over all the four language pairs, and MT-mNCD was the fourth (out of 26 systems). With translations out of English MT-mNCD performed slightly better than MT-NCD, and the positions were now five and six. On segment-level correlation analysis both systems performed slightly worse, being in the mid-range of the systems when translations where into

574

K. Kettunen

English and below mid-range with translations out of English. Overall both MT-NCD and MT-mNCD showed even performance in all the categories of the WMT10 evaluations task.

4 Discussion and Conclusions The first aim of this paper was to introduce usage of a general classifier and clustering program, Normalized Compression Distance, and its specific Complearn implementation, as an MT evaluation tool. For performance testing we used three target language translations of the Universal Declaration of Human Rights with 10-12 different available MT systems for each target language. We first evaluated the quality of the translations with a language specific state-of-the-art MT metrics, METEOR 0.6, by using available reference translations as comparison. After that we compared the translations to references with NCD. The scores given by METEOR and NCD correlated very highly in all three target languages (0.99-0.995). Based on the preliminary results of our study Complearn NCD seems to be as good as METEOR 0.6 in the sense that it can pick the outliers of MT systems, the best and the worst translators, very well. Google's outstanding translation performance with all the language pairs might have a very simple explanation: the parallel texts of UDHR are most obviously in the training corpus of the Google MT system, and thus it is able to outperform all the other systems with such a margin. Those systems that produce midrange output are not very well separated by METEOR, and they are also given quite similar scores by NCD, as was seen in Tables 2-4. This, in turn, is a specific problem of MT metrics: we do not know what the real meaning of the differences in scores is, and thus we cannot really say, if a small score difference really matters [6, 26]. Our WMT08 and WMT10 results show, that bare NCD without linguistic tools performs on an equal level with most of the state-of-the-art MT metrics. MT-mNCD with linguistic add-ons performed in some tests slightly better than basic NCD, but as these add-ons also change the basic language-independent nature of NCD, this improved performance can be considered a mixed blessing. It is also obvious that if language tools are used, there could be better choices of them available. Basic Snowball stemmers do not obviously improve performance much, and only Wordnet seems to work better. But as we only tested Wordnet of English, these results should also be evaluated with Wordnets or other lexical resources for other languages. In conclusion we can now state, that NCD is also a noteworthy method as an MT metric. We do not suggest that NCD overcomes all the difficulties related to automated MT metrics, but it offers clear benefits. The special advantage of NCD is that it is an information theoretic general measure of similarity. It is feature- and parameter-free. As it works with character strings instead of word n-grams, it is also language independent. It has already been shown to work in many real-world applications that range from bioinformatics to music clustering [27]. This gives it an advantage with respect to common MT evaluation systems that might need parameter setting and are usually n-gram based, and sometimes language dependent. One of the problems of n-gram based metrics is the assumption “that a good translation will be similar to other good translations” [28]. Everything an MT metric actually does is just

Packing It All Up in Search for a Language Independent MT Quality Measure Tool

575

to compare the sameness of the n-gram distribution of the translations to reference translation(s), and the result is thus basically a measure of document similarity. This, as such, may not tell much about the translation’s goodness itself [28]. Whether a low score given by NCD to translation really indicates the quality of translation, is also left somehow open. In this respect NCD is similar to common measures of MT quality. Anyhow our WMT08 and WMT10 evaluations showed convincingly, that results of NCD evaluation correlate well with human evaluations of translations, and in this respect NCD can also be considered as a state-of-the-art MT metrics.

References [1] Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a Method for Automatic Evaluation of Machine Translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), pp. 311–318 (2002) [2] Doddington, G.: Automatic Evaluation of Machine Translation Quality Using N-Gram Co-Occurrence Statistics. In: Proceedings of the Second International Conference on Human Language Technology Research, pp. 138-145 (2002) [3] Lavie, A., Agarwal, A.: METEOR: An automatic Metric for MT Evaluation with High Levels of Correlation with Human judgements. In: Proceedings of the Second Workshop on Statistical Machine Translation, Prague, pp. 228–231 (June 2007) [4] Giménez, J., Amigó, E.: IQMT: A Framework for Automatic Machine Translation Evaluation. In: Proceedings of the 5th LREC (2006) [5] Callison-Burch, C., Osborne, M., Koehn, P.: Re-evaluating the Role of BLEU in Machine Translation Research. In: EACL 2006, pp. 249–256 (2006) [6] Turian, J.P., Shen, L., Melamed, D.: Evaluation of Machine Translation and its Evaluation. In: MT Summit IX, New Orleans, USA, September 23-27, pp. 386–393 (2003) [7] Cilibrasi, R., Vitányi, P.M.B.: Clustering by Compression. IEEE Transactions on Information Theory 51, 1523–1545 (2005) [8] Cilibrasi, R., Vitányi, P.M.B.: The Google Similarity Distance. IEEE Transactions on Knowledge and Data Engineering 19, 370–383 (2007) [9] Li, M., Chen, X., Li, X., Ma, B., Vitányi, P.: The Similarity Metric. IEEE Transactions on Information Theory 50, 3250–3264 (2004) [10] Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., Schroeder, J.: Further MetaEvaluation of Machine Translation. In: Proceedings of the Third Workshop on Statistical Machine Translation, pp. 70–106 (2008) [11] Väyrynen, J., Tapiovaara, T., Kettunen, K., Dobrinkat, M.: Normalized Compression Distance as an Automatic MT Evaluation Metric. In: Twenty-Five Years on Machine Translation, Cranfield, U.K (2009) [12] Dobrinkat, M., Tapiovaara, T., Väyrynen, J., Kettunen, K.: Normalized Compression Distance Based Measures for MetricsMATR 2010. In: ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and Metrics MATR. Proceedings of the Workshop, Uppsala University, Uppsala, Sweden, July 15-16, pp. 343–348 (2010) [13] Dobrinkat, M., Tapiovaara, T., Väyrynen, J., Kettunen, K.: Evaluating Machine translations Using mNCD. In: ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics: Proceedings of the Conference Short Papers, Uppsala, Sweden, July 11-16, pp. 80–85 (2010)

576

K. Kettunen

[14] Lavie, A., Agarwal, A.: The METEOR Automatic Machine Translation Evaluation System, http://www.cs.cmu.edu/~alavie/METEOR/ [15] Banerjee, S., Lavie, A.: METEOR: Automatic Metric for MT Evaluation with Improved Correlation with Human Judgements. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, pp. 65–72 (2005) [16] Li, M., Vitányi, P.M.B.: An Introduction to Kolmogorov Complexity and its Applications, 2nd edn. Springer, Heidelberg (1997) [17] What is NCD?, http://www.complearn.org/ncd.html [18] Parker, S.: BADGER: a New Machine Translation Metric (2008), http://www.amtaweb.org/papers/badger_babblequest_matr08.pdf [19] Lavie, A., Sagae, K., Jayaraman, S.: The significance of recall in automatic metrics for MT evaluation. In: Frederking, R.E., Taylor, K.B. (eds.) AMTA 2004. LNCS (LNAI), vol. 3265, pp. 134–143. Springer, Heidelberg (2004) [20] Kettunen, K.: Choosing the best MT programs for CLIR purposes – can MT metrics be helpful? In: Boughanem, M., Berrut, C., Mothe, J., Soule-Dupuy, C. (eds.) ECIR 2009. LNCS, vol. 5478, pp. 706–712. Springer, Heidelberg (2009) [21] Kettunen, K.: Facing The Machine Translation Babel in CLIR - Can MT Metrics Help in Choosing CLIR Resources? In: Klopotek, M.A., Przepiórkowski, A., Wierzchon, S.T., Trojanowski, K. (eds.) Recent Advances in Intelligent Information Systems, IIS 2009, pp. 103–116 (2009) [22] Kishida, K.: Prediction of Performance of Cross-language Information Retrieval System Using Automatic Evaluation of Translation. Library & Information Science Research 30, 138–144 (2008) [23] Zhu, J., Wang, H.: The Effect of Translation Quality in MT-Based Cross-Language Information Retrieval. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th annual Meeting of the ACL, pp. 593–600 (2006) [24] Cebrián, M., Alfonseca, M., Ortega, M.: Common Pitfalls Using The Normalized Compression Distance: What To Watch Out For In A Compressor. Communications in Information and Systems 5, 367–384 (2005) [25] Callison-Burch, C., Koehn, P., Monz, C., Peterson, K., Przybocki, M., Zaidan, O.F.: Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In: Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, Uppsala, Sweden, July 15-16, pp. 17–53 (2010), Updated version: http://statmt.org/wmt10/pdf/wmt10-overview.pdf [26] Lopez, A.: Statistical Machine Translation. ACM Computing Surveys 40, 8–49 (2008) [27] Vitányi, P.M.B., Balbach, F.J., Cilibrasi, R., Ming, L.: Normalized Information Distance. In: Emmert-Streib, F., Dehmer, M. (eds.) Information Theory and Statistical Learning, pp. 39–71. Springer, Heidelberg (2009) [28] Culy, C., Riehemann, S.Z.: The Limits of N-gram Translation Metrics. In: MT Summit IX, New Orleans, USA, September 23-27, pp. 71–78 (2003)

Author Index

Gopal, Madhav 371 Gosme, Julien 127 Grali´ nski, Filip 361 Gr˚ uber, Martin 280

Abate, Solomon Teferra 82 Adda, Gilles 70 Adda-Decker, Martine 70 Alegria, I˜ naki 327 Alrahabi, Motasem 478 Aranzabe, Maxux 327 Arregi, Xabier 327 Artola, Xabier 327

Hajnicz, El˙zbieta 423 Hirahara, Kazuho 556 Hor´ ak, Aleˇs 161 Hulden, Mans 151

Baghai-Ravary, Ladan 1 Balahur, Alexandra 203 Ba´ nski, Piotr 400 Barros, Maria 12 B´edaride, Paul 257 Betkowska Cavalcante, Agnieszka Blancafort, Helena 339 Boldrini, Ester 203 Bristot, Antonella 291 Butenas, Linas 412 Cabrio, Elena 268 Carson-Berndsen, Julie 47, 94 Chalamandaris, Aimilios 35 Coleman, John 1 Culy, Chris 351 ´ de La Clergerie, Eric 522 Delmonte, Rodolfo 291 de Loupy, Claude 339 Denis, Pascal 522 Descl´es, Jean-Pierre 478 Descl´es, Julien 478 D´ıaz de Ilarraza, Arantza 327 Dipper, Stefanie 456 El-Haj, Mahmoud 490 Esuli, Andrea 500 Fox, Chris 490 Fukumoto, Fumiyo 115, 192 Furui, Sadaoki 24 Gagnon, Michel 245 Gardent, Claire 257 Ghassem-Sani, Gholamreza

213

Ion, Radu 435 Ircing, Pavel 280

24

Jakub´ıˇcek, Miloˇs 161 Jasionyte, Erika 412 Jassem, Krzysztof 361 Jha, Girish Nath 371 Juozapavicius, Algimantas Kabadjov, Mijail 203 Kane, Mark 47 Karabetsos, Sotiris 35 Kedrova, Galina 380 Kettunen, Kimmo 567 Khayyamian, Mahdy 213 Kochanski, Greg 1 Kondo, Tomoki 512 Kov´ aˇr, Vojtˇech 161 Kruschwitz, Udo 490 Kubis, Marek 444 Kurc, Roman 361 Lardilleux, Adrien 127 Leg´ at, Milan 280 Lepage, Yves 127 Lipski, Jaroslaw 172 Lyding, Verena 351 Magnini, Bernardo 268 Marciniak, Jacek 303 Marciniak, Malgorzata 139 Mauclair, Julie 47 Mayor, Aingeru 327 Menzel, Wolfgang 82 Mignot, Victor 522

412

578

Author Index

Milkowski, Marcin 172 Mirroshandel, Seyed Abolghasem Mishra, Diwakar 371 Mlodzki, Rafal 224 M¨ obius, Bernd 12 Nanba, Hidetsugu 512, 556 Nawrot, Monika 468 Okumura, Manabu 512, 556 388 Osi´ nski, Jedrzej Ozell, Benoˆıt 245 Pak, Alexander 533 Pallotta, Vincenzo 291 Palomino, Marco A. 544 Pellegrini, Thomas 59 Przepi´ orkowski, Adam 224, 400 Psutka, Josef 280 Raptis, Spyros 35 Recourc´e, Ga¨elle 522 Romaniuk, Julia 234 Romportl, Jan 280 Ryvityte, Birute 412 Sagot, Benoˆıt 183, 522 Sarasola, Kepa 327 Savary, Agata 139 Schnurrenberger, Martin 456 Sebastiani, Fabrizio 500 Shinoda, Koichi 24

213

Sikora, Piotr 139 Sinkuniene, Jolanta 412 Snoeren, Natalie D. 70 S ¸ tef˘ anescu, Dan 435 Stern, Rosa 522 Suszcza´ nska, Nina 234 Suzuki, Yoshimi 115, 192 Szmal, Przemyslaw 234 Tachbelie, Martha Yiﬁru 82 Takezawa, Toshiyuki 512, 556 Tolone, Elsa 183 Trancoso, Isabel 59 Tsiakoulis, Pirros 35 Usoniene, Aurelija

412

Vetulani, Zygmunt 303 Volkova, Maria 380 Walkowska, Justyna 315 Wilson, Stephen 94 Woli´ nski, Marcin 139 Wuytack, Tom 544 Yamashita, Kazuyuki 115 Yegorov, Anatoly 380 Zi´ olko, Bartosz 105 Zi´ olko, Mariusz 105 Zouaq, Amal 245